Model training method and device, data cleaning method and electronic equipment

Abstract

The invention provides a model training method and device, a data cleaning method and electronic equipment, and the method comprises the steps: obtaining a first data set; performing multi-scale clustering processing on the first data set to obtain a clustered data set; performing anomaly detection on the first data set to obtain an anomaly feature set; performing multi-feature extraction on the first data set according to a plurality of preset feature types to obtain a target feature set corresponding to each feature type; performing feature fusion on the clustering data set, the abnormal feature set and each target feature set to obtain an enhanced feature set; and training the integrated model by using the enhanced feature set to obtain a trained integrated model. By applying the model training method provided by the invention, the integrated module with higher precision can be obtained, and when the integrated model is applied to large-scale data cleaning, the data cleaning efficiency can be improved.

Description

Technical Field

[0001] The present application belongs to the field of data processing technology, and specifically relates to a model training method and device, a data cleaning method and electronic equipment. Background Art

[0002] With the rapid development of Internet technology, the generation and collection of large-scale traffic data are becoming increasingly common. This data is of great value for network management, security analysis, and service quality optimization. However, raw traffic data often has problems such as noise, outliers, and missing values, which seriously affect the quality and availability of data.

[0003] The existing technology removes noise, outliers, missing values ​​and other problems in raw traffic data by cleaning the raw traffic data. Traditional data cleaning methods mainly rely on simple statistical techniques and rule filtering, but for large-scale traffic data, traditional data cleaning methods are difficult to effectively process and have low processing efficiency. Summary of the invention

[0004] In view of this, the purpose of this application is to provide a model training method and device, a data cleaning method and electronic device to improve the efficiency of data cleaning.

[0005] The embodiment of the present application is implemented as follows: In a first aspect, an embodiment of the present application provides a model training method, including: obtaining a first data set, which is a traffic data set obtained by performing feature extraction according to a time series; performing multi-scale clustering processing on the first data set to obtain a clustered data set; performing anomaly detection on the first data set to obtain an anomaly feature set; performing multi-feature extraction on the first data set according to a plurality of preset feature types to obtain a target feature set corresponding to each of the feature types; performing feature fusion on the clustered data set, the anomaly feature set and each of the target feature sets to obtain an enhanced feature set; and using the enhanced feature set to train an integrated model to obtain a trained integrated model.

[0006] In an embodiment of the present application, by capturing the similarity and complexity between different features of traffic data in the time dimension and the dependency between features, the abnormal information of the traffic data is identified and processed, and then a high-precision integrated model is trained. The integrated model is applied to clean the traffic data, which can not only improve the data quality and availability, but also improve the efficiency of data cleaning.

[0007] In an optional embodiment of the first aspect, obtaining the first data set includes: obtaining an initial data set; performing time alignment and segmentation processing on the initial data set according to a time series; indexing the aligned and segmented initial data set; and performing feature extraction on the indexed initial data set according to a time series to obtain a first data set.

[0008] In an embodiment of the present application, the initial data set is a traffic data set composed of traffic data obtained from at least one data source. Since each traffic data in the initial data set may come from different data sources, and different data sources may have different time granularities, in order to facilitate the subsequent unified processing of each traffic data, each traffic data in the initial data set is time-aligned and segmented, and the segmented traffic data can extract the structural change points of the traffic data in the time series. The aligned and segmented initial data set is indexed to facilitate the management of each traffic data in the initial data set. Feature extraction is performed on the indexed initial data set according to the time series to obtain the features of the traffic data in different time dimensions.

[0009] In an optional embodiment of the first aspect, the feature extraction of the initial data set for which the index has been constructed according to the time series to obtain the first data set includes: decomposing the initial data set for which the index has been constructed to obtain the second data set; extracting the statistical features of the second data set within a preset time window to obtain the third data set; based on the statistical features of the third data set, extracting relevant features of the third data set to obtain a relevant feature set; extracting time-frequency features of the relevant feature set and / or the second data set to obtain a time-frequency feature set, wherein the time-frequency features are three-dimensional features used to characterize time-frequency-scale; extracting complexity features of the time-frequency feature set and / or the second data set to obtain a complex feature set, wherein the complexity features are used to characterize the degree of irregularity of the time series; and performing feature fusion on the complex feature set and / or the time-frequency feature set and / or the relevant feature set to obtain the first data set.

[0010] In the embodiment of the present application, by decomposing the traffic data, extracting statistical features, video features and complexity features, various aspects of the traffic data in the time series are considered, for example, by decomposing and analyzing the trend and periodicity of the traffic data in the time series, by extracting statistical features, analyzing the local statistical characteristics and correlation of the traffic data, by extracting time-frequency features, obtaining the three-dimensional representation of the traffic data in time-frequency-scale, and by extracting complexity features, obtaining the complexity of the traffic data. After obtaining the complex feature set, the time-frequency feature set and the related feature set, feature fusion is performed on at least one of the sets to provide a rich information basis for subsequent feature selection, model construction and training.

[0011] In an optional embodiment of the first aspect, after the complex feature set and / or the time-frequency feature set and / or the related feature set are subjected to feature fusion to obtain the first data set, the process further includes: performing dimensionality reduction processing on the first data set, wherein the dimensionality reduction processing on the first data set includes: screening the main features in the first data set to obtain a screened feature set, wherein the main features are data features with high correlation with the target variable, and the target variable is a main traffic indicator used for prediction; applying an autoencoder to perform nonlinear dimensionality reduction processing on the screened feature set to obtain a nonlinear dimensionality reduction feature set; performing high-order tensor decomposition on the nonlinear dimensionality reduction feature set to obtain a structural feature set; performing visualization processing on the structural feature set to obtain a visualization feature set; clustering the visualization feature set to obtain multiple similar feature groups; resampling each of the similar feature groups to obtain multiple sample sets corresponding to each of the similar feature groups; for each sample set corresponding to the same similar feature group, calculating the similarity coefficient of each feature in the similar feature group in different sample sets; filtering the features whose similarity coefficients in each of the similar feature groups are lower than a preset threshold to obtain the first data set after dimensionality reduction.

[0012] In the embodiment of the present application, feature screening is performed on the first data set to obtain important features of the traffic data and to simplify the data, which can improve the model accuracy in the subsequent model training process. The feature set after feature screening is subjected to dimensionality reduction processing, which can realize the visualization of high-dimensional data and intuitively understand the distribution and clustering results of the data. The first data set after feature screening and dimensionality reduction not only retains the main information of the original data, but also greatly reduces the feature space dimension.

[0013] In an optional embodiment of the first aspect, the performing anomaly detection on the first data set to obtain an anomaly feature set includes: applying a preset local anomaly factor algorithm to perform anomaly detection on the first data set to obtain a local anomaly detection result; applying a preset seasonal autoregressive integrated sliding average model to perform anomaly detection on the first data set to obtain a contextual anomaly detection result; applying a preset pruned exact linear time algorithm to perform anomaly detection on the first data set to obtain a structural change detection result; applying a preset variational autoencoder to perform anomaly detection on the first data set to obtain a reconstructed anomaly detection result; applying a preset online anomaly detection algorithm to perform anomaly detection on the first data set to obtain a dynamic anomaly detection result; performing evidence synthesis and uncertainty quantification processing on at least one of the local anomaly detection result, the contextual anomaly detection result, the structural change detection result, the reconstructed anomaly detection result and the dynamic anomaly detection result to obtain an anomaly feature set.

[0014] In an embodiment of the present application, anomaly detection is performed on the first data set through different algorithms, and evidence and uncertainty quantification processing is performed based on each anomaly detection result to obtain an abnormal feature set, so that in the subsequent model training process, the model can better learn the patterns and rules of abnormal data through the abnormal feature set, and at the same time reduce the model's dependence on noise and irrelevant information, thereby improving the generalization ability of the model.

[0015] In an optional embodiment of the first aspect, the application of the enhanced feature set to train the integrated model to obtain a trained integrated model includes: splitting the enhanced feature set into a training set, a test set, and a validation set; applying the training set to construct a univariate model and a multivariate model; applying the validation set to perform grid search and time series cross-validation on the univariate model and the multivariate model to optimize the model parameters of the univariate model and the multivariate model according to the validation results; applying the test set to perform performance testing on the optimized univariate model and the optimized multivariate model; and if the performance test is passed, synthesizing the optimized univariate model and the optimized multivariate model to obtain a trained integrated model.

[0016] In an embodiment of the present application, a training set is first used to construct a univariate model and a multivariate model, and then a validation set and a test set are used to optimize and perform performance tests on the univariate model and the multivariate model, respectively, to improve the accuracy of the integrated model. A high-precision integrated model is used to clean the traffic data, which can not only improve data quality and availability, but also improve the efficiency of data cleaning.

[0017] In a second aspect, the present application provides a data cleaning method, comprising: obtaining an original traffic data set; inputting the original traffic data set into a trained integration model to obtain a similar data set output by the integration model, wherein the integration model is trained by the above-mentioned model training method; wherein the similar data set deletes or fills in missing values ​​in the original traffic data set, and / or formats or replaces abnormal values ​​in the original traffic data set.

[0018] In an embodiment of the present application, after obtaining the original traffic data set, it is only necessary to apply the trained integrated model to perform data cleaning on each traffic data in the original traffic data set to obtain a similar traffic data set that is similar to the original traffic data set after cleaning, which can not only improve the quality and availability of the traffic data, but also improve the efficiency of data cleaning.

[0019] In an optional embodiment of the second aspect, it also includes: smoothing the similar data set to obtain a smoothed data set; marking prominent data in the smoothed data set to obtain a marked data set; obtaining a data processing log corresponding to the marked data set; updating the difference information to the data processing log to obtain an updated data processing log, wherein the difference information is the difference information obtained by comparing the similar data set with the original traffic data set; based on the updated data processing log and the smoothed data set, obtaining a revised similar data set, wherein the revised similar data set is a smoothed data set that carries at least the data processing history record and the difference information in the updated data processing log.

[0020] In an embodiment of the present application, after obtaining a similar data set, in order to further improve the data quality, the similar data set may be smoothed to determine whether it contains prominent data. The prominent data is marked so that the marked data set can quickly locate the prominent data. According to the difference information between the similar data set and the original traffic data set, the data processing log corresponding to the marked data set is updated, and a corrected similar data set is obtained based on the updated data processing log and the smoothed data set. Through the corrected similar data set, not only the high-quality traffic data after data cleaning can be retained, but also detailed data processing history records and difference information can be included, which is convenient for subsequent analysis and tracing of traffic data.

[0021] In a third aspect, the present application provides a model training device, comprising: an acquisition module, a clustering module, a detection module, a multi-feature extraction module, a feature fusion module, and a training module; the acquisition module is used to acquire a first data set, wherein the first data set is a traffic data set obtained by feature extraction according to a time series; the clustering module is used to perform multi-scale clustering processing on the first data set to obtain a clustered data set; the detection module is used to perform anomaly detection on the first data set to obtain an anomaly feature set; the multi-feature extraction module is used to perform multi-feature extraction on the first data set according to a plurality of preset feature types to obtain a target feature set corresponding to each of the feature types; the feature fusion module is used to perform feature fusion on the clustered data set, the anomaly feature set, and each of the target feature sets to obtain an enhanced feature set; the training module is used to apply the enhanced feature set to train the integrated model to obtain a trained integrated model.

[0022] In a fourth aspect, the present application provides an electronic device, comprising: a memory and a processor, the processor being connected to the memory; the memory being used to store programs; the processor being used to call the programs stored in the memory to execute the above-mentioned data training method, or to execute the above-mentioned data cleaning method.

[0023] Other features and advantages of the present application will be described in the following description. The purpose and other advantages of the present application can be realized and obtained through the structures specifically pointed out in the written description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those of ordinary skill in the art, other drawings can also be obtained based on these drawings. The above and other purposes, features and advantages of the present application will be more clearly shown in the drawings.

[0025] Figure 1 A schematic diagram of a flow chart of a model training method provided in an embodiment of the present application is shown; Figure 2 A schematic diagram of the process of dimensionality reduction processing provided in an embodiment of the present application is shown; Figure 3 A schematic diagram of the structure related to the abnormality detection process provided in an embodiment of the present application is shown; Figure 4 A schematic diagram of a data cleaning method provided in an embodiment of the present application is shown; Figure 5 A device structure diagram of a model training device provided in an embodiment of the present application is shown; Figure 6 A device structure diagram of a data cleaning device provided in an embodiment of the present application is shown; Figure 7 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] The technical solutions in the embodiments of the present application will be described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The following embodiments can be used as examples to more clearly illustrate the technical solutions of the present application, and cannot be used to limit the scope of protection of the present application. It will be appreciated by those skilled in the art that, in the absence of conflict, the features in the following embodiments and embodiments can be combined with each other.

[0027] It should be noted that similar numbers and letters represent similar items in the following figures, so once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. At the same time, in the description of this application, relational terms such as "first", "second", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprise", "include" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0028] Furthermore, the term "and / or" in this application is merely a term used to describe the association relationship between associated objects, indicating that three relationships may exist. For example, A and / or B may represent three situations: A exists alone, A and B exist at the same time, and B exists alone.

[0029] In the description of the embodiments of the present application, unless otherwise clearly specified or limited, the technical term "connection" may refer to a direct connection or an indirect connection through an intermediate medium.

[0030] See also Figure 1 , Figure 1 A method flow chart of a model training method provided in an embodiment of the present application, the model training method can be applied to processors of various system platforms, computer terminals or various mobile devices, and the method may include the following steps: S101: Acquire a first data set.

[0031] S102: Perform multi-scale clustering processing on the first data set to obtain a clustered data set.

[0032] S103: Perform anomaly detection on the first data set to obtain an abnormal feature set.

[0033] S104: performing multi-feature extraction on the first data set according to the preset multiple feature types respectively, to obtain a target feature set corresponding to each feature type.

[0034] S105: Fusing the clustering data set, the abnormal feature set, and each target feature set to obtain an enhanced feature set.

[0035] S106: Apply the enhanced feature set to train the integrated model to obtain a trained integrated model.

[0036] In the model training method provided in the embodiment of the present application, in order to improve the cleaning efficiency of large-scale traffic data, a first data set is obtained, and multi-scale clustering, anomaly detection and multi-feature extraction are performed on the first data set according to multiple feature types to obtain a clustered data set, an abnormal feature set and multiple target feature sets, and each set is fused to obtain an enhanced feature set, and the enhanced feature set is used to perform model training to obtain a trained integrated model, and then the large-scale traffic data is cleaned through the integrated model.

[0037] The above steps are described in detail below.

[0038] S101: Acquire a first data set.

[0039] The first data set is a flow data set obtained by extracting features according to a time series. Each flow data in the flow data set is flow data obtained from multiple data sources, and each data collection point may include a network flow monitoring device and a log server. The collected flow data has time continuity, and the data type of the flow data includes IP address, timestamp, packet size, and protocol type.

[0040] When collecting traffic data from data sources, in order to ensure that the collected traffic data has temporal continuity and representativeness, you can set the data collection frequency and sampling strategy corresponding to each data source. For example, for high-traffic network nodes, you may need to sample multiple times per second, while for low-traffic nodes, you may sample once per minute.

[0041] In the embodiment of the present application, the obtained traffic data set is used as the initial data set, and the initial data set is processed to obtain the first data set. The specific process of obtaining the first data set is: obtaining the initial data set; performing time alignment and segmentation processing on the initial data set according to the time series; indexing the aligned and segmented initial data set; and extracting features from the indexed initial data set according to the time series to obtain the first data set.

[0042] Optionally, before performing time alignment and segmentation processing on the initial data set, the initial data set may be format-unified to obtain an initial data set in a unified format. The format unification processing includes at least one of unifying the time format (such as converting to UTC time), unifying the data unit (such as converting all traffic data to bytes / second), and unifying the data type (such as converting all numerical data to floating point numbers).

[0043] In this application, since the data in the initial data set may come from different data sources, and the time granularity of each data source is different, the interpolation method can be applied to time align the initial data set, and unify the traffic data of different time granularities or different timestamps to the same time point. For example: cubic spline interpolation can be applied to achieve time alignment; if the initial data set belongs to a multivariate time series, the DTW (dynamic time warping) algorithm can be used to deal with the time alignment problem between different variables. In a time series, data in different time periods may have different statistical characteristics (such as mean, variance, trend, etc.). By finding the structural change points and segmenting the data, for example: the PELT (Pruned Exact Linear Time) algorithm can be used, which can find the optimal segmentation by minimizing the following cost function. The PELT algorithm is: ; in, is the minimum segment cost up to time point t, is the cost function for the subsequence from s+1 to t, is the complexity penalty term.

[0044] After the initial data set is time-aligned and segmented, in order to ensure data integrity, the aligned and segmented initial data set can be saved in a dedicated time series database (e.g., InfluxDB (Influx Database) or TimescaleDB (Timescale Database), etc.) in a distributed storage system, and in order to facilitate quick query of traffic data for a specific time range or a specific device, the aligned and segmented initial data set is indexed. When querying data in a distributed storage system, you can query by timestamp and / or device information of the data source to which the traffic data belongs. Among them, the process of time-aligning, segmenting, and indexing the initial data set is a preprocessing process. After the initial data set is preprocessed, feature extraction is performed on the preprocessed initial data set (i.e., the indexed initial data set) according to the time series to obtain the first data set.

[0045] Optionally, after time alignment, the initial dataset can also be processed for missing values. Use the modified Z-score method to identify missing values ​​in the initial dataset, and process the missing values ​​to obtain the filled initial dataset. If the traffic data in the initial dataset is short-term missing, linear interpolation or spline interpolation methods can be used. For long-term missing, a sequence-to-sequence (Seq2Seq) model based on LSTM (Long Short-Term Memory) can be used to fill missing values.

[0046] Based on the preprocessing process of time alignment, segmentation and index construction of the initial data set described in the embodiment of the present application, the following application embodiments may be provided: Suppose you are processing traffic data from a large network. The raw data may contain information about millions of packets per second. By setting an appropriate sampling strategy (such as sampling every 10 seconds), the amount of data can be reduced to a manageable level. Using the modified Z-score method, you may find that the traffic at certain time points is abnormally high, which may indicate a network attack or device failure. In the process of format unification, you may need to unify the traffic units reported by different devices (such as bits / second and bytes / second) into bytes / second. In the process of time alignment, you may find that the clocks of some devices are slightly offset, which needs to be corrected using the DTW algorithm. For missing traffic data, use the LSTM model to predict and fill in based on historical patterns. When segmenting the time series, the PELT algorithm may find that the network traffic pattern is significantly different on weekdays and weekends, and automatically divide the data into weekday segments and weekend segments. Finally, these cleaned and pre-processed data are stored in TimescaleDB, so that subsequent analysis can quickly query the traffic data of a specific time range or a specific device.

[0047] In the embodiment of the present application, after preprocessing the initial data set, the process of extracting features according to the time series can be as follows: Decompose the indexed initial data set to obtain the second data set; extract the statistical features of the second data set within a preset time window to obtain the third data set; extract relevant features of the third data set based on the statistical features of the third data set to obtain a relevant feature set; extract time-frequency features from the relevant feature set and / or the second data set to obtain a time-frequency feature set; extract complexity features from the time-frequency feature set and / or the second data set to obtain a complex feature set; perform feature fusion on the complex feature set and / or the time-frequency feature set and / or the relevant feature set to obtain the first data set. Among them, the time-frequency feature is a three-dimensional feature used to characterize time-frequency-scale, and the complexity feature is used to characterize the irregularity of the time series.

[0048] Among the traffic data contained in the indexed initial data set, some data may have time series features of univariate time series, while other data may have time series features of multivariate time series. Before decomposing the indexed initial data set, the indexed initial data set may also be analyzed, specifically, dimension detection and correlation analysis may be performed on the indexed initial data set to determine the traffic data of univariate time series and the traffic data of multivariate time series in the indexed initial data set. Dimension detection of the indexed initial data set refers to checking the number of observations at each time point, such as a single traffic value is a univariate variable, and multiple measurement indicators are multivariate (the variables here are numerical variables (traffic size, memory usage, etc.), categorical variables (protocol type), etc.). Correlation analysis of the indexed initial data set is to calculate the correlation coefficient between variables by mutual information coefficient to determine the independence and correlation of variables, such as setting a correlation threshold (for example, 0.7), and judging the dependency between variables based on the threshold.

[0049] Different decomposition methods are used to decompose traffic data with different time series characteristics.

[0050] For the traffic data of univariate time series, the additive model decomposition method is applied, such as the Prophet (time series prediction) algorithm, to obtain the trend, seasonality and residual components by fitting the trend, seasonality and holiday effects. The Prophet algorithm is expressed as: ; in, represents the trend function, Represents periodic changes, represents the holiday effect, represents the error term.

[0051] For multivariate time series traffic data, matrix decomposition and singular value decomposition are applied to obtain multivariate decomposition results. Among them, matrix decomposition is used to process the linear relationship between multiple variables; singular value decomposition is used to reduce the dimension and extract the main change mode. Since the MSSA (Singular Spectrum Analysis) method involves the construction of the trajectory matrix and singular value decomposition, the MSSA method can be applied to realize the process of matrix decomposition and singular value decomposition, and the corresponding expression is: ; Where X is the trajectory matrix, U and V are the left and right singular vector matrices, is a diagonal matrix of singular values, V T is the transpose of V.

[0052] Since the decomposition methods used for the flow data of the univariate time series and the flow data of the multivariate time series are different, after obtaining the trend, seasonality and residual components and the multivariate decomposition results, the trend, seasonality and residual components and the multivariate decomposition results are subjected to slope calculation, cycle identification and statistical extraction to obtain the second data set. Among them, the slope calculation can be to calculate the overall change trend of the time series using a linear regression algorithm to identify the rate of increase or decrease of the flow data; the Fourier transform can be used for cycle identification, and the extracted statistics can include the mean, variance and median value, etc.

[0053] After obtaining the second data set, a time window can be set for the second data set according to a preset time rule, and statistics can be calculated in each time window. A time series can be generated by sliding the time window to capture local dynamic statistical features and obtain a corresponding statistical feature set (i.e., a third data set). Then, relevant features are extracted from each statistic and feature in the third data set to obtain a relevant feature set, and the strength of the mutual relationship between the relevant feature sets is characterized to understand the dependency between the features of the traffic data. Among them, the relevant feature extraction can be to apply the Pearson correlation coefficient to calculate the correlation coefficient between each statistic feature in the third data set and the mutual information based on the kernel density estimation, and use the kernel density estimation method to approximate the probability distribution density function of the variable, thereby calculating the mutual information between the two variables, measuring the nonlinear correlation between the variables, and capturing the nonlinear dependency that may exist between the variables. Time-frequency feature extraction is performed on the relevant feature set and / or the second data set to obtain a time-frequency feature set. Among them, a three-dimensional representation of time-frequency-scale is generated by wavelet function convolution and scale transformation to obtain a time-frequency feature, which combines information in three dimensions of time, frequency, and scale. Finally, sample entropy, approximate entropy and multi-scale entropy are calculated for the video feature set and / or the second data set to extract the complexity features of the video feature set and / or the second data set and obtain a complexity feature set. Among them, the main function of sample entropy is to measure the irregularity of the time series, and its corresponding calculation formula is: ; Among them, m is the pattern length, r is the similarity threshold, N is the time series length, and A(m,r) is the pattern matching count. The main function of approximate entropy is to measure the regularity of time series; the main function of multiscale entropy is to analyze complexity at multiple time scales. Each entropy provides complexity information in different dimensions. The results of the three entropies complement each other and together constitute a complexity feature set to characterize the irregularity of time series through complexity features.

[0054] In an optional embodiment, after the complexity feature set and / or the video feature set and / or the related feature set are subjected to feature fusion to obtain the first data set, the first data set may also be standardized and normalized to obtain a processed first data set.

[0055] If any set in the complexity feature time-frequency feature set is a set obtained by directly processing the second data set, the feature fusion process can select at least one of the complexity feature set, the video feature set and the related feature set for feature fusion processing according to business needs, and standardize and normalize the set obtained after feature fusion to obtain the first data set, wherein the feature fusion process includes at least one of feature splicing, feature weighting and feature selection. If the time-frequency feature set is a set obtained by extracting time-frequency features from the related feature set, and the complexity feature set is a set obtained by extracting complexity features from the time-frequency feature set, then the feature fusion process is a feature selection process, and since the time-frequency feature set is obtained by processing the related feature set, and the complexity feature set is obtained by processing the video feature set, the first data set obtained by feature selection is a complexity feature set, and standardizing and normalizing the first data set is actually standardizing and normalizing the complexity feature set. Therefore, if the time-frequency feature set is a set obtained by extracting time-frequency features from the relevant feature set, and the complexity feature set is a set obtained by extracting complexity features from the time-frequency feature set, then feature fusion processing may not be performed, and the complexity feature set can be directly standardized and normalized to obtain the processed first data set.

[0056] In an optional embodiment, after the complex feature set and / or the time-frequency feature set and / or the related feature set are subjected to feature fusion to obtain the first data set, the first data set may be subjected to dimensionality reduction processing; or, after the complex feature set and / or the time-frequency feature set and / or the related feature set are subjected to feature fusion to obtain the first data set, and the first data set is standardized and normalized, the processed first data set may be subjected to dimensionality reduction processing.

[0057] refer to Figure 2 , performing dimensionality reduction processing on the first data set may include the following steps: S201: Filter the main features in the first data set to obtain a filtered feature set.

[0058] Among them, the main features are data features that have a high correlation with the target variable, and the target variable is the main traffic indicator used for prediction, such as: traffic size, duration, etc.

[0059] In an embodiment of the present application, the first data set is the first data set obtained after executing the above-mentioned S101 to perform feature fusion, or the first data set obtained after performing feature fusion and standardization and normalization processing, or the first data set obtained after performing feature fusion, standardization and normalization processing and dimensionality reduction processing.

[0060] In an optional embodiment, during the screening of the main features of the first data set, the main features are initially screened by calculating the correlation between the features and evaluating the redundancy of the first data set. After the initial screening, further screening of the main features can be performed by random feature replacement and importance comparison, and then the linear correlation between the features can be quantified to evaluate the multicollinearity of the features, and based on the evaluation results, features are gradually added or deleted from the data to obtain a streamlined feature set, and finally, the important feature combinations in the streamlined feature set are retained by regularized principal component extraction to obtain a screened feature set.

[0061] Among them, a preliminary screening method for the main features of the first data set can be to use the mRMR (Minimum Redundancy Maximum Relevance) algorithm to calculate the correlation and redundancy between the features in the first data set, determine the correlation between each feature and the target variable based on the calculation results, retain features with high correlation with the target variable, delete features with high redundancy between features, and obtain a preliminary screening feature set after preliminary screening.

[0062] The Boruta (Boruta feature selection) algorithm is applied to select features from the initial screening feature set. The Boruta algorithm creates "shadow features" and performs multiple rounds of random forest training with the original features. By comparing the importance scores of the original features and the shadow features, the important features in the initial screening feature set are selected according to the scores to obtain the important feature set. Among them, the comparison of the importance of the original features and the shadow features is not only a comparison with a single shadow feature, but a comparison with the MZSA (maximum importance score) of all shadow features. Specifically, the Boruta algorithm first creates a shadow feature for each original feature, shuffles the shadow feature, eliminates its correlation with the target variable, and then merges the original feature with the shadow feature. After the merged features are processed by the model, the importance scores of the original features and the shadow features are calculated. Finally, the features whose importance scores of the original features are greater than the importance scores of the shadow features are selected as important features. In the process of comparing the importance scores, this process needs to be repeated in multiple rounds of iterations, and statistical tests are used to evaluate the significance of the importance differences; and the relationship between the features and the target variable and the mutual influence between the features are considered, and the final decision is made based on the results of multiple rounds to select the important feature set.

[0063] VIF (variance inflation factor) is used to evaluate the multicollinearity of each feature in the important feature set, and the obtained VIF value is the multicollinearity evaluation result of the feature. The calculation formula of VIF is: ; in, It is the coefficient of determination of the multivariate regression of the i-th feature on all other features. The larger the VIF value, the stronger the collinearity of the feature with other features. For the VIF value corresponding to each feature of the traffic data in the important feature set, new features are gradually added or deleted to the traffic data. In the process of gradually adding new features or deleting features, an empty model is provided. The empty model provides a performance benchmark, which can be compared with the benchmark each time a feature is added or deleted to evaluate the VIF value of each feature in the traffic data after each addition or deletion of features. For example: if you want to delete a feature, when the VIF values ​​of multiple features are greater than 10, the feature with the largest VIF value is deleted first, and the VIF values ​​of the remaining features are re-evaluated after deletion, and the process is carried out step by step until the VIF values ​​of all features are within an acceptable range; if you want to add features, the addition of new features must significantly improve the model performance, and the addition cannot cause the VIF value of any feature to exceed the threshold. If the addition causes the VIF value of other features to increase significantly, it needs to be considered carefully even if it does not exceed the threshold. After gradually adding or deleting features, a streamlined feature set is obtained until the model performance is no longer significantly improved.

[0064] Optionally, apply the F-test (joint hypothesis test) or AIC (Akaike Information Criterion) / BIC (Bayesian Information Criterion) to evaluate whether the newly added features significantly improve the model performance.

[0065] Optionally, when adding or removing features, data features related to the target variable are given priority.

[0066] The screened feature set obtained by regularizing the principal component extraction of the streamlined feature set is a linear dimensionality reduction feature set. Specifically, the Sparse PCA (sparse principal component analysis) method is used to perform regularized principal component extraction on the streamlined feature set to balance the reconstruction error and the sparsity of the load matrix, while maintaining the main information, reducing the number of features, so that the load matrix of the principal component becomes sparse. It is not just a simple retention of the principal component structure, but through the optimization algorithm while maintaining the main information of the data, a more concise and easier to interpret feature combination is obtained.

[0067] S202: Apply an automatic encoder to perform nonlinear dimensionality reduction processing on the screened feature set to obtain a nonlinear dimensionality reduction feature set.

[0068] Among them, since the screening feature set obtained in the above S201 is a linear dimension reduction feature set, in order to capture the nonlinear relationship between the features, an LSTM autoencoder can be used for nonlinear dimension reduction processing. Specifically, the LSTM autoencoder includes an encoder and a decoder. The encoder encodes the screening feature set to encode the feature sequence, compresses the input feature sequence into a vector of a fixed length, and then decodes and reconstructs the vector to reconstruct a new feature sequence.

[0069] S203: Perform high-order tensor decomposition on the nonlinear dimensionality reduction feature set to obtain a structural feature set.

[0070] This application can use Tucker decomposition to perform high-order tensor decomposition on the nonlinear dimensionality reduction feature set. The expression of Tucker decomposition is: ; Among them, X is the original data tensor, G is the core tensor, A, B, C are factor matrices, Represents the n-module tensor product. Through Tucker decomposition, the structural information of multidimensional data can be retained and the structural feature set can be obtained.

[0071] S204: Perform visualization processing on the structural feature set to obtain a visualization feature set.

[0072] In an embodiment of the present application, the t-SNE (t-distributed Stochastic Neighbor Embedding) algorithm is applied to minimize the KL divergence between the similarity of data points in a high-dimensional space and the similarity of corresponding points in a low-dimensional space, thereby realizing the visualization of high-dimensional data and intuitively understanding the distribution and clustering structure of the data.

[0073] S205: Clustering the visualization feature set to obtain multiple similar feature groups.

[0074] In an embodiment of the present application, a spectral clustering algorithm is applied to calculate the similarity between each data in a visualization feature set, a similarity graph is constructed, and then the Laplace matrix of the similarity graph is calculated. The eigenvector of each data is obtained through the Laplace matrix, and the data are clustered according to the eigenvector to obtain a plurality of similar feature groups, in which the features of each data in the similar feature groups are similar.

[0075] S206: Resample each similar feature group to obtain multiple sample sets corresponding to each similar feature group.

[0076] In the embodiment of the present application, Bootstrap is mainly used to resample similar feature groups to obtain multiple sample sets. Specifically, for each similar feature group, at least one different data is repeatedly extracted from the similar feature group as a sample set.

[0077] S207: For each sample set corresponding to the same similar feature group, calculate the similarity coefficient of each feature in the feature similarity group in different sample sets.

[0078] Since the features of each data in the same similar feature group are similar, after obtaining each sample set corresponding to the similar feature group, the features of the sample set are selected to perform similarity calculations with the features of other sample sets to obtain the similarity coefficients of the features in different sample sets. The similarity coefficient is the Jaccard coefficient.

[0079] S208: Filter the features whose similarity coefficients in each similar feature group are lower than a preset threshold to obtain a first data set after dimensionality reduction.

[0080] In the present application, if the Jaccard coefficient is lower than a preset threshold, it indicates that the similarity of the feature in the groups with the same similar feature is low, and therefore it is filtered out.

[0081] Optionally, if the Jaccard coefficient difference between a feature in any sample set and other sample sets is relatively large, the feature may be filtered.

[0082] Based on the dimensionality reduction process corresponding to S201 to S208 above, the first data set is firstly subjected to feature screening, the features in the first data set are simplified, and then the nonlinear relationship between the features is captured, and the screened feature set obtained after screening is encoded and decoded for reconstruction, and then high-order tensor decomposition is performed to retain the structural information of the multi-dimensional data, and then visualization processing is performed to map the high-dimensional data to two dimensions, and finally the first data set with stable structure and reduced dimensionality is obtained after clustering and similarity calculation on the visualized feature set. Applying the methods of S201 to S208 above not only retains the important information of the data, but also reduces the dimension of the feature space, which is convenient for related calculations in the subsequent model training process.

[0083] S102: Perform multi-scale clustering processing on the first data set to obtain a clustered data set.

[0084] In an embodiment of the present application, in order to discover the traffic behavior patterns of each traffic data in the first data set at different time scales, the process of multi-scale clustering processing of the first data set is: applying the HDBSCAN (Hierarchical Density-Based Spatial Clustering of Applications with Noise) algorithm to calculate the density of the data and the optimal number of clusters at different time scales, and obtain an initial clustering result, clustering the first data set according to the initial clustering result to obtain an initial clustering data set. Apply the Gaussian kernel function to process the initial clustering data set to obtain a similarity matrix related to the initial clustering structure. Application formula: (Λ is the eigenvalue diagonal matrix) Perform eigenvalue decomposition on the similarity matrix to determine the optimal hierarchical clustering structure of the initial clustering data set. Calculate the encoding length and select the optimal level of the hierarchical clustering structure. Use the MDL (Minimum Description Length) principle to select the optimal level by minimizing the encoding length of the clustering structure, thereby obtaining the optimized clustering result. Then, calculate the internal compactness and external separation of this optimized clustering result to obtain a clustering quality assessment report, that is, quantify the clustering quality by using the Silhouette coefficient. The calculation formula of the Silhouette coefficient is: ; in, is the average distance between data i and other data in the same cluster, It is the average distance between data i and the nearest non-clustered sample. The closer the Silhouette coefficient is to 1, the better the clustering effect.

[0085] In this application, the clustering quality assessment report is used to construct clustering related features, affect the calculation weights of Mahalanobis distance and membership degree, and determine the reliability of cluster centers. It can affect the extraction method and weight setting of clustering related features. When building a model, it can be used to guide sample weight allocation and model parameter adjustment. In the final data cleaning, the evaluation results of the clustering quality assessment report can be saved as part of the metadata to track and verify the cleaning effect.

[0086] After obtaining the clustering quality assessment report, clustering related features are extracted from the first data set based on the clustering quality assessment report to form a clustering data set.

[0087] S103: Perform anomaly detection on the first data set to obtain an abnormal feature set.

[0088] refer to Figure 3The structural schematic diagram related to the anomaly detection process shown in the figure, the specific implementation process of performing anomaly detection on the first data set and obtaining the anomaly feature set is: The first data set is subjected to anomaly detection using a preset local anomaly factor algorithm to obtain a local anomaly detection result. The first data set is subjected to anomaly detection using a preset seasonal ARIMA (Autoregressive Integrated Moving Average) model to obtain a contextual anomaly detection result. The first data set is subjected to anomaly detection using a preset PELT (Pruned Exact Linear Time) algorithm to obtain a structural change detection result. The first data set is subjected to anomaly detection using a preset variational autoencoder to obtain a reconstruction anomaly detection result. The first data set is subjected to anomaly detection using a preset online anomaly detection algorithm to obtain a dynamic anomaly detection result. Evidence synthesis and uncertainty quantification processing are performed on at least one of the local anomaly detection result, the contextual anomaly detection result, the structural change detection result, the reconstruction anomaly detection result, and the dynamic anomaly detection result to obtain an anomaly detection report; and feature extraction is performed on the anomaly detection report according to the preset anomaly index to obtain an anomaly feature set.

[0089] Among them, the local anomaly factor algorithm is used to estimate the local density of the first data set and calculate the relative density deviation to obtain the local anomaly detection result corresponding to the first data set. The seasonal ARIMA model is used to perform seasonal decomposition, autoregression and moving average modeling on the first data set to obtain the contextual anomaly detection result. The PELT algorithm is used to perform anomaly detection on the first data set. The process of anomaly detection on the first data set is: the PELT algorithm is used to perform dynamic programming segmentation on the first data set to divide the time series into multiple segments with similar characteristics, and the change points of the time series are detected by minimizing the cost function, so as to obtain the structural change detection result. The variational autoencoder performs encoding-decoding reconstruction and reconstruction error calculation on the first data set to obtain the reconstruction anomaly detection result. The online anomaly detection algorithm performs real-time data stream processing and dynamic threshold update on the first data set to obtain the dynamic anomaly detection result. Then, the Dempster-Shafer evidence theory is used to synthesize evidence and quantify uncertainty on all the obtained anomaly detection results to obtain the final anomaly feature detection report. The anomaly detection report is carefully feature extracted according to anomaly indicators such as the frequency, duration and severity of abnormal samples to obtain an abnormal feature set.

[0090] Based on the anomaly detection process of the above S103 corresponding embodiment, there are the following specific application scenarios: Suppose there is a network traffic dataset containing multiple features such as packet size, protocol type, source and destination IP, etc. Through the HDBSCAN algorithm, traffic patterns that exist on different time scales may be found, such as normal traffic during daily working hours and low traffic periods on weekends. Using the LOF algorithm, some traffic with abnormally large packet sizes may be found, and through the seasonal ARIMA model, it may be found that the traffic at a certain point in time deviates significantly from the expected seasonal pattern. The PELT algorithm may help discover sudden changes in traffic patterns caused by network configuration changes, and through variational autoencoders, complex abnormal patterns that are difficult to describe with simple rules are discovered. Finally, sudden DDoS (Distributed Denial of Service) attacks are detected in real time through online algorithms. Through the Dempster-Shafer theory, evidence from these different sources is synthesized to obtain a comprehensive anomaly detection report, so that more detailed anomaly features can be obtained through the anomaly detection report, and the anomaly features are extracted to obtain an anomaly feature set.

[0091] S104: performing multi-feature extraction on the first data set according to the preset multiple feature types respectively, to obtain a target feature set corresponding to each feature type.

[0092] Among them, different feature types include: time series features, multi-scale time series similarity features, causal features, dependency structure features, domain-specific high-level statistics, and symbolic dynamics representation and complexity features.

[0093] In this application, the statistics, frequency domain features and information theory metrics of the first data set are calculated to obtain an initial time series feature set, and then the feature importance evaluation and screening are performed on the initial time series feature set. The time series of each traffic data of the first data set are aligned and compared at different time scales to extract the similarity features between each data at different time scales and obtain a similarity feature set. The multi-head self-attention mechanism is applied to calculate the attention weights and context vectors between different time steps for the first data set to obtain an attention feature set. Nonlinear causal relationships are identified for the first data set, and a causal feature set is obtained by constructing a nonlinear autoregressive model and calculating the prediction error. Multivariate dependency structure modeling is performed on the first data set, and a dependency structure feature set is obtained by constructing a binary copula function combination with a tree structure. The first data set is subjected to domain-specific advanced statistics extraction, and a domain feature set is obtained by calculating specific indicators such as protocol distribution entropy and connection duration distribution. The first data set is represented by symbolic dynamics and its complexity features are extracted. The dynamics symbols and complexity feature sets are obtained by converting the time series into symbolic sequences and calculating the permutation entropy and multi-scale entropy.

[0094] Among them, the filtered time series feature set, similarity feature set, attention feature set, causal feature set, dependency structure feature set, domain feature set, and dynamic symbol and complexity feature set are all target feature sets corresponding to different feature types.

[0095] S105: Fusing the clustering data set, the abnormal feature set, and each target feature set to obtain an enhanced feature set.

[0096] In the embodiment of the present invention, the process of feature fusion may be to connect all the features of the set in series, or to use a complex feature fusion algorithm to perform feature fusion, such as feature fusion of deep learning and multimodal feature fusion.

[0097] S106: Apply the enhanced feature set to train the integrated model to obtain a trained integrated model.

[0098] In an embodiment of the present application, the process of training the integrated model is: splitting the enhanced feature set into a training set, a test set, and a validation set; using the training set to construct a univariate model and a multivariate model; using the validation set to perform grid search and time series cross-validation on the univariate model and the multivariate model to optimize the model parameters of the univariate model and the multivariate model according to the validation results; using the test set to perform performance testing on the optimized univariate model and the optimized multivariate model; if the performance test is passed, the optimized univariate model and the optimized multivariate model are synthesized to obtain a trained integrated model.

[0099] Each traffic data contained in the enhanced feature set includes traffic data of univariate time series and traffic data of multivariate time series. After the enhanced feature set is split into training set, test set and validation set, each set still contains traffic data of univariate time series and traffic data of multivariate time series. Therefore, when the training set is used to construct a univariate model and a multivariate model, the univariate model is constructed by using the traffic data of the univariate time series in the training set, and the multivariate model is constructed by using the traffic data of the multivariate time series in the training set. Among them, the enhanced feature set is split into training set, test set and validation set in chronological order, and the amount of data in the training set is greater than that in the test set and validation set (for example, the first 70% of the enhanced feature set is used as the training set, the next 15% is used as the validation set, and the last 15% is used as the test set).

[0100] Specifically, autoregression modeling and exponential smoothing are performed on the training set to obtain at least one univariate model, wherein the ARIMA model can be used for autoregression modeling, and the Holt-Winters method can be applied for exponential smoothing. The vector autoregression (VAR) model and recursive neural network (such as LSTM) modeling are performed on the training set to obtain at least one multivariate model. Among them, the VAR model can capture the linear dependency between multiple variables, and its general form is: ; in, is a k-dimensional random vector, c is a k-dimensional constant vector, is a k×k coefficient matrix, is a k-dimensional white noise process.

[0101] The validation set is used to perform grid search and time series cross validation on at least one univariate model and at least one multivariate model to adjust the model parameters of the univariate model and the model parameters of the multivariate model. For example, adjusting the model parameters of the univariate model is to adjust different combinations of p, d, and q values ​​in the ARIMA model; adjusting the model parameters of the multivariate model is to adjust the hyperparameters such as the number of layers, the number of neurons, and the learning efficiency in the LSTM.

[0102] Then, the test set should be used to perform performance tests on the univariate model and the multivariate model. If they pass the test, model synthesis is performed to obtain a trained integrated model. If they fail the test, the model parameters of the univariate model and the multivariate model are adjusted again according to the above-mentioned model parameter adjustment method until they pass the test.

[0103] After the single variable model and variable model are tested, the multi-level prediction results of each model are combined, and the stacking method is used to build an integrated model. In Stacking, the XGBoost (eXtremeGradient Boosting) method is used as the meta-learner, and its objective function is: ; in is the training loss function, is the regularization term, and is the regularization parameter, T is the number of leaf nodes, and w is the leaf weight vector. This method can effectively combine the advantages of different models and improve prediction stability.

[0104] Optionally, after obtaining the integrated model, the integrated model can be further enhanced and optimized. The specific enhancement and optimization process is: adjust the time scale of the input data of the integrated model, such as aggregating hourly data into daily data, and injecting Gaussian noise to enhance the robustness of the model to obtain an enhanced integrated model. Finally, use the EWMA (Exponentially Weighted Moving Average) method to perform sliding window updates and weight adjustments on the enhanced integrated model. The calculation formula of EWMA is: ; in is the smoothed value at time t, is the actual observed value at time t, is the smoothing coefficient (0<α<1). The value can make the model better adapt to the changing trend of data, and finally obtain an adaptive integrated model. For example, in network traffic prediction, the dynamic adjustment of value, increases when the flow rate changes dramatically With fast response, reduce when flow is stable To maintain the stability of the forecast.

[0105] In the model training method provided in the embodiment of the present application, after obtaining the initial data set, the initial data set is subjected to time series feature extraction, and in the process of time series feature extraction, the data set is subjected to data decomposition, statistical feature extraction, related feature extraction, time-frequency feature extraction, complexity feature extraction, and dimensionality reduction. Through the above processing methods, the features of large-scale traffic data are finely screened to improve the corresponding foundation for the subsequent model training process. After obtaining the first data set, multi-scale clustering processing is performed on the first data set to obtain a clustering data set corresponding to the multi-scale clustering evaluation report. At the same time, anomaly detection is performed on the first data set, and the anomaly detection results under different anomaly detection conditions are analyzed. The corresponding anomaly detection report is obtained based on the anomaly detection results, and then the anomaly feature set in the first data set is obtained in combination with the anomaly detection report and related anomaly indicators. In addition, although the first data set is a data set obtained after processing such as time series feature extraction, statistical feature extraction, correlation feature extraction, time-frequency feature extraction, complexity feature extraction and dimensionality reduction, further feature extraction can be continued to obtain the target feature set corresponding to different feature types such as the filtered time series feature set, similarity feature set, attention feature set, causal feature set, dependency structure feature set, domain feature set, and dynamic symbol and complexity feature set. Finally, the obtained clustering data set, abnormal feature set and each target feature set are feature fused to obtain an enhanced feature set; the enhanced feature set is used to train the integrated model to obtain a trained integrated model. The method provided in the embodiment of the present application is used to extract and optimize different features of large-scale traffic data for multiple times, and an integrated model is trained to ensure that the integrated model has high-precision processing capabilities. When the integrated model is used for data cleaning, not only the data quality and availability can be improved, but also the efficiency of data cleaning can be improved.

[0106] Based on the model training method provided in the above embodiment, corresponding to the above model training method, refer to Figure 4 The flowchart shown in the figure, the embodiment of the present application also provides a data cleaning method, which can be applied to processors of various system platforms, computer terminals or various mobile devices, and the method includes the following steps: S401: Obtaining an original traffic data set.

[0107] Optionally, after obtaining the original traffic data set, the original traffic data set may be processed to be in a unified format to obtain a data set in a unified format; specifically, the unification of timestamp format (such as conversion to UTC time), the unification of data units (such as conversion of all traffic data into bytes / second), and the unification of data types (such as conversion of all numerical data into floating-point numbers).

[0108] S402: Input the original traffic data set into the trained integrated model to obtain a similar data set output by the integrated model.

[0109] Among them, the integrated model is trained by the model training method provided in the above embodiment. The specific implementation process of the model training method refers to the steps corresponding to S101 to S106 above, which will not be repeated here.

[0110] The similar data set is to delete or fill missing values ​​in the original traffic data set, and / or to format or replace abnormal values ​​in the original traffic data set.

[0111] In the present application, an integrated model is applied to identify missing values ​​and / or outliers in the original traffic data set, and when missing values ​​are identified, the missing values ​​are deleted or filled, and when outliers are identified, the outliers are replaced or formatted.

[0112] After the integrated model identifies the missing values ​​and / or outliers in the original traffic data set, the missing values ​​and / or outliers are marked. The integrated model identifies the missing values ​​and outliers in the traffic data set by applying the modified Z-score method to predict the missing values. The expression of the modified Z-score method is: ; Among them, X is the original data, is the median, is the median absolute deviation, and k is a constant (usually 1.4826). When |Z| exceeds a certain threshold (such as 3.5), the data is marked as missing values ​​or outliers. When the modified Z-score method is used to predict outliers and missing values, different thresholds can be set to distinguish outliers and missing values. For example, a first threshold and a second threshold are set. When |Z| exceeds the first threshold, the data point is marked as a missing value, and when |Z| exceeds the second threshold, the data point is marked as an outlier.

[0113] When missing values ​​are identified in the original traffic data, they can be processed by linear interpolation or spline interpolation. If they are long-term, a sequence-to-sequence model based on LSTM can be applied to fill in the missing values.

[0114] Optionally, the integrated model can also identify outliers in the original traffic data set based on anomaly detection reports, where the anomaly detection reports are obtained by executing the implementation process corresponding to step S103 in the above-mentioned model training method.

[0115] In the embodiment of the present application, after the integrated model outputs the similar data set, the similar data set can be further processed. The specific processing process is as follows: Smoothing the similar data set to obtain a smoothed data set; marking prominent data in the smoothed data set to obtain a marked data set; obtaining a data processing log corresponding to the marked data set; updating the difference information to the data processing log to obtain an updated data processing log, wherein the difference information is the difference information obtained by comparing the similar data set with the original traffic data set; based on the updated data processing log and the smoothed data set, obtaining a revised similar data set, wherein the revised similar data set is a smoothed data set that carries at least the data processing history record and the difference information in the updated data processing log.

[0116] The similar data sets are smoothed by applying moving average and wavelet transform. The moving average adopts the EWMA method. The formula corresponding to the EWMA method can refer to the expression sampled by applying EWMA to optimize the integrated model after obtaining the integrated model in the above embodiment, which will not be repeated here. The wavelet transform is a discrete wavelet transform, which can be specifically applied to the Daubechies wavelet basis function for multi-scale decomposition and reconstruction to remove high-frequency noise in the similar data sets.

[0117] Marking the prominent data in the smooth data set specifically involves performing abnormal clustering and threshold division on the smooth data set, and then identifying the prominent data in the smooth data set. The prominent data may be abnormal data in the smooth data set, or data that is significantly different from other data. The DBSCAN algorithm is used to perform density clustering on the smooth data set for abnormal clustering, and then an abnormal threshold is set based on the Mahalanobis distance. Each data in each cluster that exceeds the abnormal threshold in each cluster is a prominent data.

[0118] After marking the prominent data, obtain the data processing log, which contains the processing records of the prominent data after marking the prominent data. After marking the prominent data, conditional probability calculation and graphical model reasoning are performed on the prominent data to obtain causally adjusted data. The conditional probability calculation uses the Bayesian network to construct the probabilistic dependency relationship between variables. Graphical model reasoning uses causal discovery algorithms such as the PC algorithm to infer the causal direction between variables. Based on the inference results, the data is causally adjusted to obtain causally adjusted data. Calculate statistical indicators and consistency measures for the causally adjusted data. Statistical indicators include mean, variance, skewness, kurtosis, etc. The consistency measure uses Cronbach's α coefficient, and the calculation formula is: ; Where k is the number of items, is the variance of the ith item, is the total variance.

[0119] According to the above process of conditional probability calculation and graphical model reasoning for the prominent data in the labeled data set, as well as the process of calculating statistical indicators and consistency metrics, a data processing log for processing the labeled data set is generated. The data processing log includes a processing history of smoothing similar data sets, a processing history of processing the labeled data set, parameter settings, data sources, version numbers, processing method descriptions, timestamps, operators and other information.

[0120] While processing the labeled data set, the similar data set is compared with the original traffic data set to obtain the difference information between the two. The edit distance algorithm (Levenshtein) is applied to calculate the difference between the two. The calculation formula of Levenshtein is:

[0121] Among them, a and b are two data versions to be compared, and i and j are character positions.

[0122] After obtaining the data processing log and the difference information, the difference information is updated to the data processing log. At least the data processing history records and the difference information in the updated data processing log are added to the smoothed data set to obtain a corrected similar data set. The data processing history records include the processing history records of smoothing the similar data set and the processing history records of processing the marked data set.

[0123] In an embodiment of the present application, after applying the integrated model to clean the original traffic data set to obtain a similar data set, the similar data set can be further smoothed and the prominent data therein can be marked. According to the data processing history records and difference information, the corrected similar data set can be subsequently analyzed and traced.

[0124] Based on the same inventive concept of the above-mentioned model training method, a model training device 500 is also provided in the embodiment of the present application. Figure 5 , Figure 5 A device structure diagram of a model training device 500 provided in an embodiment of the present application, the model training device 500 can be applied to processors of various system platforms, computer terminals or various mobile devices, and the model training device 500 includes: an acquisition module 501, a clustering module 502, a detection module 503, a multi-feature extraction module 504, a feature fusion module 505, and a training module 506.

[0125] The acquisition module 501 is used to acquire a first data set, where the first data set is a flow data set obtained by extracting features according to a time series; A clustering module 502 is used to perform multi-scale clustering processing on the first data set to obtain a clustered data set; A detection module 503, configured to perform anomaly detection on the first data set to obtain an abnormal feature set; A multi-feature extraction module 504, configured to perform multi-feature extraction on the first data set according to a plurality of preset feature types, to obtain a target feature set corresponding to each of the feature types; A feature fusion module 505 is used to fuse the clustering data set, the abnormal feature set and each target feature set to obtain an enhanced feature set; The training module 506 is used to apply the enhanced feature set to train the integrated model to obtain a trained integrated model.

[0126] In an optional embodiment, the acquisition module 501 acquires the first data set, specifically for: Acquire an initial data set; perform time alignment and segmentation processing on the initial data set according to a time series; construct an index for the aligned and segmented initial data set; perform feature extraction on the indexed initial data set according to a time series to obtain a first data set.

[0127] In an optional embodiment, the acquisition module 501 extracts features from the indexed initial data set according to a time series to obtain a first data set, which is specifically used for: Decomposing the indexed initial data set to obtain a second data set; extracting statistical features of the second data set within a preset time window to obtain a third data set; extracting relevant features of the third data set based on the statistical features of the third data set to obtain a relevant feature set; extracting time-frequency features of the relevant feature set and / or the second data set to obtain a time-frequency feature set, wherein the time-frequency features are three-dimensional features used to characterize time-frequency-scale; extracting complexity features of the time-frequency feature set and / or the second data set to obtain a complex feature set, wherein the complexity features are used to characterize the degree of irregularity of the time series; and fusion of the complex feature set and / or the time-frequency feature set and / or the relevant feature set to obtain a first data set.

[0128] In an optional embodiment, the model training device 500 also includes a dimensionality reduction module; the dimensionality reduction module is used to perform dimensionality reduction processing on the first data set after performing feature fusion on the complex feature set and / or the time-frequency feature set and / or the related feature set in the acquisition module 501 to obtain the first data set.

[0129] The dimension reduction module performs dimension reduction processing on the first data set specifically for: The main features in the first data set are screened to obtain a screened feature set, wherein the main features are data features that have a high correlation with the target variable, and the target variable is a main traffic indicator used for prediction; an autoencoder is used to perform nonlinear dimensionality reduction processing on the screened feature set to obtain a nonlinear dimensionality reduction feature set; a high-order tensor decomposition is performed on the nonlinear dimensionality reduction feature set to obtain a structural feature set; the structural feature set is visualized to obtain a visualization feature set; the visualization feature set is clustered to obtain a plurality of similar feature groups; each of the similar feature groups is resampled to obtain a plurality of sample sets corresponding to each of the similar feature groups; for each sample set corresponding to the same similar feature group, a similarity coefficient of each feature in the similar feature group in different sample sets is calculated; the features whose similarity coefficients in each of the similar feature groups are lower than a preset threshold are filtered to obtain the first data set after dimensionality reduction.

[0130] In an optional embodiment, the detection module 503 performs anomaly detection on the first data set to obtain an abnormal feature set, which is specifically used for: Apply a preset local anomaly factor algorithm to perform anomaly detection on the first data set to obtain a local anomaly detection result; apply a preset seasonal autoregressive integrated moving average model to perform anomaly detection on the first data set to obtain a contextual anomaly detection result; apply a preset pruned exact linear time algorithm to perform anomaly detection on the first data set to obtain a structural change detection result; apply a preset variational autoencoder to perform anomaly detection on the first data set to obtain a reconstructed anomaly detection result; apply a preset online anomaly detection algorithm to perform anomaly detection on the first data set to obtain a dynamic anomaly detection result; perform evidence synthesis and uncertainty quantification processing on at least one of the local anomaly detection result, the contextual anomaly detection result, the structural change detection result, the reconstructed anomaly detection result and the dynamic anomaly detection result to obtain an anomaly detection report; perform feature extraction on the anomaly detection report according to the preset anomaly indicator to obtain an anomaly feature set.

[0131] In an optional embodiment, the training module 506 applies the enhanced feature set to train the integrated model to obtain a trained integrated model, which is specifically used for: The enhanced feature set is split into a training set, a test set and a validation set; the training set is used to construct a univariate model and a multivariate model; the validation set is used to perform grid search and time series cross-validation on the univariate model and the multivariate model, so as to optimize the model parameters of the univariate model and the multivariate model according to the validation results; the test set is used to perform performance testing on the optimized univariate model and the optimized multivariate model; if the performance test is passed, the optimized univariate model and the optimized multivariate model are synthesized to obtain a trained integrated model.

[0132] The model training device provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned model training method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned method embodiment.

[0133] Based on the same inventive concept of the above data cleaning method, the present application also provides a data cleaning device 600. Figure 6 , Figure 6 A device structure diagram of a data cleaning device 600 provided in an embodiment of the present application, the data cleaning device 600 can be applied to processors of various system platforms, computer terminals or various mobile devices, and the data cleaning device 600 includes: an original traffic data acquisition module 601 and a data cleaning module 602.

[0134] The original traffic data acquisition module 601 is used to acquire the original traffic data set; A data cleaning module 602 is used to input the original traffic data set into the trained integrated model to obtain a similar data set output by the integrated model, wherein the integrated model is trained using the above-mentioned model training method; The similar data set is to delete or fill missing values ​​in the original traffic data set, and / or to format or replace abnormal values ​​in the original traffic data set.

[0135] In an optional embodiment, the data cleaning device 600 further includes: an updating module, configured to: Smoothing the similar data set to obtain a smoothed data set; marking prominent data in the smoothed data set to obtain a marked data set; obtaining a data processing log corresponding to the marked data set; updating difference information to the data processing log to obtain an updated data processing log, wherein the difference information is difference information obtained by comparing the similar data set with the original traffic data set; based on the updated data processing log and the smoothed data set, obtaining a revised similar data set, wherein the revised similar data set is a smoothed data set that carries at least the data processing history record in the updated data processing log and the difference information.

[0136] The data cleaning device 600 provided in the embodiment of the present application has the same implementation principle and technical effects as those of the aforementioned data cleaning method embodiment. For the sake of brief description, for matters not mentioned in the device embodiment, reference may be made to the corresponding contents in the aforementioned data cleaning method embodiment. like Figure 7 As shown, Figure 7 The electronic device 700 provided in the embodiment of the present application is shown in a structural block diagram. The electronic device 700 includes: a transceiver 701 , a memory 702 , a communication bus 703 and a processor 704 . The transceiver 701, the memory 702, and the processor 704 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these components can be electrically connected to each other via one or more communication buses 703 or signal lines. The transceiver 701 is used to send and receive data. The memory 702 is used to store computer programs, such as storing Figure 5 or Figure 6 The software function module shown in, namely, the model training device 500 or the data cleaning device 600. Among them, the model training device 500 or the data cleaning device 600 includes at least one software function module that can be stored in the memory 702 in the form of software or firmware or solidified in the operating system (OS) of the electronic device 700. The processor 704 is used to execute the executable module stored in the memory 702, such as the software function module or computer program included in the model training device 500 or the data cleaning device 600. For example, the processor 704 is used to execute the model training method or data cleaning method provided in the above embodiment.

[0137] Among them, the memory 702 can be, but is not limited to, random access memory (Random Access Memory, RAM), read only memory (Read Only Memory, ROM), programmable read-only memory (Programmable Read-Only Memory, PROM), erasable programmable read-only memory (Erasable Programmable Read-Only Memory, EPROM), electrically erasable read-only memory (Electric Erasable Programmable Read-Only Memory, EEPROM), etc.

[0138] The processor 704 may be an integrated circuit chip with signal processing capabilities. The above-mentioned processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a graphics processor (GPU), an accelerated processing unit (Accelerated Processing Unit), a multimedia application processor (MAP), a microprocessor, etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The disclosed methods, steps and logic block diagrams in the embodiments of the present application may be implemented or executed. Alternatively, the processor 704 may also be any conventional processor, etc.

[0139] The electronic device 700 mentioned above includes but is not limited to mobile terminals such as mobile phones, laptop computers, tablet computers, etc., and fixed terminals such as digital TVs, desktop computers, etc.

[0140] An embodiment of the present application also provides a non-volatile computer-readable storage medium (hereinafter referred to as storage medium), on which a computer program is stored. When the computer program is run by a computer such as the above-mentioned electronic device 700, the model training method or data cleaning method provided in the above-mentioned embodiment is executed.

[0141] It should be noted that the various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0142] In several embodiments provided in the present application, it should be understood that the disclosed devices and methods can also be implemented in other ways. The device embodiments described above are merely schematic. For example, the flowcharts and block diagrams in the accompanying drawings show the possible architecture, functions and operations of the devices, methods and computer program products according to multiple embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, a program segment or a part of a code, and the module, a program segment or a part of a code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two consecutive boxes can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based system that performs a specified function or action, or can be implemented with a combination of dedicated hardware and computer instructions.

[0143] In addition, the functional modules in the various embodiments of the present application may be integrated together to form an independent part, or each module may exist separately, or two or more modules may be integrated to form an independent part.

[0144] If the functions are implemented in the form of software function modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a computer-readable storage medium, including several instructions to enable a computer device (which can be a personal computer, a laptop, a server, or an electronic device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned computer-readable storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0145] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. A model training method, characterized in that: include: Acquire a first data set, where the first data set is a flow data set obtained by extracting features according to a time series; Performing multi-scale clustering processing on the first data set to obtain a clustered data set; Performing anomaly detection on the first data set to obtain an abnormal feature set; Performing multi-feature extraction on the first data set according to a plurality of preset feature types respectively to obtain a target feature set corresponding to each of the feature types; Performing feature fusion on the clustering data set, the abnormal feature set and each of the target feature sets to obtain an enhanced feature set; The enhanced feature set is used to train the integrated model to obtain a trained integrated model.

2. The method according to claim 1, characterized in that The obtaining of the first data set includes: Get the initial data set; Performing time alignment and segmentation processing on the initial data set according to time series; constructing an index for the aligned and segmented initial data set; Feature extraction is performed on the indexed initial data set according to time series to obtain a first data set.

3. The method according to claim 2, characterized in that The step of extracting features from the indexed initial data set according to the time series to obtain a first data set includes: Decomposing the indexed initial data set to obtain a second data set; Extracting statistical features of the second data set within a preset time window to obtain a third data set; Based on the statistical features of the third data set, extract relevant features from the third data set to obtain a relevant feature set; Extracting time-frequency features from the relevant feature set and / or the second data set to obtain a time-frequency feature set, wherein the time-frequency feature is a three-dimensional feature used to characterize time-frequency-scale; Performing complexity feature extraction on the time-frequency feature set and / or the second data set to obtain a complex feature set, wherein the complexity feature is used to characterize the degree of irregularity of the time series; The complex feature set and / or the time-frequency feature set and / or the related feature set are subjected to feature fusion to obtain a first data set.

4. The method according to claim 2 or 3, characterized in that: After fusing the complex feature set and / or the time-frequency feature set and / or the related feature set to obtain the first data set, the method further includes: Performing dimensionality reduction processing on the first data set, wherein the performing dimensionality reduction processing on the first data set includes: Screening main features in the first data set to obtain a screened feature set, wherein the main features are data features that have a high correlation with a target variable, and the target variable is a main traffic indicator used for prediction; Applying an automatic encoder to perform nonlinear dimensionality reduction processing on the screening feature set to obtain a nonlinear dimensionality reduction feature set; Performing high-order tensor decomposition on the nonlinear dimensionality reduction feature set to obtain a structural feature set; Performing visualization processing on the structural feature set to obtain a visualization feature set; Clustering the visualization feature set to obtain multiple similar feature groups; Resampling each of the similar feature groups to obtain a plurality of sample sets corresponding to each of the similar feature groups; For each sample set corresponding to the same similar feature group, calculate the similarity coefficient of each feature in the similar feature group in different sample sets; The features whose similarity coefficients in each of the similar feature groups are lower than a preset threshold are filtered to obtain the first data set after dimensionality reduction.

5. The method according to claim 1, characterized in that: The performing anomaly detection on the first data set to obtain an abnormal feature set includes: Applying a preset local anomaly factor algorithm to perform anomaly detection on the first data set to obtain a local anomaly detection result; Applying a preset seasonal autoregressive integrated moving average model to perform anomaly detection on the first data set to obtain a contextual anomaly detection result; Applying a preset pruning exact linear time algorithm to perform anomaly detection on the first data set to obtain a structural change detection result; Applying a preset variational autoencoder to perform anomaly detection on the first data set to obtain a reconstructed anomaly detection result; Applying a preset online anomaly detection algorithm to perform anomaly detection on the first data set to obtain a dynamic anomaly detection result; Performing evidence synthesis and uncertainty quantification processing on at least one of the local anomaly detection result, the context anomaly detection result, the structural change detection result, the reconstruction anomaly detection result, and the dynamic anomaly detection result to obtain an anomaly detection report; Feature extraction is performed on the anomaly detection report according to preset anomaly indicators to obtain an anomaly feature set.

6. The method according to claim 1, characterized in that The applying the enhanced feature set to train the integrated model to obtain a trained integrated model includes: Splitting the enhanced feature set into a training set, a test set, and a validation set; Using the training set to construct univariate models and multivariate models; Applying the validation set to perform grid search and time series cross validation on the univariate model and the multivariate model, so as to optimize model parameters of the univariate model and the multivariate model according to the validation results; Using the test set to perform performance testing on the optimized univariate model and the optimized multivariate model; When the performance test is passed, the optimized univariate model and the optimized multivariate model are synthesized to obtain a trained integrated model.

7. A data cleaning method, characterized in that: include: Get the original traffic data set; Inputting the original traffic data set into the trained integrated model to obtain a similar data set output by the integrated model, wherein the integrated model is trained using the model training method according to any one of claims 1 to 6; The similar data set is to delete or fill missing values ​​in the original traffic data set, and / or to format or replace abnormal values ​​in the original traffic data set.

8. The method according to claim 7, characterized in that Also includes: Performing smoothing on the similar data set to obtain a smoothed data set; Marking the prominent data in the smoothed data set to obtain a marked data set; Get the data processing log corresponding to the labeled dataset; Updating the difference information to the data processing log to obtain the updated data processing log, wherein the difference information is the difference information obtained by comparing the similar data set with the original traffic data set; Based on the updated data processing log and the smoothed data set, the revised similar data set is obtained, and the revised similar data set is a smoothed data set that carries at least the data processing history records in the updated data processing log and the difference information.

9. A model training device, characterized in that: include: An acquisition module, used to acquire a first data set, where the first data set is a flow data set obtained by extracting features according to a time series; A clustering module, used for performing multi-scale clustering processing on the first data set to obtain a clustered data set; A detection module, used to perform anomaly detection on the first data set to obtain an abnormal feature set; A multi-feature extraction module, used to perform multi-feature extraction on the first data set according to a plurality of preset feature types, to obtain a target feature set corresponding to each of the feature types; A feature fusion module, used for fusing the clustering data set, the abnormal feature set and each of the target feature sets to obtain an enhanced feature set; The training module is used to apply the enhanced feature set to train the integrated model to obtain a trained integrated model.

10. An electronic device, characterized in that: include: A memory and a processor, wherein the processor is connected to the memory; The memory is used to store programs; The processor is used to call the program stored in the memory to execute the data training method as described in any one of claims 1 to 6, or to execute the data cleaning method as described in claim 7 or 8.

Images