High-information-retention preprocessing method and system for cold-rolled strip steel data based on hierarchical clustering and multi-stage robust threshold
Through the preprocessing method of hierarchical clustering and multi-level robust threshold, the problems of noise and outlier processing in cold-rolled strip data are solved, and higher information retention rate and model prediction accuracy are achieved. It is suitable for data preprocessing of cold-rolled strip production lines.
Patent Information
- Application Number
- CN202510867401.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-06-26
- Publication Date
- 2025-10-17
AI Technical Summary
Existing data preprocessing methods for cold-rolled strip steel are unstable and prone to misjudging local samples when dealing with noise and data imbalance, leading to information loss. Furthermore, they fail to distinguish between different types of outliers, affecting the accuracy of mechanical performance prediction models.
A preprocessing method based on hierarchical clustering and multi-level robust threshold is adopted. The data set is divided by hierarchical clustering, and the conservative correction threshold, significant anomaly threshold and extreme outlier threshold are dynamically calculated to accurately identify and process outliers and retain key data information.
It improves the stability and information retention rate of data preprocessing, enhances the accuracy and predictive performance of mechanical performance prediction models, and adapts to different types of outlier handling.
Smart Images
Figure CN120804743A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application provides a cold-rolled strip data high-information-retaining preprocessing method and system based on hierarchical clustering and multi-level robust threshold, and relates to the field of cold-rolled strip sample data preprocessing. BACKGROUND
[0002] A cold-rolled strip production line is relatively long, and there are many factors that affect the mechanical properties of the strip. Due to production environment fluctuations, process interference and measurement errors, etc., noise and data imbalance may occur in the collected data. If the collected data is not preprocessed before the establishment of a mechanical property prediction model, the training of the model will be disturbed, and the prediction model will be unstable and have low precision.
[0003] Traditional data preprocessing methods often use global fixed thresholds or mean-standard deviation to identify outliers, which not only has poor stability, but also easily causes misjudgment of local sample data and information loss. In addition, existing outlier processing methods are too single, and do not distinguish noise types: slight abnormalities caused by environmental fluctuations may contain effective process information, and direct removal will lose key data; and extreme outliers caused by measurement equipment failure may pollute the model. SUMMARY
[0004] Therefore, in order to make up for the shortcomings of the prior art, the application provides a cold-rolled strip data high-information-retaining preprocessing method and system based on hierarchical clustering and multi-level robust threshold, which can effectively avoid the defect that the global threshold is not sensitive to local data distribution and can accurately identify outliers.
[0005] The application provides a cold-rolled strip data high-information-retaining preprocessing method and system based on hierarchical clustering and multi-level robust threshold, which includes the following contents:
[0006] A cold-rolled strip data high-information-retaining preprocessing method based on hierarchical clustering and multi-level robust threshold, characterized by comprising the following steps:
[0007] Step S1: dividing the original data set into a training set and a test set;
[0008] Step S2: training the preprocessing model using the training set, wherein the hyperparameters of the preprocessing model include the hierarchical clustering cluster number n, the PCA dimension reduction dimension p, the minimum number of samples m allowed in the cluster, and the conservative correction threshold, the robust threshold coefficients z1, z2 and z3 of the significant abnormal threshold and the extreme outlier threshold;
[0009] Step S3: inputting the training set and the test set into the trained preprocessing model for data preprocessing;
[0010] Step S4: After the training set and the test set are respectively preprocessed, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output, and the prediction result is evaluated.
[0011] Further, step S2 includes the following contents:
[0012] Step S21: The training set is standardized and then dimensionality reduction is performed.
[0013] Step S22: The data after dimensionality reduction is classified by hierarchical clustering, including the following contents:
[0014] The dimensionality reduction dimension is determined by component analysis, and the cumulative variance contribution rate is ≥90%;
[0015] The number of clustering clusters is determined by the silhouette coefficient method, and the average silhouette coefficient of all data points is ≥0.4;
[0016] The minimum sample number is dynamically set according to the size of the data set, and the sample number of the cluster that is insufficient is oversampled to the set value m;
[0017] Further, step S2 further includes the following contents:
[0018] Step S23: The data after dimensionality reduction is classified by hierarchical clustering, and the cluster with a small sample is oversampled, including that the data with a small sample in the cluster obtained by hierarchical clustering is enhanced by random replication to enhance the stability of the cluster;
[0019] Step S24: The dynamic threshold value is obtained by robust statistics, that is, the threshold value corresponding to each cluster.
[0020] Further, step S2 further includes the following contents:
[0021] Step S25: Three threshold values are respectively obtained in each cluster, which are a conservative correction threshold value, a significant abnormal threshold value and an extreme outlier threshold value, including the following contents:
[0022]
[0023] AMAD i =1.4826×med({d i1 ,d i2 ,...,d iq})
[0024]
[0025] Wherein, i∈(1,2,3,...,n) represents the i-th cluster; j∈(1,2,3,...,q i ) represents the j-th sample in the cluster; q iN represents the number of samples in the cluster; X ij is the feature vector; is the cluster center; d ij is the median deviation; Y ij is the mechanical property; is the median of the mechanical property; AMAD i is the corrected median deviation, 1.4826 is the correction coefficient to make the corrected median deviation closer to the standard deviation of the traditional normal distribution; med(·) is the function of finding the median; T i1 is the normal sample region; T i2 is the correctable sample region; T i3 is the invalid sample region; and are the conservative correction threshold, the significant anomaly threshold and the extreme outlier threshold respectively obtained by robust statistics.
[0026] Step S26: After training, store the standardized model, the PCA dimension reduction model, the cluster center of each cluster and the conservative correction threshold, the significant anomaly threshold and the extreme outlier threshold trained by the training set.
[0027] Further, step S3 includes the following content:
[0028] Step S31: After the data set to be preprocessed (training set and test set) is processed by the standardized model and the dimension reduction model, the preprocessed sample is classified, and the classification calculation method includes the following content:
[0029]
[0030] cluster(X)=argmin i∈(1,2,...,n) D i
[0031] where X is the feature vector of the sample to be preprocessed; D i is the distance between the sample to be preprocessed and the i cluster center; cluster(X) is the cluster to which the sample X belongs; argmin(·) returns the minimum cluster number, that is, the shortest distance between the sample and the cluster center is the cluster to which the sample belongs.
[0032] Further, the sample classification is divided into the cluster with the shortest distance by calculating the Euclidean distance between the sample to be processed and each cluster center, and the processing method of the sample is decided by comparing with the threshold value.
[0033] Further, step S3 further includes the following content:
[0034] Step S32: After the samples are classified, the mechanical properties of the samples are compared with the conservative correction threshold, the significant abnormality threshold and the extreme outlier threshold respectively; if the mechanical properties of the samples are within the significant abnormality threshold, they are judged as normal samples and no processing is performed; if the mechanical properties of the samples are outside the significant abnormality threshold and within the extreme outlier threshold, they are judged as correctable samples, and the abnormal values exceeding the upper limit of the significant abnormality threshold are corrected to the upper limit of the conservative correction threshold, and the abnormal values exceeding the lower limit of the significant abnormality threshold are corrected to the lower limit of the conservative correction threshold; if the mechanical properties of the samples are outside the extreme outlier threshold, they are judged as invalid samples and are eliminated, and finally the preprocessed data is output.
[0035] Furthermore, step S4 includes the following contents:
[0036] Step S41: After the training set and the test set are subjected to data preprocessing respectively, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output and the prediction result is evaluated.
[0037] According to a second aspect of the present invention, a high-information-retention preprocessing system for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds is characterized by comprising the following modules:
[0038] Data partitioning module: used to divide the training set and test set;
[0039] Standardization and Dimensionality Reduction Module: performs standardization and PCA dimensionality reduction according to information retention rate;
[0040] Clustering and oversampling module: implements hierarchical clustering and oversampling according to the set number of clusters n and the minimum number of samples allowed in a cluster m;
[0041] Threshold calculation module: dynamically generates three-level thresholds for each cluster;
[0042] Abnormal processing module: correct or eliminate abnormal samples based on threshold comparison results;
[0043] Output module: outputs preprocessed data.
[0044] According to the third aspect of the present invention, a high-information-retention preprocessing system for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds includes an electronic device, wherein the electronic device includes a memory, a processor, and a computer program stored in the memory and runnable on the processor, and is characterized in that when the processor executes the computer program, it implements a high-information-retention preprocessing method for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds as described in any one of the present invention.
[0045] According to a fourth aspect of the present application, a cold strip data high information retention preprocessing system based on hierarchical clustering and multi-level robust threshold value comprises a computer readable storage medium, and the computer readable storage medium stores a computer program, wherein the computer program is executed by a processor to implement a cold strip data high information retention preprocessing method based on hierarchical clustering and multi-level robust threshold value according to any one of the present application.
[0046] The present application has the following advantages:
[0047] The present application provides a cold strip data high information retention preprocessing method and system based on hierarchical clustering and multi-level robust threshold value. Small sample oversampling makes the calculation of sample cluster center more stable, and can ensure sufficient sample support when calculating the threshold value; dynamic threshold value design avoids the defect that global threshold value is not sensitive to local data distribution, and can more accurately identify abnormal values; setting hierarchical threshold value abnormal value correction and rejection strategy, taking into account data quality and information utilization rate; using robust statistics to calculate the threshold value, which has low sensitivity to noise and strong stability. BRIEF DESCRIPTION OF DRAWINGS
[0048] Figure 1 The data preprocessing and prediction flowchart of the present application.
[0049] Figure 2 The hierarchical clustering and robust statistical data preprocessing model packaging diagram of the present application.
[0050] Figure 3 The mechanical property hierarchical threshold value region division schematic diagram of the present application.
[0051] Figure 4 The yield strength data processing Gaussian kernel distribution curve diagram of the present application.
[0052] Figure 5 The tensile strength data processing Gaussian kernel distribution curve diagram of the present application.
[0053] Figure 6 The fracture elongation data processing Gaussian kernel distribution curve diagram of the present application.
[0054] Figure 7 The yield strength test set real value and predicted value scatter plot after data processing of the present application.
[0055] Figure 8 The tensile strength test set real value and predicted value scatter plot after data processing of the present application.
[0056] Figure 9 The fracture elongation test set real value and predicted value scatter plot after data processing of the present application. DETAILED DESCRIPTION
[0057] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings.
[0058] It should be noted that the following detailed descriptions are illustrative and intended to provide further explanation of the present application. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which the present application belongs.
[0059] It should be noted that the terms used herein are only for describing specific embodiments and are not intended to limit the exemplary embodiments according to the present application; as used herein, unless the context clearly indicates otherwise, the singular form is also intended to include the plural form, and it should be understood that when the terms "comprise" and / or "include" are used in this specification, they indicate the presence of features, steps, operations, devices, components and / or their combinations.
[0060] like Figure 1 、 Figure 2 As shown, the present invention proposes a charging guidance method and system for heterogeneous electric vehicle users oriented to the interests of multiple subjects, wherein: Figure 1 The data preprocessing steps are shown in Figure 2 The internal structure of the data preprocessing model is shown, including the following:
[0061] A high-information-retention preprocessing method for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds, characterized by comprising the following steps:
[0062] Step S1: Divide the original data set into training set and test set;
[0063] Step S2: Use the training set to train the preprocessing model. The hyperparameters of the preprocessing model include the number of hierarchical clustering clusters (n), the PCA dimension reduction dimension (p), the minimum number of samples allowed in a cluster (m), and the robust threshold coefficients (z1, z2, z3) of the conservative correction threshold, the significant anomaly threshold, and the extreme outlier threshold;
[0064] Step S3: input the training set and test set into the trained preprocessing model respectively to perform data preprocessing;
[0065] Step S4: After the training set and the test set are subjected to data preprocessing respectively, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output and the prediction result is evaluated.
[0066] Furthermore, step S2 includes the following contents:
[0067] Step S21: standardize the training set and then perform dimensionality reduction;
[0068] Step S22: classifying the data after dimension reduction by hierarchical clustering, including the following:
[0069] The dimension reduction dimension is determined by component analysis, and the cumulative variance contribution rate is greater than or equal to 90%;
[0070] The number of clustering clusters is determined by the contour coefficient method, and the average contour coefficient of all data points is greater than or equal to 0.4;
[0071] The minimum sample number is dynamically set according to the size of the data set, and the sample number of the cluster is insufficient. The sample number is supplemented to the set value m by oversampling;
[0072] Further, step S2 further includes the following:
[0073] Step S23: classifying the data after dimension reduction by hierarchical clustering, and oversampling the small sample, including enhancing the stability of the small sample by randomly duplicating the data containing a small number of samples in the class obtained by hierarchical clustering;
[0074] Step S24: obtaining a dynamic threshold value, i.e. a threshold value corresponding to each cluster, by robust statistics;
[0075] Further, step S2 further includes the following:
[0076] Step S25: three threshold values are respectively obtained in each cluster, which are conservative correction threshold value, significant abnormal threshold value and extreme outlier threshold value, including the following:
[0077]
[0078] AMAD i = 1.4826 x med({d i1 , d i2 ,..., d iq})
[0079]
[0080] Wherein, i∈(1,2,3,...,n) represents the i-th cluster; j∈(1,2,3,...,q i ) represents the j-th sample in the cluster; q i represents the number of samples in the cluster; X ij is the feature vector; is the cluster center; d ij is the median deviation; Y ij is the mechanical property; is the median of the mechanical property; AMAD i1.4826 is the correction coefficient to make the corrected median deviation closer to the standard deviation of the traditional normal distribution; med(·) is the function of finding the median; T i1 is the normal sample area; T i2 is the correctable sample area; T i3 is the invalid sample area; and are the conservative correction threshold, the significant anomaly threshold and the extreme outlier threshold respectively obtained by robust statistics.
[0081] Step S26: After training, store the standardized model, PCA dimension reduction model, cluster center of each cluster and conservative correction threshold, significant anomaly threshold and extreme outlier threshold trained by the training set.
[0082] Further, step S3 includes the following contents:
[0083] Step S31: After the data set to be preprocessed (including the training set and the test set) is processed by the standardized model and the dimension reduction model, the preprocessed sample is classified, and the calculation method of classification includes the following contents:
[0084]
[0085] cluster(X)=argmin i∈(1,2,...,n) D i
[0086] Where X is the feature vector of the sample to be preprocessed; D i is the distance between the sample to be preprocessed and the i cluster center; cluster(X) is the cluster to which the X sample belongs; argmin(·) returns the minimum cluster number, that is, the shortest distance between the sample and the cluster center is the cluster to which the sample belongs.
[0087] Further, the sample classification is performed by calculating the Euclidean distance between the sample to be processed and each cluster center, and the sample is assigned to the cluster with the smallest distance to obtain the threshold of each level of the cluster, and the processing method of the sample is decided by comparing with the threshold.
[0088] Further, step S3 further includes the following contents:
[0089] Step S32: After the sample is classified, the mechanical properties of the sample are compared with the conservative correction threshold, the significant abnormal threshold and the extreme outlier threshold respectively; if the mechanical properties of the sample are within the significant abnormal threshold range, the sample is determined as a normal sample and is not processed; if the mechanical properties of the sample are outside the significant abnormal threshold and within the extreme outlier threshold, the sample is determined as a correctable sample, the abnormal value exceeding the upper limit of the significant abnormal threshold is corrected to the upper limit of the conservative correction threshold, and the abnormal value exceeding the lower limit of the significant abnormal threshold is corrected to the lower limit of the conservative correction threshold; if the mechanical properties of the sample are outside the extreme outlier threshold, the sample is determined as an invalid sample and is rejected, and finally the preprocessed data is output.
[0090] Further, step S4 includes the following contents:
[0091] Step S41: After the training set and the test set are preprocessed respectively, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output, and the prediction result is evaluated.
[0092] According to the second aspect of the present application, a cold-rolled strip data high-information-retention preprocessing system based on hierarchical clustering and multi-level robust threshold is provided, characterized in that it comprises the following modules:
[0093] A data division module is used to divide the training set and the test set;
[0094] A standardization and dimension reduction module is used to perform standardization and PCA dimension reduction according to the information retention rate;
[0095] A clustering and oversampling module is used to realize hierarchical clustering and oversampling according to the set cluster number (n) and the minimum number of samples allowed in the cluster (m);
[0096] A threshold calculation module is used to dynamically generate three-level thresholds for each cluster;
[0097] An abnormality processing module is used to correct or reject abnormal samples according to the threshold comparison result;
[0098] An output module is used to output the preprocessed data.
[0099] According to the third aspect of the present application, a cold-rolled strip data high-information-retention preprocessing system based on hierarchical clustering and multi-level robust threshold is provided, which comprises an electronic device, wherein the electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, it realizes a cold-rolled strip data high-information-retention preprocessing method based on hierarchical clustering and multi-level robust threshold according to any one of the present application.
[0100] According to a fourth aspect of the present application, a high information retention preprocessing system for cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold comprises a computer readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements a high information retention preprocessing method for cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to any one of the present application.
[0101] In addition to the above, the present application also includes the following embodiments:
[0102] In an embodiment of the present application, step S25 comprises the following:
[0103] Step S25: three thresholds are respectively calculated in each cluster, including a conservative correction threshold, a significant abnormal threshold and an extreme outlier threshold, comprising the following:
[0104]
[0105] AMAD i = 1.4826 * med({d i1 , d i2 ,..., d iq})
[0106]
[0107] Wherein, i∈(1,2,3,...,n) represents the i-th cluster; j∈(1,2,3,...,q i ) represents the j-th sample in the cluster; q i represents the number of samples in the cluster; X ij is the feature vector; is the cluster center; d ij is the median deviation; Y ij is the mechanical property; is the median of the mechanical property; AMAD i is the corrected median deviation, and 1.4826 is the correction coefficient to make the corrected median deviation closer to the standard deviation of the traditional normal distribution; med(·) is the function of the median; T i1 is the normal sample region; T i2 is the correctable sample region; T i3 is the invalid sample region; and are respectively the conservative correction threshold, the significant abnormal threshold and the extreme outlier threshold obtained by robust statistics.
[0108] Further, in an embodiment of the present application, the related parameters include the following:
[0109] Conservative correction threshold: used for correcting the slightly abnormal data that can be reserved;
[0110] Significant abnormal threshold: used for determining the correctable samples affected by process environment and other disturbances;
[0111] Extremely outlier threshold: used for eliminating invalid data caused by measurement equipment failure and the like.
[0112] Further, in an embodiment of the present application, z1∈[0, 2], z2∈[2, 3], z3∈[3, 4], and z1 < z2 < z3; z1 is a robust threshold coefficient of the conservative correction threshold, and its size indicates the degree of conservative correction; when z1 = 0, it indicates the lowest degree of reserving information of the corrected sample; when z1 = 2, it indicates the maximum degree of reserving information of the corrected sample; z2 is a robust threshold coefficient of the significant abnormal threshold; when z2 = 2, it indicates a loose determination of the correctable sample, and allows more process fluctuation data to enter the correction process; when z2 = 3, it indicates a strict determination of the correctable sample, and only the data that is obviously abnormal but not outlier is corrected; z3 is a robust threshold coefficient of the extremely outlier threshold; when z3 = 3, it indicates a conservative elimination of outlier, and reserves potentially valid but dispersed data; when z3 = 4, it indicates an aggressive elimination of outlier, and only the absolutely invalid data such as detection equipment failure is eliminated.
[0113] In an embodiment of the present application, step S32 includes the following contents:
[0114] As Figure 3 If the mechanical property of the sample is within the significant abnormal threshold range and is determined as a normal sample, no processing is performed; if the mechanical property of the sample is outside the significant abnormal threshold range and within the extremely outlier threshold range, the sample is determined as a correctable sample; the abnormal value exceeding the upper limit of the significant abnormal threshold is corrected to the upper limit of the conservative correction threshold, and the abnormal value exceeding the lower limit of the significant abnormal threshold is corrected to the lower limit of the conservative correction threshold; if the mechanical property of the sample is outside the extremely outlier threshold range, the sample is determined as an invalid sample and is eliminated.
[0115] In an embodiment of the present application, step S4 includes the following contents:
[0116] After the training set and the test set are respectively subjected to data preprocessing, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output, and the prediction result is evaluated.
[0117] For the 5459 data from the production performance, the hierarchical clustering-multilevel robust threshold algorithm is used for preprocessing, the hierarchical clustering cluster number (n) is 900; the PCA dimension reduction dimension (p) is 18; the minimum number of samples allowed in the cluster (m) is 8; the robust threshold coefficients of the conservative correction threshold, the significant anomaly threshold and the extreme outlier threshold (z1, z2, z3) are 2, 3 and 3.5 respectively. After processing, the yield strength, tensile strength and fracture elongation data sets are 4984, 5044 and 5009 respectively, and the corrected data accounts for 1.8%, 1.9% and 1.6% of the total data respectively. Isolation forest as a contrast model, after processing, there are 4459 data left.
[0118] In an embodiment of the present application, Figure 4 The Gaussian kernel distribution curves of the yield strength data processing in the mechanical properties of the original data, the isolation forest and the hierarchical clustering-multilevel robust threshold processing are shown.
[0119] Figure 5 The Gaussian kernel distribution curves of the tensile strength data processing in the mechanical properties of the original data, the isolation forest and the hierarchical clustering-multilevel robust threshold processing are shown.
[0120] Figure 6 The Gaussian kernel distribution curves of the fracture elongation data processing in the mechanical properties of the original data, the isolation forest and the hierarchical clustering-multilevel robust threshold processing are shown.
[0121] The results show that in terms of yield strength and tensile strength, the data distribution after hierarchical clustering-multilevel robust threshold processing is closer to the original data, and the information retention is higher; in terms of fracture elongation, the data distribution of the two processing methods is relatively close.
[0122] In an embodiment of the present application, according to the results in Table 1, the model trained by using the hierarchical clustering-multilevel robust threshold algorithm for preprocessing data shows the best performance: within the allowable error range (yield strength ± 35 MPa, tensile strength ± 30 MPa, and elongation after fracture ± 0.9%), the hit rate (AHR) reaches 93.72%, 93.27% and 81.24% respectively, ranking first in all comparison data sets; at the same time, the RMSE and MAE values of each mechanical property index are also lower than those of other data sets. It is worth noting that although the data distribution of the fracture elongation rate is similar to the isolation forest processing result, the model trained by this method still shows higher prediction accuracy. These results show that this preprocessing method not only effectively guarantees the data information, but also significantly improves the prediction performance and industrial applicability of the model. Figure 7 、 Figure 8 and Figure 9The scatter diagram of the predicted value and the actual value of the yield strength, the tensile strength and the elongation after fracture after the pretreatment method is shown.
[0123]
[0124] Table 1 Comparison of prediction accuracy under different pretreatments based on XGboost
[0125] The above is the preferred embodiment of the present application, any changes made according to the technical solutions of the present application, as long as the generated function does not exceed the scope of the technical solutions of the present application, belongs to the protection scope of the present application.
Claims
1. A high information retention preprocessing method for cold-rolled strip data based on hierarchical clustering and multi-level robust threshold, characterized in that: The following steps are involved: Step S1: Divide the original data set into training set and test set; Step S2: Use the training set to train the preprocessing model. The hyperparameters of the preprocessing model include the number of hierarchical clustering clusters n, the PCA dimension reduction dimension p, the minimum number of samples allowed in a cluster m, and the robust threshold coefficients z1, z2, and z3 of the conservative correction threshold, the significant anomaly threshold, and the extreme outlier threshold; Step S3: input the training set and test set into the trained preprocessing model respectively to perform data preprocessing; Step S4: After the training set and the test set are subjected to data preprocessing respectively, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output and the prediction result is evaluated.
2. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 1, characterized in that: Step S2 includes the following contents: Step S21: standardize the training set and then perform dimensionality reduction; Step S22: Classify the data after dimensionality reduction using hierarchical clustering, including the following: Determine the dimension reduction dimension through component analysis and meet the cumulative variance contribution rate ≥ 90%; The silhouette coefficient method is used to determine the number of clusters, requiring the average silhouette coefficient of all data points to be ≥0.4; The minimum number of samples is dynamically set according to the size of the data set, and clusters with insufficient sample size are oversampled to make up to the set value m.
3. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 2, characterized in that: Step S2 also includes the following: Step S23: using hierarchical clustering to classify the data after dimensionality reduction, and oversampling the clusters with small samples, including randomly replicating the data with a small number of samples in the clusters obtained by hierarchical clustering to enhance the stability of the clusters; Step S24: Use robust statistics to find the dynamic threshold, that is, the threshold corresponding to each cluster.
4. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 3, characterized in that: Step S2 also includes the following: Step S25: Three thresholds are calculated for each cluster, namely, the conservative correction threshold, the significant anomaly threshold coefficient, and the extreme outlier threshold, including the following: AMAD i =1.4826×med({d i1 ,d i2 ,...,d iq }) Among them, i∈(1,2,3,...,n) represents the i-th cluster; j∈(1,2,3,...,q i ) represents the jth sample in the cluster; q i represents the number of samples in the cluster; X ij is the eigenvector; is the cluster center; d ij is the median deviation; Y ij For mechanical properties; is the median of mechanical properties; AMAD i To correct the median deviation, 1.4826 is the correction coefficient to make the corrected median deviation closer to the standard deviation of the traditional normal distribution; med(·) is the function for finding the median; T i1 is the normal sample area; T i2 is the correctable sample area; T i3 It is the invalid sample area; and They are the conservative correction threshold, significant anomaly threshold and extreme outlier threshold obtained by robust statistics respectively; Step S26: After training, the standardized model, PCA dimensionality reduction model, cluster center of each cluster, and conservative correction threshold, significant anomaly threshold, and extreme outlier threshold trained by the training set are stored.
5. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 1, characterized in that: Step S3 includes the following contents: Step S31: After the data set to be preprocessed is processed by the standardization model and the dimensionality reduction model, the samples to be preprocessed are classified. The calculation method of the classification includes the following: cluster(X)=argmin i∈(1,2,...,n) D i Where X is the feature vector of the sample to be preprocessed; D i is the distance between the sample to be preprocessed and the center of cluster i; cluster(X) is the cluster to which the sample X belongs; argmin(·) returns the cluster number with the minimum distance, that is, the cluster to which the sample belongs is the one with the shortest distance from the cluster center; Furthermore, sample classification is performed by calculating the Euclidean distance between the sample to be processed and the center of each cluster, assigning the sample to the cluster with the smallest distance, obtaining the thresholds of each level of the cluster to which it belongs, and deciding how to process the sample by comparing with the thresholds.
6. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 5, characterized in that: Step S3 also includes the following: Step S32: After the samples are classified, the mechanical properties of the samples are compared with the conservative correction threshold, the significant abnormality threshold and the extreme outlier threshold respectively; if the mechanical properties of the samples are within the significant abnormality threshold, they are judged as normal samples and no processing is performed; if the mechanical properties of the samples are outside the significant abnormality threshold and within the extreme outlier threshold, they are judged as correctable samples, and the abnormal values exceeding the upper limit of the significant abnormality threshold are corrected to the upper limit of the conservative correction threshold, and the abnormal values exceeding the lower limit of the significant abnormality threshold are corrected to the lower limit of the conservative correction threshold; if the mechanical properties of the samples are outside the extreme outlier threshold, they are judged as invalid samples and are eliminated, and finally the pre-processed data is output.
7. The method for high information retention preprocessing of cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold according to claim 1, characterized in that: Step S4 includes the following contents: Step S41: After the training set and the test set are subjected to data preprocessing respectively, the training set is used to train the mechanical property prediction model, the test set is input into the mechanical property prediction model, the predicted value is output and the prediction result is evaluated.
8. A high information retention preprocessing system for cold-rolled strip steel data based on hierarchical clustering and multi-level robust threshold, characterized in that: Includes the following modules: Data partitioning module: used to divide the training set and test set; Standardization and Dimensionality Reduction Module: performs standardization and PCA dimensionality reduction according to information retention rate; Clustering and oversampling module: implements hierarchical clustering and oversampling according to the set number of clusters n and the minimum number of samples allowed in a cluster m; Threshold calculation module: dynamically generates three-level thresholds for each cluster; Abnormal processing module: correct or eliminate abnormal samples based on threshold comparison results; Output module: outputs preprocessed data.
9. A high-information-retention preprocessing system for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds, comprising an electronic device, wherein the electronic device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, it implements a high-information-retention preprocessing method for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds as described in any one of claims 1 to 7.
10. A high information retention preprocessing system for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds, comprising a computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the computer program implements a high-information-retention preprocessing method for cold-rolled strip data based on hierarchical clustering and multi-level robust thresholds as described in any one of claims 1 to 7.