Multi-coal-source coal blending data processing method based on WSMOTE algorithm

Abstract

The invention discloses a multi-coal-source coal blending data processing method based on a WSMOTE algorithm, and belongs to the technical field of coal processing data processing, and the method comprises the steps: data collection and classification: collecting multi-dimensional data of a raw coal floating and sinking test, a coal blending test and the like, carrying out the preprocessing, dividing the data into a missing data set and a complete data set, carrying out the interpolation optimization of a missing value, and carrying out the calculation of the complete data set. And carrying out adaptive processing according to data distribution types, or carrying out weighted combination normalization according to feature importance, analyzing the number and distribution characteristics of minority class samples, dynamically adjusting the neighbor sample size and the new sample synthesis amount, and completing data enhancement and standardization. According to the method, standardized parameters are dynamically adjusted to adapt to data distribution changes, abnormal values are accurately processed, the combined interpolation model is subjected to weight optimization, errors are reduced, and data integrity is improved. WSMOTE parameters are adaptively combined with data distribution to synthesize samples, data are balanced, and the over-fitting risk is reduced. In practical application, the quality of coal blending model training data can be improved, and optimization of a multi-coal-source coal blending scheme is assisted.

Description

A Multi-Source Coal Blending Data Processing Method Based on WSMOTE Algorithm Technical Field

[0001] This invention belongs to the field of coal processing data processing technology, specifically relating to a multi-coal source blending data processing method based on the WSMOTE algorithm. Background Technology

[0002] Coal, as a crucial component of my country's energy consumption, plays a vital role in economic development. Multi-source coal blending technology can rationally utilize coal resources of different types and qualities, reduce reliance on high-quality coking coal, decrease pollutant emissions, and improve coke quality, making it an important pathway to achieving clean and efficient coal utilization. However, the refining effect of multi-source coal blending is affected by various factors such as raw coal composition and moisture content during the refining process, resulting in coal preparation plants often collecting small amounts of blending test data with uneven distribution.

[0003] In existing technologies, the standardization of multi-coal source blending data mainly relies on traditional resampling methods (such as SMOTE) to handle imbalanced data, combined with machine learning models for analysis. However, traditional resampling methods are prone to overfitting, resulting in low-quality synthesized samples. Furthermore, the SMOTE method suffers from blindness in synthesizing samples, failing to fully consider the distribution characteristics of data near minority class samples, and generating low-quality synthesized samples when minority class samples are sparse. Summary of the Invention

[0004] The purpose of this invention is to provide a multi-coal source coal blending data processing method based on the WSMOTE algorithm to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multi-coal source blending data processing method based on the WSMOTE algorithm, comprising the following steps:

[0006] S1. Data Acquisition and Classification: Collect data related to coal blending from multiple coal sources, preprocess the collected data, and filter out missing and complete datasets.

[0007] S2, Missing Value Imputation Optimization: Based on the complete dataset, a new missing dataset is constructed by randomly generating a preset proportion of missing items. The new missing dataset is then imputed using random forest imputation algorithm and regression imputation algorithm. The weights are dynamically adjusted or optimized through cross-validation or evaluation of actual application effects. Finally, the combined model with optimized weights is used to impute the missing items in the original missing dataset to obtain the final imputed values.

[0008] S3. Adaptive Data Normalization: Analyze the distribution type of each data feature. For features that are approximately normally distributed, use Z-score standardization. For features that are skewed, use logarithmic transformation or Box-Cox transformation before normalization. Alternatively, assign different weights to different normalization methods according to the importance of the features and combine them for normalization.

[0009] S4. Adaptive adjustment of WSMOTE algorithm parameters: Analyze the sample size and distribution characteristics of each minority class in the dataset, and then dynamically adjust the nearest neighbor sample size parameter and / or the number of new samples to be synthesized for each minority class sample based on the analysis results. Fit the distribution characteristics of the imputed dataset, randomly generate a large number of data as majority class samples, use the original imputed dataset as minority class samples, synthesize new samples through the WSMOTE algorithm based on the adjusted parameters, merge the minority class samples with the new samples to obtain the enhanced dataset, calculate the mean and standard deviation of each class of data in the enhanced dataset, and standardize the raw coal data of each group to complete the data standardization in the multi-coal source blending process.

[0010] S5. Model Training and Evaluation: The coal blending data processing model is trained using the enhanced dataset. Accuracy and F-value are used to evaluate the processing effect of categorical data, and the root mean square error is used to evaluate the processing effect of continuous data.

[0011] Preferably, in the data preprocessing stage, the standardization parameters are dynamically adjusted based on the results of data imputation and normalization, or conditions are set to standardize only the features that still have significant outliers or outliers after imputation or normalization, while retaining the original processing results for features with stable distribution and no outliers.

[0012] Preferably, in S1, the relevant data includes raw coal float-sink test data and coal blending test data, covering the yield, ash content and coal quality composition information of different raw coals at various sorting densities;

[0013] The missing dataset refers to the data rows containing missing data items, while the rest are complete datasets;

[0014] The preprocessing includes missing value identification and data format conversion.

[0015] Preferably, in S2, the preset ratio is 10%, and 10% of the data rows of the complete dataset are randomly extracted by row coding and replaced with missing value identifiers to construct a new missing dataset.

[0016] Preferably, the specific process of the random forest imputation algorithm is as follows:

[0017] Step 1: Use baseline methods to perform preliminary imputation on the new missing dataset. For categorical variables, use mode imputation, and for continuous variables, use median imputation of adjacent data.

[0018] Step 2: Construct a random forest based on the data after preliminary imputation, and generate a similarity matrix by the frequency of samples falling into the same leaf node in the random forest;

[0019] Step 3: Perform weighted imputation based on the similarity matrix. For categorical variables, select the category with the highest similarity as the imputation value. For continuous variables, use the weighted average of the similarity of the non-missing data as the imputation value.

[0020] Step 4: Repeat the above steps of building a random forest and interpolation until the preset stopping condition is met.

[0021] Preferably, the regression imputation algorithm uses a linear regression model, estimates the model parameters using the least squares method, and calculates the imputed values ​​for missing values ​​based on the intrinsic relationship between the independent and dependent variables.

[0022] Preferably, in S3, the Z-score standardized calculation formula is as follows: ,in For the original data of the i-th sample in the j-th category, and These are the mean and standard deviation of the j-th category of data, respectively. This is the standardized data.

[0023] Preferably, in S4, the nearest neighbor sample size parameter and the new sample size are dynamically adjusted according to the minority class sample size and distribution characteristics. The calculation formula for synthesizing new samples by the WSMOTE algorithm is: Xnew[j]=Xi[j]+ωj×random(0,1)×(Yi[j]-Xi[j]), where Xi[j] is the j-th class data of the minority class sample, Yi[j] is the j-th class data of the nearest neighbor sample, ωj is the entropy weight of the j-th class, and random(0,1) is a random number in the interval 0-1.

[0024] Preferably, in S5, Accuracy = (TP + TN) / (TP + TN + FP + FN), Recall = TP / (TP + FN), Precision = TP / (TP + FP), where TP represents the number of samples that actually belong to and are correctly predicted as the minority class, TN represents the number of samples that actually belong to and are correctly predicted as the majority class, FP represents the number of samples that actually belong to the majority class but are incorrectly predicted as the minority class, and FN represents the number of samples that actually belong to the minority class but are incorrectly predicted as the majority class. The evaluation metric F = Recall × Precision.

[0025] The formula for calculating the root mean square error is: , where y[j] is the j-th data in the augmented dataset, ynew[j] is the j-th data in the synthetic sample, and N is the number of samples after data augmentation.

[0026] Preferably, the process of dynamically adjusting the standardization parameters is as follows: if there are significant outliers in the imputed data, the calculation range of the standardized mean and standard deviation is limited to the data after removing outliers; if the data distribution changes due to imputation or normalization, the standardization method is switched.

[0027] Compared with the prior art, the beneficial effects of the present invention are:

[0028] 1. This invention, by dynamically adjusting standardization parameters, can flexibly adjust key parameters in the standardization process according to the actual distribution of data, effectively identifying and handling outliers in the data, and avoiding their adverse effects on model training. Simultaneously, by balancing the differences between data from different coal sources, it ensures that various types of data play a balanced role in model training, thereby significantly improving the overall quality of coal blending model training data and laying a solid foundation for subsequently building accurate and reliable coal blending models.

[0029] 2. This invention calibrates the weights of random forest and regression imputation by constructing a new missing dataset, combining the models to adapt to different types of missing data, resulting in lower imputation errors and improved integrity of the original data. First, the characteristics of different types of missing data are analyzed, and the initial weight ratios of random forest and regression imputation in the new missing dataset are determined accordingly. During the construction of the new missing dataset, the characteristic information of various data types is fully utilized to meticulously classify data with different missing patterns. Through extensive experiments and data analysis, the weight allocation is continuously optimized, enabling the combined model to accurately adapt to different types of missing data. Practical verification shows that compared to a single imputation method, this combined model significantly reduces imputation errors, maximizes the recovery of the true information of the original data, effectively improves the integrity of the original data, and provides a more reliable data foundation for subsequent coal blending data processing.

[0030] 3. This invention addresses the data imbalance problem and reduces the risk of overfitting by adaptively adjusting the parameters of the WSMOTE algorithm and synthesizing new samples based on the distribution characteristics of minority class samples. Specifically, the WSMOTE algorithm automatically and intelligently adjusts the parameters for generating new samples based on the distribution density and distance of minority class samples in the feature space. Instead of using fixed parameter settings, it dynamically determines the generation location and feature values ​​of new samples based on the actual data distribution. This results in new samples that not only effectively supplement the minority class samples in quantity but also more closely resemble the actual minority class samples in feature distribution, greatly enriching the diversity of minority class samples. This approach effectively solves the data imbalance problem in multi-coal source blending data, making the influence of various data types more balanced during model training, thereby reducing the risk of model overfitting and improving the model's generalization ability to different coal source data. Attached Figure Description

[0031] Figure 1 is a flowchart of the present invention. Detailed Implementation

[0032] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0033] Example 1

[0034] Please refer to Figure 1. This invention provides a multi-coal source blending data processing method based on the WSMOTE algorithm, comprising the following steps:

[0035] S1. Data Acquisition and Classification: Collect data related to coal blending from multiple coal sources, preprocess the collected data, and filter out missing and complete datasets.

[0036] S2, Missing Value Imputation Optimization: Based on the complete dataset, a new missing dataset is constructed by randomly generating a preset proportion of missing items. The new missing dataset is then imputed using random forest imputation algorithm and regression imputation algorithm. The weights are dynamically adjusted or optimized through cross-validation or evaluation of actual application effects. Finally, the combined model with optimized weights is used to impute the missing items in the original missing dataset to obtain the final imputed values.

[0037] S3. Adaptive Data Normalization: Analyze the distribution type of each data feature. For features that are approximately normally distributed, use Z-score standardization. For features that are skewed, use logarithmic transformation or Box-Cox transformation before normalization. Alternatively, assign different weights to different normalization methods according to the importance of the features and combine them for normalization.

[0038] S4. Adaptive adjustment of WSMOTE algorithm parameters: Analyze the sample size and distribution characteristics of each minority class in the dataset, and then dynamically adjust the nearest neighbor sample size parameter and / or the number of new samples to be synthesized for each minority class sample based on the analysis results. Fit the distribution characteristics of the imputed dataset, randomly generate a large number of data as majority class samples, use the original imputed dataset as minority class samples, synthesize new samples through the WSMOTE algorithm based on the adjusted parameters, merge the minority class samples with the new samples to obtain the enhanced dataset, calculate the mean and standard deviation of each class of data in the enhanced dataset, and standardize the raw coal data of each group to complete the data standardization in the multi-coal source blending process.

[0039] S5. Model Training and Evaluation: The coal blending data processing model is trained using the enhanced dataset. Accuracy and F-value are used to evaluate the processing effect of categorical data, and the root mean square error is used to evaluate the processing effect of continuous data.

[0040] Furthermore, in S1, the data related to multi-source coal blending specifically includes: basic raw coal data, float-sink test data, coal blending test data, and production process data. The basic raw coal data includes coal type and origin, coal petrographic composition (proportion of vitrinite, inertinite, and chrysinite), industrial analysis data (moisture, ash, volatile matter, and fixed carbon), and elemental analysis data (carbon, hydrogen, oxygen, nitrogen, and sulfur content, etc.). The float-sink test data includes yield, ash, and sulfur content at various separation densities. The coal blending test data includes the blending ratio (the proportion of each raw coal in the blend), industrial analysis indicators after blending, and coke performance indicators (reactivity, post-reaction strength, etc.). The production process data includes process parameters such as moisture control, stirring time, and molding pressure during the coal blending process.

[0041] Preprocessing operations are detailed: Preprocessing includes data deduplication, preliminary screening of outliers (based on business rules, such as directly removing obviously unreasonable values ​​like yield ≤0 or ≥100% and ash content ≥50%), and data type conversion (such as encoding string-type coal type names into numerical labels).

[0042] In S2, the preset ratio is preferably 10%. 10% of the data rows are randomly sampled from the complete dataset using the row index, and their specified data items are replaced with missing value identifiers (such as np.nan). A new missing dataset is then constructed for weight calibration.

[0043] Iterative details of the random forest imputation algorithm: The iteration stopping condition of the random forest imputation algorithm can be set to the mean square error (MSE) of the imputation result being less than 0.1% for two consecutive iterations or the number of iterations reaching 5.

[0044] The specific implementation of cross-validation: Five-fold cross-validation is adopted to ensure that the proportion of each category in each fold of data is consistent with the proportion in the original dataset. The mean absolute error (MAE) of each fold is calculated, and the weight combination that minimizes the mean MAE is selected. By clarifying the proportions, iteration conditions, and cross-validation strategies, the weight optimization process for missing value imputation is made more operable, ensuring the accuracy and stability of the imputation results.

[0045] In S3, the quantification and weighting of feature importance are performed as follows: A random forest model (sklearn.ensemble.RandomForestClassifier or RandomForestRegressor) is used to train the complete data before normalization. The importance score of each feature is obtained through the feature_importances_ attribute. The score is then normalized and used as the weight of the combined normalization (e.g., if the importance score of a feature is 0.6, the weight of the corresponding normalization method is 0.6).

[0046] Determining λ for the Box-Cox transformation: The range of λ is [-2, 2]. By iterating through this interval using the lmbda parameter of scipy.stats.boxcox, we select the λ value that minimizes the skewness of the transformed data (skewness is calculated using scipy.stats.skew).

[0047] Example of combined normalization for multiple distribution features: For feature groups containing both normal and skewed distributions, Z-score normalization (weight 0.7) can be applied to the normal distribution component, while Box-Cox transformation and min-max normalization (weight 0.3) can be applied to the skewed distribution component. By quantifying feature importance and accurately determining transformation parameters, adaptive data normalization can more flexibly adapt to complex data distribution scenarios while preserving key feature information of the data.

[0048] In S4, the quantitative metrics for minority class distribution characteristics include sample size proportion, local density, and cluster compactness. Sample size proportion is calculated using `pandas.Series.value_counts(normalize=True)` to determine the percentage of samples from each class. Local density is calculated by taking the average distance of each minority class sample's K nearest neighbors (K = 5), and the reciprocal of this distance is the local density. Cluster compactness is achieved using DBSCAN clustering (sklearn.cluster.DBSCAN), and the silhouette coefficient (sklearn.metrics.silhouette_score) for each cluster is calculated; a coefficient closer to 1 indicates a more compact cluster.

[0049] The distribution fitting method for majority class sample generation is as follows: a Gaussian mixture model (GMM) (sklearn.mixture.GaussianMixture) is used to fit the majority class distribution of the imputed dataset. The optimal number of components (i.e. the number of Gaussian distributions) is determined according to the Bayesian Information Criterion (BIC), and then synthetic data with a sample size equivalent to the original majority class sample size is generated.

[0050] The specific steps of data standardization are as follows: The mean and standard deviation of each category of data in the augmented dataset are calculated. The Z-score standardization formula is used to standardize each group of raw coal data, ensuring that the data distribution conforms to a standard normal distribution with a mean of 0 and a standard deviation of 1. By quantifying distribution characteristics, employing precise distribution fitting methods, and performing standardization, the realism of the synthesized samples by the WSMOTE algorithm and the rationality of the data augmentation distribution are improved, providing a high-quality dataset for subsequent model training.

[0051] In S5, model hyperparameter optimization: The hyperparameters of the coal blending data processing model are optimized using a combination of grid search and Bayesian optimization (scikit-optimize.BayesSearchCV). Taking the random forest model as an example, the optimized hyperparameters include:

[0052] n_estimators (number of trees): 50-500, step size 50.

[0053] max_depth (maximum depth of the tree): 5-20, step size 1.

[0054] min_samples_split (minimum number of samples for node split): 2-10, step size 1.

[0055] Comprehensive decision-making based on multiple indicators: For categorical data, in addition to accuracy and F-score, the area under the receiver operating characteristic curve (AUC-ROC) can be calculated to assess the model's ability to distinguish class boundaries. For continuous data, in addition to the root mean square error (RMSE), the coefficient of determination (R²) can be calculated to assess the model's ability to explain data variation. A comprehensive score of model performance is formed by weighted averaging (e.g., accuracy weighted 0.3, F-score weighted 0.3, AUC-ROC weighted 0.2, R² weighted 0.2).

[0056] Model generalization validation: Two methods were employed to validate the model's generalization ability: time series partitioning (using early data as the training set and later data as the test set) and cross-coal type validation (training with data from certain coal types and testing with data from the remaining coal types). This ensured the model could operate stably under different time periods and coal type combinations. Hyperparameter optimization and multi-dimensional evaluation were used to improve the performance and generalization ability of the coal blending data processing model, ensuring it could provide reliable prediction and decision support for coal blending scheme optimization.

[0057] During model training, cross-validation is used to train and validate the model multiple times to obtain more reliable performance evaluation results. For evaluating the performance on categorical data, accuracy directly reflects the model's ability to correctly classify data, while the F-score comprehensively considers precision and recall, providing a more complete assessment of the model's performance on classification tasks. For evaluating the performance on continuous data, the root mean square error (RMSE) measures the difference between the model's predicted values ​​and the actual values; a smaller RMS value indicates a better model performance on continuous data. These evaluation metrics accurately determine the performance of the coal blending data processing model, providing a basis for subsequent optimization and improvement.

[0058] During the data preprocessing stage, the standardization parameters are dynamically adjusted based on the results of data imputation and normalization, or conditions are set to standardize only the features that still have significant outliers or outliers after imputation or normalization, while retaining the original processing results for features with stable distribution and no outliers.

[0059] Furthermore, this dynamic adjustment strategy can more accurately adapt to the data distribution characteristics of different features, avoiding information loss or bias that may result from indiscriminate standardization of all features. By setting reasonable threshold conditions, features that truly require standardization can be identified, such as outliers or isolated points that still deviate significantly from the overall distribution after imputation or normalization. Standardizing these features can significantly improve data quality and model training performance. For features with stable distributions and no outliers, their original processing results are retained to reduce unnecessary computational overhead and potential information distortion.

[0060] In S1, the relevant data includes raw coal float-sink test data and coal blending test data, covering the yield, ash content, and coal quality composition information of different raw coals at various sorting densities. The missing data set consists of data rows containing missing data items; the rest are complete datasets. Preprocessing includes missing value identification and data format conversion.

[0061] Furthermore, in terms of missing value identification, advanced data scanning algorithms are employed to meticulously examine each row of data in the dataset, accurately locating data rows with missing items and marking them as missing datasets, while data rows without missing items are classified as complete datasets. In the data format conversion stage, based on the needs of subsequent analysis, data such as yield, ash content, and coal composition information from raw coal float-sink test data and coal blending test data are uniformly converted into specific standard formats. For example, yield data is converted to percentage form, and ash content data is converted to a unified unit of measurement, ensuring consistency and compatibility of data in subsequent processing and analysis.

[0062] In S2, the preset ratio is 10%. 10% of the data rows in the complete dataset are randomly selected by row coding and replaced with missing value identifiers to construct a new missing dataset.

[0063] Furthermore, in constructing the new missing data set, a specific random number generation mechanism was employed to control the extraction of row codes to ensure randomness and representativeness. This mechanism guarantees that each extracted sample is both random and covers all parts of the complete dataset. After being replaced with missing value identifiers, the new missing data set not only retains the distribution characteristics of the original data but also increases the diversity of missing data, providing a richer and more challenging data foundation for subsequent WSMOTE algorithm processing.

[0064] The specific process of the random forest imputation algorithm is as follows:

[0065] Step 1: Use baseline methods to perform preliminary imputation on the new missing dataset. For categorical variables, use mode imputation, and for continuous variables, use median imputation of adjacent data.

[0066] Step 2: Construct a random forest based on the data after preliminary imputation, and generate a similarity matrix by the frequency of samples falling into the same leaf node in the random forest;

[0067] Step 3: Perform weighted imputation based on the similarity matrix. For categorical variables, select the category with the highest similarity as the imputation value. For continuous variables, use the weighted average of the similarity of the non-missing data as the imputation value.

[0068] Step 4: Repeat the above steps of building a random forest and interpolation until the preset stopping condition is met.

[0069] Furthermore, after processing with the random forest imputation algorithm, the WSMOTE algorithm operates on the imputed dataset. Its core objective is to oversample the minority class samples to balance the number of samples in each class, thereby effectively addressing the model bias problem caused by imbalanced samples. Specifically, the WSMOTE algorithm first calculates the Euclidean distance between minority class samples to determine the K nearest neighbors of each minority class sample. Subsequently, the algorithm randomly generates new minority class samples along the lines connecting the minority class samples and their nearest neighbors, according to a certain ratio, until a preset sample balance standard is reached. Through this series of refined processing steps, the WSMOTE algorithm can significantly improve the balance of the dataset, providing more reliable data support for subsequent model training and prediction.

[0070] The regression imputation algorithm uses a linear regression model, estimates the model parameters using the least squares method, and calculates the imputed values ​​for missing values ​​based on the intrinsic relationship between the independent and dependent variables.

[0071] Furthermore, this algorithm is suitable for datasets where there is a clear linear relationship between variables. It can accurately predict the possible values ​​of missing data points based on the distribution patterns of existing data points. In its application, a comprehensive exploratory analysis of the dataset is first required to clarify the independent and dependent variables, ensuring the applicability of the linear regression model. Next, the least squares method is used to accurately estimate the model parameters, enabling the model to fit the existing data to the greatest extent possible. Finally, based on the constructed linear regression model, imputation calculations are performed for the missing values ​​in the dataset, resulting in a complete and reliable dataset, laying a solid foundation for subsequent processing steps based on the WSMOTE algorithm.

[0072] In S3, the formula for calculating the Z-score standardization is as follows: ,in For the original data of the i-th sample in the j-th category, and These are the mean and standard deviation of the j-th category of data, respectively. This is the standardized data.

[0073] Furthermore, after Z-score standardization, the data will be transformed to a standard normal distribution with a mean of 0 and a standard deviation of 1. This effectively eliminates differences in units and numerical ranges between different categories of data, making subsequent data processing and analysis more accurate and stable. At the same time, the standardized data is also easier to visualize and compare, helping to understand the distribution characteristics and patterns of the data more intuitively.

[0074] In S4, the nearest neighbor sample size parameter and the new sample size are dynamically adjusted according to the number and distribution characteristics of minority class samples. The calculation formula for synthesizing new samples by the WSMOTE algorithm is: Xnew[j]=Xi[j]+ωj×random(0,1)×(Yi[j]-Xi[j]), where Xi[j] is the j-th class data of minority class samples, Yi[j] is the j-th class data of nearest neighbor samples, ωj is the entropy weight of the j-th class, and random(0,1) is a random number in the interval 0-1.

[0075] Furthermore, this formula achieves dynamic synthesis of new samples by combining the differences between minority class samples and nearest neighbor samples, and introducing entropy weights and random numbers. The introduction of entropy weight ωj reflects the importance of different classes of data, making the synthesized new samples more consistent with the distribution characteristics of the actual data. The randomness of random(0,1) increases the diversity of new samples, helping to improve the model's generalization ability. Through this dynamic adjustment of the nearest neighbor sample size and the number of new samples, the WSMOTE algorithm can better handle the class imbalance problem in multi-coal source blending data.

[0076] In S5, the formula for calculating the root mean square error is: , where y[j] is the j-th data in the augmented dataset, ynew[j] is the j-th data in the synthetic sample, and N is the number of samples after data augmentation.

[0077] Furthermore, this formula effectively evaluates the data augmentation effect by calculating the root mean square error (RMSE) between the augmented dataset and the synthetic sample. Specifically, the smaller the RMSE value, the smaller the difference between the synthetic sample and the augmented dataset, indicating a better data augmentation effect. This evaluation metric provides an important reference for subsequent data processing and model training, contributing to further improving the accuracy and reliability of multi-coal source blending data processing methods.

[0078] The process of dynamically adjusting the standardization parameters is as follows: If there are significant outliers in the imputed data, the calculation range of the standardized mean and standard deviation is limited to the data after removing outliers. If the data distribution changes due to imputation or normalization, the standardization method is switched.

[0079] Furthermore, this mechanism of dynamically adjusting standardization parameters effectively avoids the problem of poor standardization results caused by outliers or changes in data distribution. In practical applications, significant outliers are first identified using statistical methods. These outliers may arise from data acquisition errors or special operating conditions, interfering with the standardization process. After removing outliers, the mean and standard deviation are recalculated to ensure the accuracy of the standardization parameters. Simultaneously, when the data distribution changes significantly due to interpolation or normalization operations, the system automatically switches to a standardization method more suitable for the current data distribution, such as switching from Z-score standardization to Min-Max standardization, to adapt to the data processing needs of different scenarios.

[0080] Example 2

[0081] We collected float-sink test data and coal blending test data for four types of raw coal (Shitai, Shuanglong, Zhuzhuang, and Yuanyi) from the Huaibei Coal Preparation Plant, including information on yield, ash content, and sulfur content at various separation densities. The data was preprocessed to remove invalid and redundant data, identify missing items, and divide the data rows containing missing items into missing datasets, while the rest were considered complete datasets.

[0082] Further, S1, data collection and classification.

[0083] Data collected includes float and sink test data for four types of raw coal (yield, ash and sulfur content at separation densities of 1.3, 1.4 and 1.5, etc.) and coal blending test data (coal blending ratio, mixed ash, mixed sulfur and coke reactivity, etc.).

[0084] Preprocessing: The Python Pandas library was used to identify missing values ​​(a total of 128 missing items, involving 6 indicators such as yield and ash content). Three abnormal records with a yield > 100% were removed, resulting in a missing dataset (65 records) and a complete dataset (935 records).

[0085] S2, Missing value imputation optimization.

[0086] Feature and missing pattern analysis: Yield and ash content are continuous normal distribution features (random missing), while coal quality category is a categorical feature (non-random missing). Based on this, the initial weight of the random forest was set to 0.6, and the initial weight of the regression was set to 0.4. 10% (94 records) of the complete dataset were randomly selected, and missing items (102 missing points in total) were manually generated.

[0087] Random forest imputation uses a random forest of 100 trees. After initial imputation, it is optimized by weighted similarity matrix. After three iterations, the MAE is 0.85. Regression imputation uses linear regression for continuous features and logistic regression for categorical features, with an MAE of 1.02.

[0088] Weight optimization: After traversing the weight combinations through 5-fold cross-validation, the optimal weights were determined to be 0.65 for random forest and 0.35 for regression. At this point, the MAE of the validation set imputation dropped to 0.78.

[0089] Original data imputation: The weight combination is applied to imput 65 missing records to obtain complete data without missing records.

[0090] S3, Adaptive Data Normalization.

[0091] Yield and ash content: Shapiro-Wilk test p-value > 0.05, approximately normal distribution.

[0092] Sulfur content: The skewness coefficient is 1.2 (right-skewed), and the skewness coefficient decreases to 0.3 after Box-Cox transformation.

[0093] Coal quality category: a category-type characteristic, which does not require special normalization after being coded by label.

[0094] Normalization implementation: Yield and ash were standardized using Z-scores with means of 0.02 and -0.01, and standard deviations of 0.98 and 1.03, respectively. Sulfur was normalized using a Box-Cox transformation (λ=0.4) followed by min-max normalization within the range [0,1]. Finally, all features were normalized to the fit interval.

[0095] S4 and WSMOTE algorithm parameters are adaptively adjusted.

[0096] Minority category analysis: In the coal blending test, the "high ash, low sulfur" category had only 42 entries (accounting for 4.2%), and DBSCAN clustering showed that its distribution was scattered (local density 0.25). The "medium ash, medium sulfur" category had 310 entries (accounting for 31%), with a concentrated distribution (local density 0.8).

[0097] Parameter adjustment: "High ash, low sulfur" k=6, N=10. "Medium ash, medium sulfur" k=3, N=5.

[0098] Majority class generation: 10,000 majority class samples are generated by fitting a Gaussian mixture model.

[0099] WSMOTE synthesis: 420 new samples (10x) were generated for "high ash and low sulfur" and 1550 new samples (5x) were generated for "medium ash and medium sulfur".

[0100] Enhanced dataset: The difference in the proportion of each category is less than 8%, and the data distribution is more balanced.

[0101] S5. Model Training and Evaluation.

[0102] Model selection: Random Forest (300 trees, maximum depth 15).

[0103] Evaluation indicators:

[0104] Classification task (whether the coal blending plan is qualified): accuracy rate 92.5%, F1-score 89.3%.

[0105] Regression task (predicting coke reactivity): RMSE 2.1, MAE 1.5 (industry standard RMSE < 3 is excellent).

[0106] Generalization validation: Cross-coal type test (coal A training, coal B test), accuracy 88.7%, RMSE 2.5, meeting industrial requirements.

[0107] Outlier detection: After interpolation, there were 3 outliers in the sulfur content (>Q3+1.5IQR).

[0108] Standardization implementation: Only the mean and standard deviation of sulfur content are recalculated (outliers are removed), and Z-score standardization is performed. Other features retain the normalized results.

[0109] Final output: A standardized multi-coal source blending database is formed, containing 15 indicators of 5 types of raw coal, supporting intelligent optimization of blending schemes.

[0110] The working principle and usage process of this invention are as follows: Data from raw coal float-sink tests and coal blending tests related to multi-source coal blending are collected, covering key information such as yield, ash content, and coal quality composition. The data is preprocessed to identify missing data items, dividing the data into missing datasets and complete datasets, laying the foundation for subsequent imputation processing.

[0111] Then, data characteristics (such as numerical / categorical type, data distribution, and missing rate) or missing patterns (such as random missing, completely random missing, and non-random missing) are analyzed to assign different initial weights to the random forest imputation algorithm and the regression imputation algorithm. Next, 10% of the missing items are randomly generated based on the complete dataset to construct a new missing dataset for algorithm weight calibration. The random forest imputation algorithm and the regression imputation algorithm are used for imputation. Through cross-validation or evaluation of practical application effects, the weights are dynamically adjusted or optimized to achieve the best overall performance of the combined imputation model. The combined model with optimized weights is then used to achieve accurate imputation of the original missing data, improving data integrity.

[0112] Adaptive data normalization: Analyze the distribution type of each data feature (such as normal, skewed, and multimodal distributions). For features that are approximately normally distributed, Z-score standardization is used. For skewed features, logarithmic or Box-Cox transformations are applied before normalization, or different weights are assigned to different normalization methods based on feature importance for combined normalization, eliminating dimensional differences while adapting to the data distribution characteristics.

[0113] The WSMOTE algorithm features adaptive parameter adjustment: It analyzes the sample size and distribution characteristics of each minority class in the dataset. Based on the analysis results, it dynamically adjusts the nearest neighbor sample size parameter and / or the number of new samples to be synthesized for each minority class. For example, it uses a larger number of nearest neighbors and a larger synthesis quantity for classes with extremely small sample sizes, and a more flexible synthesis strategy for classes with more dispersed distributions. Parameter adjustments can be made in conjunction with cluster analysis or density estimation results. By fitting the data distribution characteristics, a large number of majority class samples are randomly generated and combined with the original data as minority class samples. Based on the adjusted parameters, new samples are synthesized using the WSMOTE algorithm to solve the data imbalance problem while ensuring consistency between the synthesized samples and the real data, thus improving sample quality.

[0114] Model Training and Evaluation: The model is trained using the enhanced dataset, and the data processing effect is comprehensively evaluated through indicators such as accuracy, F-score, and root mean square error to ensure that the data processing quality meets the optimization requirements of the coal blending scheme.

[0115] During the data preprocessing stage, standardization parameters (such as the calculation range or benchmark for the mean and standard deviation) are dynamically adjusted based on the results of data interpolation and normalization. Alternatively, conditions can be set to standardize only features that still have significant outliers or isolated points after interpolation or normalization. By integrating data standardization into the preprocessing workflow, iterative optimization or conditional standardization methods can be formed to balance the differences in data from various coal sources and provide standardized, high-quality data support for coal blending scheme optimization.

[0116] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. A multi-coal source blending data processing method based on the WSMOTE algorithm, characterized in that, Includes the following steps: S1. Data Acquisition and Classification: Collect data related to coal blending from multiple coal sources, preprocess the collected data, and filter out missing and complete datasets. S2. Missing Value Imputation Optimization: Based on the complete dataset, a new missing data set is constructed by randomly generating a predetermined proportion of missing items. Random forest imputation and regression imputation algorithms are then used to impute the new missing data set. Weights are dynamically adjusted or optimized through cross-validation or practical application evaluation. Finally, a combined model with optimized weights is used to impute the missing items in the original missing data set, yielding the final imputed values. S3. Adaptive Data Normalization: The distribution type of each data feature is analyzed. For features with approximately normal distributions, Z-score standardization is used. For skewed features, logarithmic transformation or Box-Cox transformation is used before normalization. Alternatively, different weights are assigned to different normalization methods based on feature importance for combined normalization. S4. Adaptive Adjustment of WSMOTE Algorithm Parameters: The dataset is analyzed... The sample size and distribution characteristics of each minority class are determined. Then, based on the analysis results, the nearest neighbor sample size parameter and / or the number of new samples to be synthesized for each minority class sample are dynamically adjusted. The distribution characteristics of the imputed dataset are fitted, and a large number of data are randomly generated as majority class samples. The original imputed dataset is used as minority class samples. Based on the adjusted parameters, new samples are synthesized using the WSMOTE algorithm. The minority class samples and the new samples are merged to obtain the enhanced dataset. The mean and standard deviation of each class of data in the enhanced dataset are calculated, and the raw coal data of each group are standardized to complete the data standardization in the multi-coal source coal blending process. S5, Model Training and Evaluation: The coal blending data processing model is trained using the enhanced dataset. The accuracy and F-value are used to evaluate the processing effect of categorical data, and the root mean square error is used to evaluate the processing effect of continuous data.

2. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, During the data preprocessing stage, the standardization parameters are dynamically adjusted based on the results of data imputation and normalization, or conditions are set to standardize only the features that still have significant outliers or outliers after imputation or normalization, while retaining the original processing results for features with stable distribution and no outliers.

3. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, In S1, the relevant data includes raw coal float-sink test data and coal blending test data, covering the yield, ash content and coal quality composition information of different raw coals at various sorting densities; the missing dataset is the data row containing missing data items, and the rest is the complete dataset; the preprocessing includes missing value identification and data format conversion.

4. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, In S2, the preset ratio is 10%. 10% of the data rows in the complete dataset are randomly extracted by row coding and replaced with missing value identifiers to construct a new missing dataset.

5. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, The specific process of the random forest imputation algorithm is as follows: Step 1: Initially imput the new missing dataset using a baseline method. Categorical variables are imputed using the mode, and continuous variables are imputed using the median of adjacent data. Step 2: Construct a random forest based on the data after initial imputation, and generate a similarity matrix by the frequency of samples falling into the same leaf nodes in the random forest. Step 3: Perform weighted imputation based on the similarity matrix. For categorical variables, the category with the highest similarity is selected as the imputation value, and for continuous variables, the weighted average of the similarity of non-missing data is used as the imputation value. Step 4: Repeat the above steps of constructing the random forest and imputation until the preset stopping condition is met.

6. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, The regression imputation algorithm uses a linear regression model, estimates model parameters using the least squares method, and calculates imputed values ​​for missing values ​​based on the intrinsic relationship between independent and dependent variables.

7. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, In S3, the formula for calculating the Z-score standardization is as follows: ,in For the original data of the i-th sample in the j-th category, and These are the mean and standard deviation of the j-th category of data, respectively. This is the standardized data.

8. The multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, In S4, the nearest neighbor sample size parameter and the new sample size are dynamically adjusted according to the number and distribution characteristics of minority class samples. The calculation formula for synthesizing new samples by the WSMOTE algorithm is: Xnew[j]=Xi[j]+ωj×random(0,1)×(Yi[j]-Xi[j]), where Xi[j] is the j-th class data of minority class samples, Yi[j] is the j-th class data of nearest neighbor samples, ωj is the entropy weight of the j-th class, and random(0,1) is a random number in the interval 0-1.

9. A multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, In S5, Accuracy = (TP + TN) / (TP + TN + FP + FN), Recall = TP / (TP + FN), and Precision = TP / (TP + FP), where TP represents the number of samples that actually belong to and are correctly predicted as the minority class, TN represents the number of samples that actually belong to and are correctly predicted as the majority class, FP represents the number of samples that actually belong to the majority class but are incorrectly predicted as the minority class, and FN represents the number of samples that actually belong to the minority class but are incorrectly predicted as the majority class. The evaluation metric F = Recall × Precision; the formula for calculating the root mean square error is... , where y[j] is the j-th data in the augmented dataset, ynew[j] is the j-th data in the synthetic sample, and N is the number of samples after data augmentation.

10. A multi-coal source blending data processing method based on the WSMOTE algorithm according to claim 1, characterized in that, The process of dynamically adjusting the standardization parameters is as follows: if there are significant outliers in the imputed data, the calculation range of the standardized mean and standard deviation is limited to the data after removing outliers; if the data distribution changes due to imputation or normalization, the standardization method is switched.

Images