A flue gas acid making data cleaning and optimization method based on isolated forest and weighted random forest

Abstract

The application discloses a flue gas acid making data cleaning and optimization method based on isolated forest and weighted random forest, which analyzes the flue gas acid making desulfurization process, combines a large amount of production monitoring data, adopts a maximum information coefficient analysis method to perform correlation analysis on process variables such as the O2 concentration at the fan outlet, the flue gas temperature at the fan outlet, the first power wave inlet pressure, the furnace pressure, the fan inlet flow, the converter inlet temperature and the like, and obtains key variables affecting SO2 conversion rate and sulfuric acid production and the like indexes. Then, for the key variables, the original data change trend is analyzed, the isolated forest algorithm is used to identify and eliminate abnormal values and outliers in the data set, and a missing data set is obtained. Finally, the weighted random forest algorithm is used to fit and predict the missing data set, to compensate for the missing data therein, realize cleaning and optimization of the flue gas acid making process data, and thus achieve the purpose of improving the desulfurization efficiency and the sulfuric acid production.

Description

Technical Field

[0001] This invention belongs to the field of data processing, and specifically relates to a method for cleaning and optimizing flue gas acid production data based on isolated forest and weighted random forest. Background Technology

[0002] Non-ferrous metals such as copper, lead, aluminum, and magnesium are important strategic materials for the development of my country's national economy and defense industry, and are also raw materials for manufacturing many types of equipment such as aircraft, rockets, missiles, and computers. With the continuous acceleration of my country's industrialization and rapid economic development, the demand for non-ferrous metal resources in various industries is increasing. Therefore, the production of non-ferrous metals has occupied an important position in my country's industrial production. However, in nature, most non-ferrous metal minerals exist in the form of sulfides, and smelting them produces large amounts of flue gas containing SO2. Direct emission of SO2-containing flue gas into the atmosphere causes a series of environmental problems such as air pollution and soil acidification. At the same time, SO2, as a Group 3 carcinogen, poses a significant threat to human health. Therefore, with increasing environmental awareness, how to effectively control SO2 in flue gas has become an urgent issue to be addressed. Due to the high concentration and wide variation of SO2 in smelting flue gas, a relatively mature flue gas desulfurization process—flue gas sulfuric acid production—has been developed. The smelting flue gas sulfuric acid production industry recovers SO2 from flue gas to produce high-concentration sulfuric acid.

[0003] Flue gas sulfuric acid production is a complex, multivariate, and strongly coupled nonlinear process. Data from its operation is crucial for monitoring process status, optimizing control, and diagnosing faults, serving as the information foundation for improving sulfuric acid production efficiency and quality. However, due to the complex operating environment, numerous devices, and strong coupling between various stages, data obtained from monitoring equipment can be severely contaminated, prone to missing data and outliers, posing significant challenges to data analysis and processing. Therefore, accurately removing outliers and compensating for missing data is essential for subsequent modeling and control of the flue gas sulfuric acid production process.

[0004] Currently, to address the difficulty in identifying outliers in datasets, various anomaly identification methods have been proposed, including those based on data probability distribution, density, and distances between data points. Specific methods include Laida's rule, quartiles, and DBSCAN clustering. However, probability distribution-based analysis methods are only applicable to data with known distribution characteristics, and clustering-based outlier detection methods can only identify global outliers, failing to recognize local anomalies. For missing dataset compensation, widely used methods include interpolation, support vector machine regression, and backpropagation (BP) neural network fitting. However, interpolation relies heavily on the quality of historical and neighboring data, making it ineffective for accurately compensating for arbitrary sets of outliers. Using neural networks for data compensation requires ensuring the validity of the training data and necessitates additional algorithms for judgment. In actual flue gas acid production processes, outliers not only contain anomalous features of single variables but also multiple variables exhibiting synchronous or asynchronous characteristics, which existing anomaly compensation methods cannot effectively address.

[0005] Therefore, in the data cleaning and optimization methods for flue gas sulfuric acid production based on isolated forests and weighted random forests, isolated forests can quickly and accurately identify and remove outlier data, while weighted random forests can fit and predict data trends based on the relationships between variables using regression tree ensemble learning, effectively compensating for the removed outlier data. By identifying, removing, and compensating for outlier data, valuable datasets are obtained, further improving the accuracy of subsequent modeling and control, thereby ensuring the stable operation of the flue gas sulfuric acid production system and increasing SO2 conversion rate and sulfuric acid production. Summary of the Invention

[0006] This invention proposes a data cleaning and optimization method for flue gas-based sulfuric acid production based on isolated forest and weighted random forest. The method uses actual data from the flue gas-based sulfuric acid production process of a copper smelter as input. Maximum information coefficient analysis is used to select data variables that significantly affect SO2 conversion rate and sulfuric acid production. An outlier identification model is established using the isolated forest algorithm to identify and remove outliers from the extracted key variable dataset. Then, a missing data compensation model is established using the weighted random forest algorithm to compensate for missing data in the dataset, resulting in a valuable dataset and achieving data cleaning and optimization for the flue gas-based sulfuric acid production process.

[0007] A method for cleaning and optimizing flue gas acid production data based on isolated forests and weighted random forests, characterized by the following steps:

[0008] Step 1: The flue gas sulfuric acid production process monitoring system monitors and collects data in real time during the sulfuric acid production process from the flue gas generated by copper smelting.

[0009] Step 2: The maximum information coefficient analysis method is used to analyze the correlation between process variables and capacity indicators to obtain the key variables affecting SO2 conversion rate and sulfuric acid production.

[0010] Step 3: Based on the changing trends of key variable data, design an outlier identification model based on isolated forest to identify and remove outliers from the dataset.

[0011] Step 4: Establish a weighted random forest-based fitting and prediction model to fit and predict the missing dataset, compensate for the missing data, and obtain a valuable dataset.

[0012] This invention presents a flue gas sulfuric acid production cleaning and optimization method based on isolated forests and weighted random forests. Using flue gas sulfuric acid production process data from a copper smelter as the research object, it analyzes the correlation between process variables such as blower outlet O2 concentration, blower outlet flue gas temperature, primary power wave inlet pressure, furnace pressure, blower inlet flow rate, and converter inlet temperature and capacity indicators such as SO2 conversion rate and sulfuric acid production. Key variables with significant impact on subsequent modeling and control accuracy are selected. The changing trends of the selected key variable data are analyzed, and an anomaly identification model based on isolated forests is established to identify and remove anomalies from the dataset. Then, a missing data compensation model based on weighted random forests is established to compensate for the removed missing data, providing valuable datasets for subsequent modeling and control, improving the accuracy of control and modeling, further ensuring the stable operation of the flue gas sulfuric acid production system, and increasing SO2 conversion rate and sulfuric acid production.

[0013] In addition, the flue gas acid production data cleaning and optimization method based on isolated forest and weighted random forest according to this embodiment also has the following additional technical features:

[0014] In step 2, based on actual production process data, the maximum information coefficient analysis method is used to analyze the correlation between process variables such as flue gas flow rate, inlet temperature of each stage of the converter, and fan outlet pressure, and capacity indicators such as SO2 conversion rate and sulfuric acid production. Key variables with a maximum information coefficient absolute value greater than 0.75 are extracted. This analysis method further calculates the maximum information coefficient between two variables by calculating the mutual information between them. The calculation formula is as follows:

[0015]

[0016]

[0017] B(n)=n 0.6 (12)

[0018] In this context, variable X represents the SO2 conversion rate, variable Y represents the various variables in the flue gas sulfuric acid production process, I[X;Y] represents the mutual information between variables X and Y, p(X,Y) represents the joint probability between variables X and Y, p(X) represents the probability distribution of variable X, p(Y) represents the probability distribution of variable Y, MIC[X;Y] represents the maximum information coefficient between variables X and Y, and n represents the amount of data. B(n) is a single variable whose magnitude is related to the amount of data. This allows us to derive the key variables affecting the SO2 conversion rate. Similarly, the key variables affecting sulfuric acid production can be obtained.

[0019] The main steps for analyzing the correlation between variables using the maximum information coefficient analysis method are as follows:

[0020] (1) Given the values ​​of i and j, grid the scatter plot of variables X and Y in columns i and rows j, and find the maximum mutual information value.

[0021] (2) Normalize the largest mutual information value.

[0022] (3) Select the maximum value of mutual information at different scales as the MIC value.

[0023] In step 3, an outlier identification model based on isolated forest is established to identify and remove outliers and anomalies in the extracted dataset of key variables affecting SO2 conversion rate and sulfuric acid production. The isolated forest algorithm performs multiple binary splits on the sample points until each or a few sample points are segmented into the same region. Normal data often requires multiple splits and is located in high-density regions; outlier data requires fewer splits and is located in low-density regions.

[0024] After processing by the anomaly detection model, the flue gas acid production dataset is divided into different high and low density regions. The anomaly score is calculated to characterize the region where the data is located, and data with high scores are removed. The calculation method is as follows:

[0025]

[0026]

[0027] Where C(u) is the average path length of all data in the flue gas acid production dataset, and S(h) ij (,u) represents the outlier score for the flue gas acid production variable data, where u is the number of samples in the flue gas acid production data, and h is the number of samples. ij Data on acid production from flue gas x ij The path length, ξ is Euler's constant, E(h) ij ) is the data x ij The average path length among n isolated trees.

[0028]

[0029] According to the above calculation method, when S(h ij When the value of S(h) is close to 0.5, it is not easy to distinguish whether the data is an outlier in the flue gas acid production dataset; when S(h) is close to 0.5, it is not easy to distinguish whether the data is an outlier in the flue gas acid production dataset; ij When the value of S(h) is close to 0, the data is considered normal; when the value of S(h) is close to 0, the data is considered normal. ij When the value of u is close to 1, the data is considered an outlier. Based on the outlier score of each data point, it is removed from the flue gas acidification dataset. The steps for outlier identification and removal are as follows:

[0030] (1) Randomly select a sample of size n from the key variable dataset extracted in step 2 as the training set for training the isolated tree.

[0031] (2) Randomly select variable Q as the root node in the training set, and randomly select a cut point T within the range of values ​​of Q.

[0032] (3) Place samples with variable values ​​greater than or equal to T in the left node and samples with values ​​less than T in the right node.

[0033] (4) Repeat steps (2) and (3) for the data of the left and right nodes until the termination condition is met, thus completing the establishment of the isolated forest model. The termination condition is one of the following three cases:

[0034] ① Reach the maximum height of the tree;

[0035] ② The values ​​of the corresponding features of the samples at the nodes are all equal;

[0036] ③ Each node has only one sample.

[0037] In step 4, a missing data compensation model based on weighted random forest is established to fit and predict the missing dataset, thereby compensating for the missing data and achieving the cleaning and optimization of the flue gas acid production process data.

[0038] Weighted random forests improve prediction accuracy by assigning different weights to base learners, thus weighting their prediction results. Weighted random forest regression uses the mean absolute error (MAPE) of out-of-bag data as an evaluation metric to assess the predictive ability of the base learners and assigns weights to them.

[0039]

[0040]

[0041] Where MAPE is the mean absolute error percentage of the random forest regression model for flue gas acid production data outside the bag, t is the number of data points outside the bag, and yi is the true value of the flue gas to sulfuric acid data, f(x i ) is the predicted value of the random forest regression, MAPE i is the percentage of the mean absolute error of the prediction of the i-th regression tree, w i is the weight of the i-th regression tree, and n is the number of regression trees in the random forest algorithm. When the value of MAPE i is larger, the prediction accuracy of this learner is lower, and the value of w i will decrease correspondingly, indicating that the influence of this learner on the prediction result becomes smaller. The specific steps of the weighted random forest algorithm are as follows:

[0042] (1) Draw a subsample matrix with replacement from the flue gas to sulfuric acid data training matrix T as the training sample for the regression tree, and its size is the same as that of the training matrix.

[0043] (2) The feature dimension of each flue gas to sulfuric acid data sample is M. Specify a constant m, where m << M. Randomly select m feature subsets from the M features, and select the optimal one from these m features each time the regression tree is split.

[0044] (3) Each tree grows to the maximum extent and there is no pruning process until it reaches the tree height and then stops growing.

[0045] (4) When training the i-th regression tree, use the out-of-bag data as the test set and input it into this regression tree, and calculate the percentage of the mean absolute error of the prediction of this regression tree MAPE i and the weight w i .

[0046] (5) Repeat the above steps to complete the construction and training of n regression trees.

[0047] Finally, n weighted regression trees are integrated to obtain a weighted random forest, and the final model output is:[[]]

[0048]

[0049] where w<o000018>is the weight of the i-th regression tree, and T i is the prediction result of the i-th regression tree. Description of the Drawings

[0050] Figure 1 is the Isolation Forest model diagram.

[0051] Figure 2 is the Random Forest Regression model diagram.

[0052] Figure 3 is the Isolation Forest algorithm flowchart.

[0053] Figure 4 This is the flowchart of the weighted random forest algorithm. Detailed Implementation

[0054] The embodiments of the present invention are described in detail below. These embodiments are implemented based on the technical solutions of the present invention, and provide detailed implementation methods and specific operation processes. However, the scope of protection of the present invention is not limited to the following embodiments.

[0055] Step 1: The flue gas sulfuric acid production process monitoring system monitors and collects data in real time during the sulfuric acid production process from the flue gas generated by copper smelting.

[0056] Step 2: The maximum information coefficient analysis method is used to analyze the correlation between process variables and capacity indicators to identify the key variables affecting SO2 conversion rate and sulfuric acid production. These key variables will be the main focus of subsequent data cleaning, modeling, and control.

[0057] Step 3: Analyze the changing trends of key variable data, design an anomaly identification model based on isolated forest, and identify and remove outliers from the dataset.

[0058] Step 4: Establish a weighted random forest-based fitting and prediction model to fit and predict the missing dataset, compensate for the missing data, obtain valuable datasets, and achieve data cleaning and optimization of the flue gas acid production process.

Claims

1. A method for cleaning and optimizing flue gas acid production data based on isolated forests and weighted random forests, characterized in that, It includes the following steps: Step 1: During the process of producing sulfuric acid from the flue gas generated by copper smelting in the flue gas sulfuric acid production process monitoring system, the production process is monitored in real time and data is collected. Step 2: Based on the real-time monitoring and data collection in Step 1, the maximum information coefficient analysis method is used to analyze the correlation between process variables and production capacity indicators, and the key variables affecting the SO2 conversion rate and sulfuric acid production are obtained. The above key variables include flue gas flow rate, inlet temperature of each layer of the converter, and fan outlet pressure. Step 3: According to the change trend of the key variable data obtained in Step 2, an outlier data recognition model based on the isolation forest is designed to identify and eliminate outliers and abnormal values in the data set. Step 4: A fitting prediction model based on the weighted random forest is established to perform fitting prediction on the missing data set, compensate for the missing data therein, and obtain a valuable data set. A missing data compensation model based on the weighted random forest is established to perform fitting prediction on the missing data set, compensate for the missing data therein, and achieve the optimization of the flue gas sulfuric acid production process data. The traditional random forest obtains the prediction result by averaging the output results of all regression trees, that is, the base learners, which affects the prediction accuracy of the random forest; the weighted random forest weights the prediction results of the base learners by assigning different weights to the base learners, improving the prediction accuracy of the random forest; the weighted random forest regression uses the mean absolute percentage error MAPE of the out-of-bag data prediction as an evaluation index to evaluate the prediction ability of the base learners and assigns weights to them. ， ； Where MAPE is the mean absolute error percentage of the random forest regression model for flue gas acid production data outside the bag, t is the number of data points outside the bag, and y i f(x) represents the actual value of the flue gas acid production data. i ) represents the predicted value from random forest regression, MAPE i w represents the percentage of the mean absolute error of the predictions for the i-th regression tree. i Let n be the weight of the i-th regression tree, and n be the number of regression trees in the random forest algorithm; when MAPE i The larger the value of w, the lower the prediction accuracy of the learner. i The value will decrease accordingly, indicating that the learner has a smaller impact on the prediction results; the specific steps of the weighted random forest algorithm are as follows: Step 1: Randomly draw a subsample matrix with replacement from the flue gas sulfuric acid data training matrix T as the training sample of the regression tree, and its size is the same as the size of the training matrix. Step 2: The feature dimension of each flue gas sulfuric acid data sample is M, and a constant m is specified, where m << M; randomly select m feature subsets from the M features, and each time the regression tree is split, select the optimal one from these m features. Step 3: Each tree grows to the maximum extent and there is no pruning process until it reaches the height of the tree and then stops growing. Step 4: When training the i-th regression tree, input the out-of-bag data as the test set into the regression tree, and calculate the mean absolute error (MAPE) of the regression tree for each tree. i and weight w i ; Step 5: Repeat the above steps to complete the construction and training of n regression trees. Finally, n weighted regression trees are integrated to obtain a weighted random forest, and the final model output is: ； Among them, w i Let T be the weight of the i-th regression tree. i This represents the prediction result of the i-th regression tree.

2. The method for cleaning and optimizing flue gas acidification data based on isolated forests and weighted random forests according to claim 1, characterized in that, Use the maximum information coefficient analysis method to analyze the correlation between variables. Calculate the maximum information coefficient between two variables by calculating the mutual information between the two variables. The calculation formula is as follows; ； ； ； Among them, variable X is the SO2 conversion rate, variable Y is each variable in the flue gas sulfuric acid production process, I[X;Y] is the mutual information between variable X and variable Y, p(X,Y) is the joint probability between variable X and variable Y, p(X) is the probability distribution of variable X, p(Y) is the probability distribution of variable Y, MIC[X;Y] is the maximum information coefficient between variable X and variable Y, n is the amount of data; B(n) is a variable, and its size is related to the amount of data; thus, the key variables affecting the SO2 conversion rate are obtained; similarly, the key variables affecting the sulfuric acid production can be obtained. The steps of using the maximum information coefficient analysis method to analyze the correlation between variables are as follows: Step 1: Given the values ​​of i and j, grid the scatter plot of variables X and Y in column i and row j, and find the maximum mutual information value; Step 2: Normalize the maximum mutual information value; Step 3: Select the maximum mutual information at different scales as the MIC value; Based on the above method, the correlation between variables in the flue gas sulfuric acid production process and SO2 conversion rate and sulfuric acid production was analyzed, and variables with high correlation were extracted as the objects of data cleaning.

3. The method for cleaning and optimizing flue gas acidification data based on isolated forests and weighted random forests according to claim 1, characterized in that, An outlier data identification model based on isolated forests was established to identify and remove outliers and abnormal values ​​in the extracted dataset of key variables affecting SO2 conversion rate and sulfuric acid production. The Isolation Forest algorithm performs multiple binary splits on the sample points until each or a few sample points are split into the same region. Normal data often needs to be segmented multiple times and is located in high-density areas; Outlier data requires a few partitions and is located in low-density areas; After processing by the anomaly detection model, the flue gas acid production dataset is divided into different high and low density regions. The anomaly score is calculated to characterize the region where the data is located, and data with high scores are removed. The calculation method is as follows: ； ； Where C(u) is the average path length of all data in the flue gas acid production dataset, and S(h) ij (,u) represents the outlier score for the flue gas acid production variable data, where u is the number of samples in the flue gas acid production data, and h is the number of samples. ij Data on acid production from flue gas x ij Path length, Let E(h) be the Euler constant. ij ) is the data x ij The average path length in n isolated trees; According to the above calculation method, when S(h ij When the value of S(h) is close to 0.5, it is not easy to distinguish whether the data is an outlier in the flue gas acid production dataset; when S(h) is close to 0.5, it is not easy to distinguish whether the data is an outlier in the flue gas acid production dataset; ij When the value of S(h) is close to 0, the data is considered normal; when the value of S(h) is close to 0, the data is considered normal. ij When the value of u is close to 1, the data is determined to be an outlier; based on the outlier score of each data point, it is removed from the flue gas acidification dataset; the steps for outlier identification and removal experiments are as follows: Step 1: Randomly select a sample of size n from the key variable dataset extracted in claim 2 as the training set for training the isolation tree; Step 2: Randomly select variable Q as the root node in the training set, and randomly select a cut point T within the range of values ​​of Q; Step 3: Place samples with variable values ​​greater than or equal to T in the left node and samples with values ​​less than T in the right node; Step 4: Repeat steps 2 and 3 for the data of the left and right nodes until the termination condition is met, thus completing the establishment of the isolated forest model; the termination condition is one of the following three cases: 1) Reach the maximum tree height; 2) The values ​​of all features corresponding to samples at a node are equal; 3) The node has only one sample.

Images