Method for improving LIBS quantitative analysis precision based on abnormal data detection and removal
By performing principal component analysis and abnormal data removal on LIBS spectral data, a high-precision and high-rootability quantitative analysis model was established, which solved the problem that abnormal data in LIBS technology affects the model accuracy and improved the efficiency of industrial applications.
Patent Information
- Application Number
- CN202510099743.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-01-22
- Publication Date
- 2025-04-25
AI Technical Summary
LIBS technology is susceptible to abnormal data in quantitative analysis, resulting in poor model prediction accuracy and robustness, especially in the case of fluctuations in experimental environment, human errors and large differences in matrix.
By performing principal component analysis on LIBS spectral data, abnormal data are identified and removed, and a quantitative analysis model based on effective data is established. Specific steps include spectral pretreatment, principal component analysis, abnormal data identification and removal, and finally establish a quantitative analysis model.
It significantly improves the prediction accuracy and robustness of the LIBS quantitative analysis model, reduces the repeated calibration work caused by abnormal data, and improves the efficiency of industrial applications.
Smart Images
Figure CN119889497A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to a method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal. Specifically, the method is based on obtaining spectral data using a laser induced breakdown spectroscopy (LIBS) device, identifying abnormal data on the spectrum before establishing a quantitative analysis model, removing the abnormal data, training and establishing a prediction model, and greatly improving the prediction accuracy and robustness of the model. The method belongs to the application field of laser induced breakdown spectroscopy. Background Art
[0002] LIBS technology is fast, accurate, and does not require sample preparation, and is widely used in many industrial scenarios such as coal, thermal power, and cement. Typical LIBS industrial applications mainly rely on sampling and pressing analysis of belt materials or directly embedding detection equipment across the belt for cross-line analysis. However, no matter which method is used, the equipment needs to be calibrated with standard samples before testing to obtain a quantitative analysis model for the corresponding analysis parameters.
[0003] LIBS spectral data has the characteristic of high dimensionality, that is, the number of spectral intensity variables contained in the data is usually around 10 4The mainstream approach is to reduce the dimension of the data through principal component analysis (PCA), map the data to the principal component space, and use the main variance information in the spectral data to establish a quantitative analysis model. However, principal component analysis is extremely sensitive to abnormal data. In the principal component space, abnormal data will force the hyperplane fitted by normal data points to rotate to the direction of the abnormal data points, causing different principal components to pay too much attention to abnormal data, and introduce the information of abnormal data as effective variance information into the quantitative analysis model, which ultimately affects the model prediction performance. However, it is inevitable to introduce erroneous samples and spectral data during the calibration of LIBS equipment. The fundamental manifestation is that the data has serious deviations in representativeness of the samples. Specifically, it mainly includes the following three aspects: First, due to changes in the experimental environment and fluctuations in laser energy, the spectral signal may fluctuate greatly, and the spectral data may become less representative of the physical and chemical properties of the samples, or even have no representativeness of the samples. In this case, although the training process of the model converges, it will inevitably affect the prediction accuracy of the model; second, due to human factors, incorrect sample number labeling and confusion of the true values of different standard samples will inevitably cause the model training process to not converge or the model prediction accuracy and robustness to be poor; third, mixing different types of samples with large matrix differences as standard samples will also lead to model non-convergence and poor model prediction ability. For example, in the analysis of coal quality, the industrial parameters of natural coal and blended coal of different types of coal may be close or even the same, but these two samples have significant differences in spectral performance due to the huge difference in the matrix. The spectral data involved in the above three situations are not clustered into one data cluster in the principal component analysis (PCA) space, but are clustered into different data clusters, resulting in a large prediction error in the quantitative analysis model based on principal component analysis. Therefore, it is very necessary to identify abnormal samples and abnormal spectral data in the process of establishing a quantitative analysis model. The method proposed in the present invention to improve the accuracy of LIBS quantitative analysis based on abnormal data detection and removal can realize the detection and identification of abnormal spectral data, effectively distinguish the samples with matrix heterogeneity introduced in the calibration, and screen out the spectral data with true value labeling errors. This method is feasible in practical applications and can quickly locate the cause of model abnormalities. The quantitative model established after the abnormal data is removed has the characteristics of high precision and high robustness, avoiding repeated calibration work caused by the presence of abnormal data. This method has great potential value in industrial applications. Summary of the invention
[0004] The present invention relates to a method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal. The method uses principal component analysis of spectral data to obtain the spatial relationship between data points in the principal component space, thereby identifying and removing abnormal spectral data, and establishing a quantitative analysis model based on truly effective data. The method is characterized in that it specifically includes the following steps: The LIBS system was used to collect spectral data. This set of data was used to train the quantitative analysis model. The number of samples used was ( ), preferably The number of spectra collected for each sample is ( ), preferably .
[0005] The spectral data of each sample collected is preprocessed, including the The effective data of the spectral data is screened, the background is eliminated, and the spectrum is normalized. Then the pre-processed spectral data is averaged, and each sample obtains a corresponding spectrum, and finally obtains The spectral matrix after preprocessing of samples for: (1) in Indicates The sample spectrum The pixel intensity value.
[0006] Using principal component analysis (PCA) Perform dimensionality reduction analysis to determine the variance information explanation rate k% of the original data ( , preferably ) is .
[0007] According to the number of principal components determined in 3 , select the first two principal components to calculate the maximum ellipse boundary of the ability to summarize the data variance information in the principal component space, and draw a principal component score graph, with the x-axis being the first principal component score and the y-axis being the second principal component score. The long axis of the ellipse boundary in the graph is and short axis The calculation method of is based on formula (2) and (3). The data within the boundary is considered valid data, and the data outside the boundary is marked as type I abnormal data. ( ).
[0008] (2) (3) in For the The variance of all data on the principal components, express Distribution confidence value =0.05, which can be obtained by looking up the F probability distribution table. is the total number of samples.
[0009] Further calculation of The distance between the projection point of a data point on the PCA model hyperplane and the center of the data point in the hyperplane , and use formula (5) to calculate the abnormal data judgment threshold .when , it means that there is a significant difference between this point and other data points, and this value is judged to be type II abnormal data .
[0010] (4) (5) in is the total number of samples, is the number of principal components selected or used in the model, For the The variance of all data on the principal components, Indicates The sample in The scores on the principal components are for The average of the principal component scores, Representation distribution The confidence value is =0.01, which can be obtained by querying the F probability distribution table.
[0011] According to formula (6), calculate The spatial distance between the data point and the hyperplane of the PCA model , and calculated according to (7)~(9) ,when , then the data point is judged to be type III abnormal data .
[0012] (6) (7) (8) (9) in is the number of data points, It represents the residual after the data is explained by the principal components, K is the total number of variables, A is the number of principal components, and M is [K, 100, ], Representation distribution The confidence value is =0.01.
[0013] The calculated abnormal data are eliminated according to the elimination rule (11), that is, the abnormal data belonging to both Class I and Class II are eliminated, and the abnormal data belonging to Class III are eliminated. The new spectral data matrix is obtained according to Establish a quantitative analysis model.
[0014] (11) Beneficial effects: (1) The method described in the present invention is based on the spatial relationship analysis of LIBS spectral data points in the principal component space, identifies and eliminates abnormal data caused by experimental errors or human factors, and greatly improves the model prediction performance and accuracy.
[0015] (2) The method of outlier elimination and modeling proposed in the present invention avoids repetitive work such as repeated data collection and multiple model building due to the existence of abnormal data, thereby improving the application efficiency of this technology in industrial sites. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 The first two principal component scores of coal quality calibration data and the distribution of type I abnormal points Figure 2 The distribution of the distance between the projection point of the principal component space data and the origin in the principal component plane Figure 3 The distance distribution between the data points in the principal component space and the principal component hyperplane space Figure 4 Modeling results without removing abnormal data Figure 5 Modeling results after removing abnormal data using this method DETAILED DESCRIPTION
[0017] The present invention is further described below in conjunction with the accompanying drawings and specific embodiments, wherein the methods are conventional methods unless otherwise specified, and the raw materials can be obtained from public commercial channels unless otherwise specified.
[0018] Example 1: Improving the quantitative analysis accuracy of coal ash by LIBS based on abnormal data detection and removal (1) This example verifies the effectiveness of the method described in the invention, and specifically compares the modeling results of the method of the invention after removing abnormal data in the spectrum with the modeling results of the traditional method without removing abnormal values. The samples used in this example are coal powder samples with a particle size of <200um, totaling 121. . Take 5g of powder from each sample and press it into a round cake with a diameter of 30mm and a thickness of 4mm using 30 tons of pressure. The LIBS device uses a 100 mJ pulsed laser (Nd: YAG@1064nm), and the spectrometer detector is a linear array CCD with 4 channels × 2048 pixels. The number of spectra collected for each sample is 200. .
[0019] (2) Spectral preprocessing was performed on the spectral data of each sample collected, including effective data screening of 200 spectral data of each sample. After screening, the remaining spectrum of each sample was about 68% of the original data volume. Subsequently, background elimination and spectral normalization were performed after screening. Finally, the preprocessed spectral data were averaged. A total of 121 sets of spectral data were finally obtained for 121 samples. .
[0020] (1) (3) Using the principal component analysis (PCA) method, Perform dimensionality reduction analysis to determine the number of principal components when the variance information explanation rate of the principal component to the original data is 97% .
[0021] (4) Based on the principal component analysis in (3), select the first two principal components to draw the principal component score graph, as shown in Figure 1 As shown, the x-axis is the first principal component score, which explains 44.3% of the original data variance, and the y-axis is the second principal component score, which explains 31.4% of the original data variance. According to formulas (2) and (3), the maximum elliptical confidence boundary of the ability to summarize data variance information in the principal component space is calculated. The major axis of the elliptical boundary is , short axis The data within the boundary are considered valid data, and there are 5 data outside the boundary, namely data No. 9, 13, 15, 16 and 110. These 5 data are marked as Class I abnormal data, that is, .
[0022] (2) (3) (5) According to formula (4), the first The distance between the projection point of the data point on the PCA model hyperplane and the origin in the plane , and draw a distance distribution diagram, such as Figure 2 As shown. And according to formula (5), the boundary value under 99% confidence condition is calculated According to the calculation, samples 9, 15, 16, 110 and 111 exceed the boundary value, and these 5 spectral data are judged to be type II abnormal data, that is, .
[0023] (4) (5) (6) According to formula (6), the first The spatial distance between the data point and the hyperplane of the PCA model , and draw a distribution diagram, such as Figure 3 The number of data variables in this example , the number of principal components used , calculate the model degrees of freedom according to equations (7) to (10) , the value of M is [K, 100, ], so M=100, and further calculate the degree of freedom of the data , and get , at this time, the sample that exceeds the threshold is sample 9, so the data point is judged to be type III abnormal data, that is, .
[0024] (6) (7) (8) (9) (7) The calculated abnormal data are eliminated. The elimination rule is based on (11), that is, the abnormal data belonging to both Class I and Class II, numbered 15, 16, and 110, are eliminated, and the abnormal data belonging to Class III, numbered 9, is eliminated, and the spectral data numbered 111 and 13 are retained. The quantitative analysis model of ash content is re-established based on the spectral data after the abnormal data are removed and combined with the PCA-PLS algorithm. The model is as follows: Figure 4 The quantitative analysis model of ash content without removing abnormal data is shown in Figure 5 shown.
[0025] (11) (8) A group of samples outside the model were randomly selected for model prediction performance test, and the quantitative analysis model established in (7) was used to predict the corresponding ash content to verify the accuracy of the model. The comparison of the ash content prediction effect of the model recalibrated by this method and the model not calibrated by this method for the same batch of samples is shown in Table 1, where the root mean square error of prediction (RMSEP), the linear fitting determination coefficient of the prediction result (R 2 ) and mean absolute error (MAE) are calculated using conventional methods.
[0026] In summary, the above are only preferred embodiments of the present invention and are not intended to limit the protection scope of the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.
[0027] Table 1 Comparison of prediction results between the prediction model established by this method and the traditional model
Claims
1. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: The LIBS system was used to collect spectral data. This set of data was used to train and establish a quantitative analysis model. The number of samples used was The number of spectra collected for each sample is .
2. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: The spectral data of each sample collected is spectrally preprocessed, and then the preprocessed spectra are averaged. The spectral matrix after preprocessing of samples for: （1） in Indicates The sample spectrum The pixel intensity value.
3. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: Using principal component analysis (PCA) Perform dimensionality reduction analysis to determine the variance information explanation rate k% of the original data ( ) is .
4. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: Based on the principal component analysis in 3, the first two principal components are selected to calculate the maximum elliptical confidence boundary of the data variance information summary ability in the principal component space, and the major axis of the elliptical confidence boundary and short axis The calculation method of is based on formula (2) and (3). The data within the boundary is considered valid data, and the data outside the boundary is marked as type I abnormal data. , （2） （3） in For the The variance of all data on the principal components, express Distribution confidence value =0.05, is the total number of samples.
5. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: As shown in formula (4), The distance between the projection point of the data point on the PCA model hyperplane and the center point of the data in the hyperplane , and calculate the abnormal data distance judgment threshold , as shown in formula (5), when , it means that the data point is significantly different from other data, and the value is determined to be Class II abnormal data. , （4） （5） in is the total number of samples, is the number of principal components selected or used in the model, For the The variance of all points on the principal components, Indicates The sample in The scores on the principal components are for The average of the principal component scores, Representation distribution The confidence value is =0.
01.
6. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: Calculate the first The spatial distance between the data point and the hyperplane of the PCA model , and calculated according to (7)~(9) ,when , then the data point is judged to be type III abnormal data , （6） （7） （8） （9） in is the number of data points, It represents the residual after the data is explained by the principal components, K is the total number of variables, A is the number of principal components, and M is [K, 100, ], Representation distribution The confidence value is =0.
01.
7. A method for improving the accuracy of LIBS quantitative analysis based on abnormal data detection and removal, characterized in that: The calculated abnormal data belonging to both Class I and Class II, and Class III Remove it and get a new spectral data matrix Establish a quantitative analysis model. （10）。