Infrared spectroscopy training set sample screening method

Abstract

The present invention relates to the field of infrared spectrum data processing technology. In order to improve the prediction accuracy of a quantitative model, a method for screening infrared spectrum training set samples is provided, comprising: 1. performing central processing on calibrated spectrum sample data; 2. obtaining the projection value of each spectrum sample data on the principal component coordinate axis; 3. dividing the spectrum sample data into four sets according to the positive and negative properties of the projection value and the calibration value; 4. obtaining the intersection of the four sets respectively; 5. judging the correlation characteristics between the projection value and the calibration value according to the number of samples in the four intersections; 6. determining a normal sample set and an abnormal sample set according to the correlation characteristics; 7. calculating the Mahalanobis distance corresponding to each sample spectrum value in the abnormal sample set; 8. setting a Mahalanobis distance threshold, and removing samples whose Mahalanobis distance exceeds the threshold, and all remaining samples are used as training set samples. The above steps can improve the prediction accuracy of the quantitative model.

Description

Technical Field

[0001] The invention relates to the technical field of infrared spectrum data processing, in particular to a method for screening infrared spectrum training set samples. Background Art

[0002] Near-infrared spectroscopy is widely used in the detection of material components. It has the characteristics of rapidity, non-destructiveness, high efficiency and low cost. Portable near-infrared spectrometer refers to a near-infrared analysis instrument that can be handheld or portable. This type of near-infrared spectrometer is light, easy to carry, low power consumption, fast detection speed and low cost. It is widely used in many fields such as agriculture, medicine, geology and environment. However, portable near-infrared spectrometers are easily affected by light sources, detectors, detection methods, environmental conditions, etc., resulting in poor stability and low precision of the collected spectral data, which in turn affects its spectral prediction and analysis capabilities. In actual application, the spectral data collected by portable near-infrared spectroscopy equipment is prone to abnormalities, and portable near-infrared spectral analysis technology is easily affected by abnormal spectral data, thereby greatly reducing its prediction and analysis capabilities. At the same time, when using a portable spectrometer to collect spectral samples, due to manual operation, sample status, instrument status and other reasons, there will be a small number of abnormal samples in the obtained spectral sample set. When the spectral sample set containing abnormal samples is directly used as a training set to establish a quantitative model, the model's own performance and prediction ability will be reduced. Summary of the invention

[0003] In order to improve the prediction accuracy of the quantitative model, the present application provides a method for screening samples of an infrared spectroscopy training set.

[0004] The technical solution adopted by the present invention to solve the above problems is:

[0005] The infrared spectrum training set sample screening method includes:

[0006] Step 1: Centralize the calibrated spectral sample data;

[0007] Step 2, obtaining the projection value of each spectral sample data on the principal component coordinate axis;

[0008] Step 3: Divide the spectral sample data into a positive projection value set and a negative projection value set according to the positive or negative nature of the projection value; divide the spectral sample data into a positive calibration value set and a negative calibration value set according to the positive or negative nature of the calibration value after the centralization processing;

[0009] Step 4, respectively obtain the intersection of four sets: the intersection of positive projection value and positive calibration value, the intersection of positive projection value and negative calibration value, the intersection of negative projection value and positive calibration value, and the intersection of negative projection value and negative calibration value;

[0010] Step 5: Determine the correlation characteristics between the projection value and the calibration value according to the number of samples in the four intersections;

[0011] Step 6: Determine a normal sample set and an abnormal sample set according to relevant characteristics;

[0012] Step 7, calculating the Mahalanobis distance corresponding to each sample spectral value in the abnormal sample set;

[0013] Step 8: Set the Mahalanobis distance threshold, and remove the samples whose Mahalanobis distance exceeds the Mahalanobis distance threshold, and use all the remaining samples as training set samples.

[0014] Furthermore, in step 1, the centralization method is: subtract the average spectral data and average calibration value of the training set samples from the spectral data and calibration value of each sample.

[0015] Further, in step 5, if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​> the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are positively correlated; if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​< the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are negatively correlated.

[0016] Furthermore, in step 6, if the projection value and the calibration value of the sample are positively correlated, the normal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values; if the projection value and the calibration value of the sample are negatively correlated, the normal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values.

[0017] Furthermore, in step 8, the number of samples eliminated accounts for 20% of the abnormal sample set.

[0018] Compared with the prior art, the present invention has the following beneficial effects: abnormal sample points are determined according to the correlation characteristics between the projection value and the calibration value of the sample data, and the accuracy of the spectral data in the training set is ensured by eliminating the abnormal sample points, thereby improving the accuracy of the model training and further improving the prediction accuracy of the quantitative model. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 Flow chart of the method for screening samples of infrared spectroscopy training set;

[0020] Figure 2is a flowchart corresponding to the embodiment;

[0021] Figure 3 Schematic diagram of four subsets and their intersection. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0023] like Figure 1 As shown, the infrared spectrum training set sample screening method includes:

[0024] Step 1: Centralize the calibrated spectral sample data;

[0025] Step 2, obtaining the projection value of each spectral sample data on the principal component coordinate axis;

[0026] Step 3: Divide the spectral sample data into a positive projection value set and a negative projection value set according to the positive or negative nature of the projection value; divide the spectral sample data into a positive calibration value set and a negative calibration value set according to the positive or negative nature of the calibration value after the centralization processing;

[0027] Step 4, respectively obtain the intersection of four sets: the intersection of positive projection value and positive calibration value, the intersection of positive projection value and negative calibration value, the intersection of negative projection value and positive calibration value, and the intersection of negative projection value and negative calibration value;

[0028] Step 5: Determine the correlation characteristics between the projection value and the calibration value according to the number of samples in the four intersections;

[0029] Step 6: Determine a normal sample set and an abnormal sample set according to relevant characteristics;

[0030] Step 7, calculating the Mahalanobis distance corresponding to each sample spectral value in the abnormal sample set;

[0031] Step 8: Set the Mahalanobis distance threshold, and remove the samples whose Mahalanobis distance exceeds the Mahalanobis distance threshold, and use all the remaining samples as training set samples.

[0032] Specifically, in step 1, the centralization method is: subtract the average spectral data and average calibration value of the training set samples from the spectral data and calibration value of each sample.

[0033] In step 5, if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​> the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are positively correlated; if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​< the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are negatively correlated. In step 6, if the projection value and the calibration value of the sample are positively correlated, the normal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values; if the projection value and the calibration value of the sample are negatively correlated, the normal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values.

[0034] Preferably, in step 8, the number of samples eliminated accounts for 20% of the abnormal sample set.

[0035] Example

[0036] like Figure 2 As shown, the spectral data of the training set is collected using a portable near-infrared spectrometer with a wavelength range of 1350nm to 2150nm and a resolution of 20nm. Each spectral data contains 300 wavelength points.

[0037] The calibrated spectral sample data is centered to obtain the centralized spectral data X0i and calibration value Y0i: the spectral data Xi and calibration value Yi of each sample are respectively subtracted from the average spectral data value Xmean and the average calibration value Ymean of the training set samples, that is: X0i=Xi–Xmean; Y0i=Yi–Ymean, i=1,2,…,m, where m is the number of samples.

[0038] The quantitative model is constructed using the partial least squares method. During the model building process, each sample needs to be projected onto mutually orthogonal principal component coordinate axes in turn, so that the projection value Xsi of the first principal component can be obtained.

[0039] The training set samples are divided into two subsets using the positivity of the first principal component in the projection value: the positive projection value set P_score_set and the negative projection value set N_score_set; the training set samples are also divided into two subsets using the positivity of the calibration value: the positive calibration value set P_Y_set and the negative calibration value set N_Y_set.

[0040] Get the four intersections of the four subsets respectively, such as Figure 3As shown, the positive calibration value intersection Ps_Py_set of the positive projection value, the negative calibration value intersection Ps_Ny_set of the positive projection value, the positive calibration value intersection Ns_Py_set of the negative projection value, and the negative calibration value intersection Ns_Ny_set of the negative projection value are obtained. The number of samples in the four subsets are num_Ps_Py_set, num_Ps_Ny_set, num_Ns_Ny_set, and num_Ns_Py_set respectively.

[0041] The correlation characteristics between the projection value and the calibration value are judged according to the number of samples in the four intersections. If the number of samples in the positive calibration value intersection of the positive projection value + the number of samples in the negative calibration value intersection of the negative projection value > the number of samples in the negative calibration value intersection of the positive projection value + the number of samples in the positive calibration value intersection of the negative projection value, then the projection value and the calibration value of the sample are positively correlated; if the number of samples in the positive calibration value intersection of the positive projection value + the number of samples in the negative calibration value intersection of the negative projection value < the number of samples in the negative calibration value intersection of the positive projection value + the number of samples in the positive calibration value intersection of the negative projection value, then the projection value and the calibration value of the sample are negatively correlated.

[0042] In order to obtain a more accurate correlation, four correlation characteristics are adopted in this embodiment: the positive and negative relationship between the projection value and the calibration value is positively correlated, negatively correlated, weakly positively correlated, and weakly negatively correlated.

[0043] (1) When the number of samples in the positive calibration value intersection of the positive projection value is greater than the number of samples in the negative calibration value intersection of the positive projection value, and the number of samples in the negative calibration value intersection of the negative projection value is greater than the number of samples in the positive calibration value intersection of the negative projection value, that is, (num_Ps_Py_set > num_Ps_Ny_set) && (num_Ns_Ny_set > num_Ns_Py_set), it is considered that the positive and negative relationship between the projection value and the calibration value of the sample is positively correlated;

[0044] (2) When the number of samples in the positive calibration value intersection of the positive projection value is less than the number of samples in the negative calibration value intersection of the positive projection value, and the number of samples in the negative calibration value intersection of the negative projection value is less than the number of samples in the positive calibration value intersection of the negative projection value, that is, (num_Ps_Py_set < num_Ps_Ny_set) && (num_Ns_Ny_set < num_Ns_Py_set), it is considered that the positive and negative relationship between the projection value and the calibration value of the sample is negatively correlated;

[0045] When the above two characteristics are not met, it is considered that the positive and negative relationship between the projection value and the calibration value of the sample is not obvious, and then the weak correlation relationship between the two is judged.

[0046] (3) When the sum of the number of samples in the intersection of positive projection values ​​and positive calibration values ​​and the number of samples in the intersection of negative projection values ​​and negative calibration values ​​is greater than the sum of the number of samples in the intersection of positive projection values ​​and negative calibration values ​​and the number of samples in the intersection of negative projection values ​​and positive calibration values, that is, when (num_Ps_Py_set+num_Ns_Ny_set)>(num_Ps_Ny_set+num_Ns_Py_set), it is considered that the positive and negative relationship between the projection value and the calibration value of the sample is weakly positively correlated;

[0047] (4) When the sum of the number of samples in the intersection of positive projection values ​​and positive calibration values ​​and the sum of the number of samples in the intersection of negative projection values ​​and negative calibration values ​​is less than the sum of the number of samples in the intersection of positive projection values ​​and negative calibration values ​​and the sum of the number of samples in the intersection of negative projection values ​​and positive calibration values, that is, when (num_Ps_Py_set+num_Ns_Ny_set)<(num_Ps_Ny_set+num_Ns_Py_set), it is considered that the positive and negative relationship between the projection value and the calibration value of the sample is weakly negatively correlated.

[0048] According to the four related characteristics of the projection value and the calibration value, the training set samples are divided into two subsets: the normal sample set (normal_set) and the abnormal sample set (unormal_set). The division rules are as follows:

[0049] (1) When the positive and negative relationship between the projection value and the calibration value of the sample is positively correlated or weakly positively correlated, the normal sample set is the union of the intersection of the positive projection value and the positive calibration value and the intersection of the negative projection value and the negative calibration value, that is, Ps_Py_set∪Ns_Ny_set, and the number of samples is (num_Ps_Py_set+num_Ns_Ny_set). The abnormal set is the union of the intersection of the positive projection value and the negative calibration value and the intersection of the negative projection value and the positive calibration value, that is, Ps_Ny_set∪Ns_Py_set, and the number of samples is (num_Ps_Ny_set+num_Ns_Py_set);

[0050] (2) When the positive and negative relationship between the projection value and the calibration value of the sample is negatively correlated or weakly negatively correlated, the normal set is the union of the intersection of the positive projection value and the negative calibration value and the intersection of the negative projection value and the positive calibration value, that is, Ps_Ny_set∪Ns_Py_set, and the number of samples is (num_Ps_Ny_set+num_Ns_Py_set). The abnormal set is the union of the intersection of the projection value and the positive calibration value and the intersection of the negative projection value and the negative calibration value, that is, Ps_Py_set∪Ns_Ny_set, and the number of samples is (num_Ps_Py_set+num_Ns_Ny_set).

[0051] Calculate the Mahalanobis distance corresponding to each sample spectral value in the abnormal sample set: average the sample spectral data in the abnormal sample set to obtain the average spectrum Xunnormal of the abnormal sample set mean , so that the Mahalanobis distance between each sample spectrum data and the average spectrum can be calculated, and the calculation formula is:

[0052]

[0053] where X_unnormal j …(j=1,…,p) is the jth sample in the abnormal sample set, P is the number of samples in the abnormal sample set, S is X_unnormal j The covariance matrix of is of size P×P.

[0054] The Mahalanobis distance threshold is set, and the sample points that exceed the threshold range are screened out as abnormal sample points, and all the remaining samples are used as training set samples. Sample points that exceed the threshold range are regarded as edge samples, and their contribution to the model is higher than that of samples with a smaller Mahalanobis distance. After these samples are screened out as abnormal sample points, the model performance can be effectively improved; the threshold is set according to the actual situation, and the number of abnormal points screened out generally accounts for about 20% of the abnormal sample set.

Claims

1. A method for screening infrared spectrum training set samples, characterized in that: include: Step 1: Centralize the calibrated spectral sample data; Step 2, obtaining the projection value of each spectral sample data on the principal component coordinate axis; Step 3: Divide the spectral sample data into a positive projection value set and a negative projection value set according to the positive or negative nature of the projection value; divide the spectral sample data into a positive calibration value set and a negative calibration value set according to the positive or negative nature of the calibration value after the centralization processing; Step 4, respectively obtain the intersection of four sets: the intersection of positive projection value and positive calibration value, the intersection of positive projection value and negative calibration value, the intersection of negative projection value and positive calibration value, and the intersection of negative projection value and negative calibration value; Step 5: Determine the correlation characteristics between the projection value and the calibration value according to the number of samples in the four intersections; Step 6: Determine a normal sample set and an abnormal sample set according to relevant characteristics; Step 7, calculating the Mahalanobis distance corresponding to each sample spectral value in the abnormal sample set; Step 8: Set the Mahalanobis distance threshold, and remove the samples whose Mahalanobis distance exceeds the Mahalanobis distance threshold, and use all the remaining samples as training set samples; In the step 5, if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​> the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are positively correlated; if the number of samples of the intersection of positive projection values ​​and positive calibration values ​​+ the number of samples of the intersection of negative projection values ​​and negative calibration values ​​< the number of samples of the intersection of positive projection values ​​and negative calibration values ​​+ the number of samples of the intersection of negative projection values ​​and positive calibration values, then the projection value and the calibration value of the sample are negatively correlated; In step 6, if the projection value and the calibration value of the sample are positively correlated, the normal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values; if the projection value and the calibration value of the sample are negatively correlated, the normal sample set is: the intersection of positive projection values ​​and negative calibration values ​​∪ the intersection of negative projection values ​​and positive calibration values, and the abnormal sample set is: the intersection of positive projection values ​​and positive calibration values ​​∪ the intersection of negative projection values ​​and negative calibration values.

2. The infrared spectrum training set sample screening method according to claim 1, characterized in that: In step 1, the centralization method is: subtract the average spectral data and average calibration value of the training set samples from the spectral data and calibration value of each sample.

3. The infrared spectrum training set sample screening method according to claim 1 or 2, characterized in that: In step 8, the number of samples removed accounts for 20% of the abnormal sample set.

Images