A spectral preprocessing method based on AdaBoost regression algorithm

Abstract

The application provides a spectrum preprocessing method based on an AdaBoost regression algorithm, comprising the following steps: S1, acquiring an infrared spectrum data set of a sample to be measured; S2, extracting a training set from the acquired spectrum data, repeatedly learning four weak learners based on the AdaBoost regression algorithm; S3, combining the four weak learners to construct a spectrum preprocessing learner; and S4, performing spectrum preprocessing on the acquired infrared spectrum data by the spectrum preprocessing learner constructed in step S3. The application can simultaneously complete spectrum preprocessing steps such as baseline correction, scattering correction and smoothing processing, not only improves the accuracy of the preprocessing task, but also solves the problem of the sequence of the preprocessing algorithms in spectrum preprocessing, and avoids the problem of error accumulation caused by serial operation of various algorithms.

Description

Technical Field

[0001] This application relates to the field of spectral detection technology, and more specifically, to a spectral preprocessing method based on the AdaBoost regression algorithm. Background Technology

[0002] In recent years, spectroscopic detection technology has been widely used due to its rapid, accurate, and non-destructive testing characteristics. Quantitative analysis of spectra is an important aspect of spectroscopic applications. In practical applications, due to factors such as temperature, humidity, and the instrument itself, unprocessed spectra can lead to errors in quantitative analysis and incorrect component predictions. Therefore, spectroscopic preprocessing is a crucial component of spectroscopic analysis. Spectral preprocessing includes baseline correction, scattering correction, and smoothing. Methods involved include Gaussian filtering, SG smoothing, polynomial fitting, and adaptive iterative weighted least squares, but most of these methods can only complete a single preprocessing task. When run sequentially, errors accumulate, distorting the preprocessing results.

[0003] Meanwhile, during the testing process, due to the characteristics of the sample itself and the shape and material of the testing container, the sample may be unevenly distributed within the container, causing severe fluctuations in the spectral curve. Therefore, a preprocessing method to mitigate these spectral fluctuations is needed. Summary of the Invention

[0004] In view of this, this application provides a spectral preprocessing method based on the AdaBoost regression algorithm, which adjusts the weights of each processing step through continuous training to achieve the best processing effect and thus improve the jitter of the detected spectrum.

[0005] To achieve the above objectives, the technical solution adopted in this application is as follows: A spectral preprocessing method based on the AdaBoost regression algorithm includes: S1: Obtain the infrared spectrum dataset of the sample to be tested; S2: Extract the training set from the acquired spectral data, and repeatedly learn to obtain 4 weak learners based on the AdaBoost regression algorithm; S3: Combine the four weak learners to construct a spectral preprocessing learner. S4: The spectral preprocessing learner constructed in step S3 is used to preprocess the acquired infrared spectral data.

[0006] Furthermore, in step S2, four weak learners are obtained through repeated learning based on the AdaBoost regression algorithm, specifically as follows: S2.1: First, extract the training set. ,in ,and i=1,2,…, N ; For the infrared spectral dataset obtained in step S1, set the initial weights for each training sample. ; S2.2: Based on the training set Train the first weak learner And calculate the weak learner The relative error of prediction : , This represents the preset relative error threshold. It is a global normalization factor; S2.3: Based on weak learners The relative error of the prediction is used to update the weak learner. The weights of the final strong learner ; S2.4: Update the weights of all samples to make the weak learner The training samples with high relative error are given higher weights in the next step of the weak learner. More attention is being paid to it, and the weight update formula is: in For normalized weights; S2.5: Train a weak learner based on the training set with adjusted weights. This process is repeated until the weak learner error reaches a pre-specified threshold or the number of iterations reaches a threshold. Finally, the T trained weak learners are integrated using an ensemble strategy to obtain the final strong learner output. .

[0007] Furthermore, the four weak learners in step S2 are used to perform baseline correction, scattering correction, smoothing, and scaling of the spectral data, respectively.

[0008] Furthermore, the weak learner uses continuous wavelet transform (CWT) for baseline correction, the basic formula of which is: In the formula, The input signal is t, which can be considered a spectral signal. t is the time-domain signal. It can be viewed as wavelet basis functions. and For translation parameters, These are wavelet coefficients.

[0009] Furthermore, the weak learner uses multivariate scattering correction (MSC) for scattering correction, specifically: (4) First, calculate the average spectrum of the validation set: (5) Then, perform a univariate linear regression on the spectrum of each sample and the average spectrum to solve the least squares problem and obtain the baseline shift and offset for each sample: in, Represents the spectrum of each sample. For each spectrum, the amount of translation. The offset for each spectrum; (6) Subtract the calculated baseline shift from the spectrum of each sample and divide by the offset to obtain the corrected spectrum: .

[0010] Furthermore, the weak learner is smoothed using the SG smoothing method.

[0011] Furthermore, the weak learner uses Pareto scaling for scaling.

[0012] Compared with the prior art, the beneficial effects of this application are: 1. This application can simultaneously complete spectral preprocessing steps such as baseline correction, scattering correction and smoothing, which not only improves the accuracy of the preprocessing task, but also solves the problem of the order of these preprocessing algorithms in spectral preprocessing, avoiding the problem of error accumulation caused by the serial running of various algorithms.

[0013] 2. At the same time, this method can adjust the weights of each processing step through continuous training to achieve the best processing effect, thereby improving the jitter of the detection spectrum. Attached Figure Description

[0014] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0015] Figure 1 This is a flowchart of a spectral preprocessing method based on the AdaBoost regression algorithm proposed in this application.

[0016] Figure 2 This is a flowchart illustrating the AdaBoost regression algorithm used in this application.

[0017] Figure 3 This is a comparison chart of the spectral preprocessing effects of our method and general processing methods. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are some embodiments of this application, but not all embodiments.

[0019] like Figure 1 As shown, a spectral preprocessing method based on the AdaBoost regression algorithm includes: S1: Obtain the infrared spectrum dataset of the sample to be tested; The infrared spectral dataset mentioned in this paper specifically refers to the set of raw spectral data obtained from the test sample by an infrared spectrometer. In this embodiment, a mixed emulsion of diesel and water was used as the test sample. A total of 100 emulsion samples with different water contents, ranging from 0.5% to 20.0%, were prepared. Infrared spectral data were collected three times for each sample, resulting in 300 raw infrared spectral curves. The infrared spectral acquisition range was set to 4000–10000 cm⁻¹, with a sampling interval of 4 cm⁻¹. Each spectral curve contained 1501 wavenumber sampling points, and each sampling point corresponded to an absorbance value.

[0020] S2: Extract the training set from the acquired spectral data, and repeatedly learn to obtain 4 weak learners based on the AdaBoost regression algorithm; Furthermore, such as Figure 2 As shown, in step S2, four weak learners are obtained through repeated learning based on the AdaBoost regression algorithm: S2.1: First, extract the training set. ,in ,and i= 1,2,…, N ; For the infrared spectral dataset obtained in step S1, set the initial weights for each training sample. ; S2.2: Based on the training set Train the first weak learner And calculate the weak learner The relative error of prediction : , This represents the preset relative error threshold, which, in the formula, is used to determine the prediction error of the weak learner on the sample. Is the error too large? If the error is greater than... This sample will be considered a "high prediction error" sample and given greater weight in subsequent iterations. y is a global normalization factor, the average of the absolute values ​​of all observations (or the target value), used to normalize the calculation of the relative error. I represents the indicator function; when I is greater than... When the function outputs 1, the function output is 1.

[0021] S2.3: Based on weak learners The relative error of the prediction is used to update the weak learner. The weights of the final strong learner ; S2.4: Update the weights of all samples to make the weak learner The training samples with high relative error are given higher weights in the next step of the weak learner. More attention is being paid to it, and the weight update formula is: in For normalized weights; S2.5: Train a weak learner based on the training set with adjusted weights. This process is repeated until the weak learner error reaches a pre-specified threshold or the number of iterations reaches a threshold. Finally, the T trained weak learners are integrated using an ensemble strategy to obtain the final strong learner output. .

[0022] in This can be seen as the degree of distrust towards weak learners. The smaller, The smaller the value, the greater the trust in the weak learner, and this is also utilized when updating sample weights. The characteristics of this feature allow sample points with prediction errors exceeding a certain threshold to be assigned larger weights, making these sample points receive more attention in subsequent iterations. Through iteration, the training error of the final strong learner gradually decreases.

[0023] The prediction relative error of each weak learner is calculated by comparing its output with the corresponding preprocessed reference result and then calculating the normalized relative error. The weight of each weak learner in the strong learner is determined based on its prediction error; the smaller the error, the larger the corresponding weight; and vice versa. The four weak learners are: f0 is the baseline correction weak learner, using continuous wavelet transform (CWT); f1 is the scattering correction weak learner, using multivariate scattering correction (MSC); f2 is the smoothing weak learner, using SG smoothing; and f3 is the scaling weak learner, using Pareto scaling.

[0024] Furthermore, the four weak learners in step S2 are used to perform baseline correction, scattering correction, smoothing, and scaling of the spectral data, respectively.

[0025] Furthermore, the weak learner uses continuous wavelet transform (CWT) for baseline correction, the basic formula of which is: In the formula, The input signal is t, which can be considered a spectral signal. t is the time-domain signal. It can be viewed as a wavelet basis function. and For translation parameters, These are wavelet coefficients.

[0026] Furthermore, the weak learner uses multivariate scattering correction (MSC) for scattering correction, specifically: (7) First, calculate the average spectrum of the validation set: The validation set was selected by dividing all samples into a training set and a validation set in a 7:3 ratio. (8) Then, perform a univariate linear regression on the spectrum of each sample and the average spectrum to solve the least squares problem and obtain the baseline shift and offset for each sample: in, Represents the spectrum of each sample. For each spectrum, the amount of translation. The offset for each spectrum; (9) Subtract the calculated baseline shift from the spectrum of each sample and divide by the offset to obtain the corrected spectrum: .

[0027] Furthermore, the weak learner is smoothed using the SG smoothing method.

[0028] Furthermore, the weak learner uses Pareto scaling for scaling.

[0029] Pareto scaling is a data standardization method that involves subtracting the mean from each variable and then dividing by the square root of its standard deviation. Compared to unit variance standardization, Pareto scaling does not excessively amplify the influence of small variance variables, and can reduce the impact of dimensional and amplitude differences on model training while preserving the relative differences of the original spectral variables. Its formula is as follows:

[0030] ,in Let be the spectral value of the i-th sample at the j-th wavenumber point. Let j be the mean of the wavenumber points. Let be the standard deviation of the j-th wavenumber point.

[0031] S3: Combine the four weak learners to construct a spectral preprocessing learner. S4: The spectral preprocessing learner constructed in step S3 is used to preprocess the acquired infrared spectral data.

[0032] like Figure 3 As shown, curve 1 is the spectral curve obtained by a general spectral preprocessing method, and the other curves are the spectral curves obtained by this method. At a wavenumber of 6000... Curve 1, where the absorption peak should have appeared, exhibited severe jitter and distortion. The curve obtained by this method, however, simultaneously performs baseline correction, scattering correction, and smoothing—spectral preprocessing steps—not only improving the accuracy of the preprocessing task but also resolving the issue of the order of these preprocessing algorithms, thus avoiding the error accumulation caused by sequential execution of various algorithms. Furthermore, this method can continuously adjust the weights of each processing step through training to achieve the best processing results, yielding a spectral curve with significantly reduced spectral jitter and presenting accurate absorption peaks.

[0033] This application provides a spectral preprocessing method for spectral jitter based on the AdaBoost regression algorithm. This method is based on the AdaBoost regression algorithm, an ensemble learning method where individual learners have strong dependencies on each other. Its adaptability lies in the fact that samples with poor training performance of the previous base learner are strengthened, and the weighted entire sample set is used again to train the next base learner. Simultaneously, a new weak learner is added in each training round until a predetermined sufficiently small error rate is reached or a pre-specified maximum number of iterations is reached, at which point training terminates. This method can simultaneously complete spectral preprocessing steps such as baseline correction, scattering correction, and smoothing, not only improving the accuracy of the preprocessing task but also resolving the issue of the order of these preprocessing algorithms in spectral preprocessing, avoiding the error accumulation caused by sequential execution of various algorithms. Furthermore, this method can continuously adjust the weights of each processing step through training to achieve the best processing effect, thereby improving the detection of spectral jitter.

[0034] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A spectral preprocessing method based on the AdaBoost regression algorithm, characterized in that, include: S1: Obtain the infrared spectrum dataset of the sample to be tested; S2: Extract the training set from the acquired spectral data, and repeatedly learn to obtain 4 weak learners based on the AdaBoost regression algorithm; S3: Combine the four weak learners to construct a spectral preprocessing learner; S4: The spectral preprocessing learner constructed in step S3 is used to preprocess the acquired infrared spectral data.

2. The spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 1, characterized in that, In step S2, the four weak learners obtained through repeated learning based on the AdaBoost regression algorithm are as follows: S2.1: First, extract the training set. ,in ,and i= 1,2,…, N ; For the infrared spectral dataset obtained in step S1, set the initial weights for each training sample. ; S2.2: Based on the training set Train the first weak learner And calculate the weak learner The relative error of prediction : ， This represents the preset relative error threshold. It is a global normalization factor; S2.3: Based on weak learners The relative error of the prediction is used to update the weak learner. In the weights of the final strong learner ; S2.4: Update the weights of all samples so that the weak learner The training samples with high relative error are given higher weights in the next step of the weak learner. More attention is being paid to it, and the weight update formula is as follows: in For normalized weights; S2.5: Train a weak learner based on the training set with adjusted weights. This process is repeated until the weak learner error reaches a pre-specified threshold or the number of iterations reaches a threshold. Finally, the T trained weak learners are integrated using an ensemble strategy to obtain the final strong learner output. 。 3. A spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 1 or 2, characterized in that, The four weak learners in step S2 are used to perform baseline correction, scattering correction, smoothing, and scaling of the spectral data, respectively.

4. The spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 3, characterized in that, The weak learner uses continuous wavelet transform (CWT) for baseline correction, and its basic formula is as follows: In the formula, The input signal is t, which can be considered a spectral signal. t is the time-domain signal. It can be viewed as a wavelet basis function. and For translation parameters, These are wavelet coefficients.

5. The spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 3, characterized in that, The weak learner uses multivariate scattering correction (MSC) for scattering correction, specifically: (1) First, calculate the average spectrum of the validation set: ; (2) Then, perform a univariate linear regression on the spectrum of each sample and the average spectrum to solve the least squares problem and obtain the baseline shift and offset for each sample: ; in, Represents the spectrum of each sample. For each spectrum, the amount of translation, The offset for each spectrum; (3) Subtract the calculated baseline shift from the spectrum of each sample and divide by the offset to obtain the corrected spectrum: .

6. The spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 3, characterized in that, The weak learner is smoothed using the SG smoothing method.

7. The spectral preprocessing method based on the AdaBoost regression algorithm as described in claim 3, characterized in that, The weak learner uses Pareto scaling for scaling.

Images