Automatic preprocessing method for spectral data
By constructing six preprocessing algorithm libraries for spectral data and genetic algorithm optimization, the optimal preprocessing method is automatically selected, which solves the problem of time-consuming and labor-consuming preprocessing of existing spectral data, and realizes efficient and accurate spectral data preprocessing, and improves the performance of the spectral prediction model.
Patent Information
- Application Number
- CN202510452317.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-04-11
- Publication Date
- 2025-08-22
AI Technical Summary
The existing spectral data preprocessing methods are time-consuming and labor-intensive, and it is difficult to find the best combination of preprocessing algorithms, resulting in poor preprocessing effects and affecting the accuracy of the spectral prediction model.
An algorithm library containing six preprocessing algorithms is built. Through a five-step preprocessing process combined with genetic algorithm optimization, the optimal preprocessing function combination is determined, and the genetic algorithm and variable selection algorithm are combined to automatically select the optimal preprocessing method.
It improves the timeliness and accuracy of spectral data preprocessing, reduces subjective bias of artificial selection, simplifies operational steps, improves the accuracy of regression and classification tasks, and avoids redundant processing and overfitting.
Smart Images
Figure CN120524079A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to an automatic preprocessing method, in particular to an automatic preprocessing method for spectral data. Background Art
[0002] Spectroscopic technology is widely used in a variety of fields, including food, medicine, agriculture, and chemistry. Spectral prediction models are used to predict the spectral characteristics (i.e., spectral data) of samples and analyze their composition. However, spectral data is often affected by various noise and unwanted signal interference, such as ambient light and baseline drift. These interferences can reduce the accuracy of spectral data, resulting in suboptimal training accuracy of prediction models and, consequently, suboptimal prediction results. Therefore, preprocessing of spectral data is essential before analysis.
[0003] Traditional spectral data preprocessing algorithms usually include smoothing algorithms, normalization algorithms, detrending algorithms, and baseline removal algorithms, etc. These preprocessing algorithms each have multiple corresponding preprocessing functions. However, due to the large differences in the effects between different preprocessing algorithms, the effects of the same preprocessing algorithm using different preprocessing functions also vary. In addition, due to the diversity of application scenarios, the actual spectral data preprocessing effect also depends on the specific application scenario. In order to improve the spectral data preprocessing effect, a combination of multiple preprocessing algorithms is usually used for preprocessing. However, current researchers mainly determine the combination of preprocessing algorithms through trial and error or empirical judgment, which is not only time-consuming and labor-intensive, but also difficult to find the optimal combination of preprocessing algorithms. As a result, the existing preprocessing methods have poor timeliness and high labor intensity, and there is still a lot of room for improvement in the preprocessing effect. Summary of the Invention
[0004] The technical problem to be solved by the present invention is to provide an automatic preprocessing method for spectral data which has high timeliness, low labor intensity, can find the best preprocessing method combination and has good preprocessing effect.
[0005] The technical solution adopted by the present invention to solve the above technical problems is: a method for automatic preprocessing of spectral data, first constructing an algorithm library containing six preprocessing algorithms, the six preprocessing algorithms are: smoothing algorithm, baseline correction algorithm, scattering correction algorithm, data centering algorithm, data normalization algorithm and variable selection algorithm, each preprocessing algorithm has multiple corresponding preprocessing functions; then constructing a preprocessing process containing five steps, the five steps of the preprocessing process are: processing using a smoothing algorithm, processing using a baseline correction algorithm, processing using a scattering correction algorithm, processing using a data centering algorithm and processing using a data normalization algorithm The spectral data are processed, and then the genetic algorithm is used to optimize the preprocessing process, and the optimal preprocessing function of each preprocessing algorithm in the preprocessing process is determined to obtain the optimal preprocessing process; then, the optimal preprocessing process is first used to perform preliminary preprocessing on the spectral data to obtain preliminary preprocessed spectral data, and then the variable selection algorithm is used to preprocess the preliminary preprocessed spectral data under each preprocessing function of the variable selection algorithm to determine the optimal preprocessing function of the variable selection algorithm; the variable selection algorithm preprocesses the preliminary preprocessed spectral data under its optimal preprocessing function, and the obtained spectral data is the final preprocessed spectral data.
[0006] Compared with the prior art, the advantage of the present invention is that after constructing a preprocessing process consisting of five steps: processing with a smoothing algorithm, processing with a baseline correction algorithm, processing with a scattering correction algorithm, processing with a data centralization algorithm and processing with a data normalization algorithm, a genetic algorithm is used to optimize the preprocessing process, and the optimal preprocessing function of each preprocessing algorithm in the preprocessing process is determined, and the optimal preprocessing process in which the preprocessing algorithms of the five steps are all their optimal preprocessing functions is obtained, and then the optimal preprocessing function of the variable selection algorithm is determined, and the optimal preprocessing process and the preprocessing process consisting of the optimal preprocessing process and the variable selection algorithm using the optimal preprocessing function are used as the optimal preprocessing method, and the optimal preprocessing method is used to preprocess the spectral data to obtain the preprocessing function. For the processed spectral data, first, the genetic algorithm and the variable selection algorithm are combined to simplify the preprocessing steps through the automated preprocessing selection process and combination strategy, which greatly reduces the need for manual intervention. This not only improves the operational efficiency, but also avoids the deviation caused by the subjectivity of human selection. Secondly, the genetic algorithm and the variable selection algorithm are used to optimize the selection of preprocessing function combinations, and the optimal preprocessing function combination is selected from a variety of preprocessing function combinations in a systematic and intelligent manner, thereby obtaining better prediction performance. This intelligent selection not only improves the accuracy in regression and classification tasks, but also reduces redundant processing and avoids overfitting or computational burden. Therefore, the present invention has high timeliness, low labor intensity, and can find the optimal combination of preprocessing methods, with good preprocessing effect.
[0007] Furthermore, a genetic algorithm is used to optimize the preprocessing process and determine the optimal preprocessing function of the preprocessing algorithm in each step. The specific steps of the optimized preprocessing process are:
[0008] Step S1, randomly generating a plurality of different preprocessing processes to constitute a contemporary seed, wherein the preprocessing algorithm of each step of each preprocessing process has a preprocessing function;
[0009] Step S2: preprocess the spectral data that need to be preprocessed using each preprocessing process in the contemporary seed, and evaluate each preprocessing process in the contemporary seed using the fitness function to screen out the optimal contemporary preprocessing process;
[0010] Step S3: Perform crossover and mutation operations on the current optimal preprocessing process to generate a new preprocessing process to form a new seed, and update the current seed to the new seed;
[0011] Step S4: Repeat steps S2 and S3 until the genetic algorithm converges to obtain the optimal preprocessing process.
[0012] Furthermore, in step S2, the fitness function models the spectral data obtained from each preprocessing process in the contemporary seeds, calculates the model accuracy by cross-validation, and evaluates each preprocessing process in the contemporary seeds. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 A flow chart of a method for automatic preprocessing of spectral data according to the present invention;
[0014] Figure 2 A schematic diagram of an algorithm library for an automatic spectral data preprocessing method of the present invention;
[0015] FIG3( a ) is a prediction curve diagram of a prediction model obtained by preprocessing wheat spectral data using an automatic spectral data preprocessing method of the present invention (based on a wheat spectral dataset);
[0016] FIG3( b ) is a prediction curve diagram of a prediction model obtained by preprocessing meat spectral data using an automatic spectral data preprocessing method of the present invention (based on a meat spectral dataset);
[0017] FIG4( a ) is a time curve diagram of an automatic spectral data preprocessing method of the present invention (based on a wheat spectral dataset);
[0018] FIG4( b ) is a time curve diagram of an automatic spectral data preprocessing method of the present invention (based on a meat spectral dataset);
[0019] Figure 5A classification result diagram of a classification model obtained by modeling barley spectral data before preprocessing using an automatic spectral data preprocessing method of the present invention (before preprocessing, based on barley spectral data) and a classification result diagram of a classification model obtained by modeling barley spectral data after preprocessing using an automatic spectral data preprocessing method of the present invention (after preprocessing, based on a barley spectral dataset);
[0020] FIG6( a ) is a principal component analysis diagram of spectral data before preprocessing using an automatic spectral data preprocessing method of the present invention (before preprocessing, based on a barley spectral dataset);
[0021] FIG6( b ) is a principal component analysis diagram of spectral data after preprocessing using an automatic spectral data preprocessing method of the present invention (after preprocessing, based on a barley spectral dataset). DETAILED DESCRIPTION
[0022] The present invention will be described in further detail below with reference to the accompanying drawings and embodiments.
[0023] Example 1: Figure 1 and Figure 2 As shown in the figure, an automatic preprocessing method for spectral data is first constructed, which includes six preprocessing algorithms: smoothing algorithm, baseline correction algorithm, scattering correction algorithm, data centering algorithm, data normalization algorithm and variable selection algorithm. Each preprocessing algorithm has multiple corresponding preprocessing functions; then a preprocessing process including five steps is constructed, which includes the following steps: processing with smoothing algorithm, processing with baseline correction algorithm, processing with scattering correction algorithm, processing with data centering algorithm and processing with data normalization algorithm. Then, a preprocessing process including five steps is constructed. The preprocessing process is optimized using a genetic algorithm to determine the optimal preprocessing function for each preprocessing algorithm within the preprocessing process, resulting in the optimal preprocessing process. The optimal preprocessing process is then used to preprocess the spectral data, obtaining preprocessed spectral data. The preprocessed spectral data are then preprocessed using the variable selection algorithm under each preprocessing function, determining the optimal preprocessing function for the variable selection algorithm. The variable selection algorithm then preprocesses the preprocessed spectral data under its optimal preprocessing function, resulting in the final preprocessed spectral data. At this point, the optimal preprocessing process and variable selection algorithm constitute the optimal preprocessing strategy.
[0024] In this embodiment, a preprocessing process consisting of five steps, namely, processing using a smoothing algorithm, processing using a baseline correction algorithm, processing using a scattering correction algorithm, processing using a data centralization algorithm, and processing using a data normalization algorithm, is first constructed. A genetic algorithm is used to optimize the preprocessing process, and the optimal preprocessing function of each preprocessing algorithm in the preprocessing process is determined. An optimal preprocessing process in which the preprocessing algorithms of the five steps are all their optimal preprocessing functions is obtained. Then, the optimal preprocessing function of the variable selection algorithm is determined. The optimal preprocessing process and the preprocessing process consisting of the variable selection algorithm using the optimal preprocessing function are used as the optimal preprocessing method. The optimal preprocessing method is used. The method preprocesses the spectral data to obtain preprocessed spectral data. First, the genetic algorithm and the variable selection algorithm are combined to simplify the preprocessing steps through automated preprocessing selection processes and combination strategies, greatly reducing the need for manual intervention. This not only improves operational efficiency, but also avoids the deviation caused by the subjectivity of human selection. Secondly, the genetic algorithm and the variable selection algorithm are used to optimize the selection of preprocessing function combinations, and the optimal preprocessing function combination is selected from a variety of preprocessing function combinations in a systematic and intelligent manner, thereby obtaining better prediction performance. This intelligent selection not only improves the accuracy in regression and classification tasks, but also reduces redundant processing and avoids overfitting or computational burden.
[0025] Example 2: This example is basically the same as Example 1, except that: in this example, a genetic algorithm is used to optimize the preprocessing process, and the optimal preprocessing function of the preprocessing algorithm in each step is determined. The specific steps of optimizing the preprocessing process are as follows:
[0026] Step S1, randomly generating a plurality of different preprocessing processes to constitute a contemporary seed, wherein the preprocessing algorithm of each step of each preprocessing process has a preprocessing function;
[0027] Step S2: preprocess the spectral data that need to be preprocessed using each preprocessing process in the contemporary seed, and evaluate each preprocessing process in the contemporary seed using the fitness function to screen out the optimal contemporary preprocessing process;
[0028] Step S3: Perform crossover and mutation operations on the current optimal preprocessing process to generate a new preprocessing process to form a new seed, and update the current seed to the new seed;
[0029] Step S4: Repeat steps S2 and S3 until the genetic algorithm converges to obtain the optimal preprocessing process.
[0030] In this embodiment, in step S2, the fitness function models the spectral data obtained by each preprocessing process in the contemporary seed, calculates the model accuracy by a cross-validation method, and evaluates each preprocessing process in the contemporary seed.
[0031] To verify the performance of our automated spectral data preprocessing method, we implemented it in MATLAB 2023 and conducted an experimental validation. We used a publicly available spectral dataset as the experimental basis to ensure an objective and fair performance evaluation of our automated spectral data preprocessing method.
[0032] For the regression task, we selected two public spectral data sets (wheat and meat) that are suitable for regression tasks. The Duplex algorithm was used to divide each spectral data set into a test set and a training set, with a ratio of 75% and 25%. The Duplex algorithm was selected to ensure that both the training set and the test set can represent the sample distribution of the entire spectral data set, so as to more accurately evaluate the performance of the automatic preprocessing method for spectral data of the present invention. First, we used the optimal preprocessing method of the present invention to preprocess the training set of each spectral data set to obtain preprocessed spectral data. Then, the preprocessed spectral data was modeled using the partial least squares regression (PLSR) method to obtain an attribute prediction model. Next, the attribute prediction model was used to predict the test set, and the root mean square error (RMSEP) of each spectral data in the test set was calculated to evaluate the performance of the automatic preprocessing method for spectral data of the present invention (abbreviated as AutoPF).
[0033] For the classification task, we selected a public spectral dataset suitable for classification. Then, we used the Duplex algorithm to divide the spectral dataset into a test set and a training set, with a ratio of 75% and 25%. The Duplex algorithm was selected to ensure that both the training set and the test set can represent the sample distribution of the entire spectral dataset, so as to more accurately evaluate the performance of the automatic preprocessing method for spectral data of the present invention. Based on the preprocessed spectral data of the training set, we used a 4-fold cross-validation method and a support vector machine (SVM) classification algorithm to establish a classification model. Then, the classification model was used to predict the test set, and the classification accuracy of the test set was calculated to evaluate the performance of the automatic preprocessing method for spectral data of the present invention (abbreviated as AutoPF).
[0034] Specifically, for regression tasks, we use the wheat spectral dataset (http: / / www.models.life.ku.dk / ) and the meat spectral dataset (http: / / lib.stat.cmu.edu / datasets / tecator). For classification tasks, we use the barley spectral dataset, which can be downloaded from: https: / / doi.org / 10.1371 / journal.pone.0193620.s003.
[0035] We also evaluated existing preprocessing methods on the wheat and meat spectral datasets. Table 1 shows the performance comparison between the automatic spectral data preprocessing method of the present invention and existing preprocessing methods on the wheat and meat spectral datasets.
[0036] Table 1 Performance comparison data of preprocessing methods
[0037]
[0038] In Table 1, AutoPF indicates that the spectral data are pre-processed and then modeled using the automatic pre-processing method of the present invention, Wheat indicates a wheat spectral dataset, Meat indicates a meat spectral dataset, None indicates that no pre-processing is performed and the original spectral data in the spectral dataset is directly used for modeling, LV indicates the number of potential variables in the partial least squares model, RMSECV indicates the root mean square error of the cross-validation, SG-WDO indicates Savitzky-Golay smoothing, using a window size of W, D-order derivatives, and an O-order polynomial, such as SG-15-0-2 in the table, which indicates using a window size of 15, a 0-order derivative, and a second-order SG smoothing. MAF indicates moving average filtering, CWT indicates continuous wavelet transform, Detrend indicates a detrending algorithm, AsLS indicates iterative least squares, SNV indicates standard normal variable transformation, MSC indicates multivariate scatter correction, EMSC indicates extended multivariate scatter correction, STACKED indicates a stacked ensemble algorithm, and SPORT indicates orthogonalized sequential pre-processing. STACKED represents the preprocessing method disclosed in the paper "Sequential preprocessing through ORThogonalization and its application to near infrared spectroscopy", and SPORT represents the preprocessing method disclosed in the related link https: / / www.sciencedirect.com / science / article / pii / S0169743919308135.
[0039] Analysis of the data in Table 1 shows that if modeling is performed directly without preprocessing, the results of the wheat spectral dataset and the meat spectral dataset are not ideal. Especially for the meat spectral dataset, the RMSECV and RMSEP errors of the corresponding models are large. At the same time, the improvement brought about by using a single preprocessing method is usually not obvious. This is because a certain spectral dataset may contain various types of information interference, and these interferences cannot be removed by a single preprocessing. Usually, a certain preprocessing method is designed to remove specific interference. For the wheat spectral dataset, EMSC can effectively remove irrelevant information, and its RMSECV and RMSEP values are even lower than SPORT and STACKED. This may be because EMSC effectively removes most of the irrelevant information in the wheat spectral data. Therefore, the automatic preprocessing method of spectral data of the present invention does not have much improvement compared with EMSC. On the other hand, EMSC produces a higher RMSEP value on the meat spectral dataset. This shows that due to the different characteristics of spectral data, different preprocessing methods are needed, which also highlights the value of this method.
[0040] A prediction curve diagram of a prediction model obtained by preprocessing wheat spectral data using an automatic spectral data preprocessing method of the present invention (based on a wheat spectral dataset) is shown in FIG3(a); a prediction curve diagram of a prediction model obtained by preprocessing meat spectral data using an automatic spectral data preprocessing method of the present invention (based on a meat spectral dataset) is shown in FIG3(b);
[0041] For the wheat spectral dataset, as shown in Figure 3(a), most curves show a steady decline after the relevant preprocessing. This is likely due to the removal of irrelevant information from the near-infrared spectra, making it easier to build a regression model. Among the various preprocessing methods, EMSC performs the best, approaching AutoPF. However, EMSC appears unable to completely remove irrelevant information from the raw spectra. Therefore, it requires more latent variables to match the performance of AutoPF. On the other hand, AutoPF can remove various types of irrelevant information and requires fewer latent variables for modeling. A similar trend can be observed in the meat spectral data in Figure 3(b). This shows that the performance of a single preprocessing method is generally inferior to that of a combination of preprocessing methods. As the LV value increases from 1 to 10, the curves have very similar shapes, especially the two curves corresponding to SG-15-0-2 and None. For these two methods, their corresponding RMSECV curves are almost identical.
[0042] To demonstrate the effectiveness of the automatic spectral data preprocessing method of the present invention, we plotted the RMSECV versus computation time for the automatic spectral data preprocessing method of the present invention using different LV values. A time curve for one automatic spectral data preprocessing method of the present invention (based on a wheat spectral dataset) is shown in Figure 4(a); a time curve for another automatic spectral data preprocessing method of the present invention (based on a meat spectral dataset) is shown in Figure 4(b). As shown in Figure 4(a), the RMSECV values for both the wheat and meat spectral datasets decrease rapidly from the beginning and eventually converge. This demonstrates that the genetic algorithm (GA) is very effective in finding the optimal preprocessing process. As shown in Figure 4(b), the choice of LV affects both RMSECV and computation time. For RMSECV, this is readily understood, as the performance of the PLSR model always varies with the LV value. As for the latter, the search path of the GA process varies depending on the LV value, resulting in different computation times. Furthermore, it is important to note that different LV values may result in different preprocessing strategies. Despite this, these strategies may have similar performance. As shown in the figure, the optimal preprocessing process with LV=5 and LV=7 produced similar RMSECV values. Figure 4(a) and 4(b) As shown in Figure 2, the computation for the wheat spectral dataset is more time-consuming than that for the meat spectral dataset. This is mainly because the wheat spectral dataset has a larger data volume.
[0043] We also evaluated the classification task of the automatic spectral data preprocessing method of the present invention on a barley spectral dataset. The classification result diagram of the classification model obtained by modeling the barley spectral data before preprocessing using the automatic spectral data preprocessing method of the present invention (before preprocessing, based on the barley spectral data) is shown as follows: Figure 5 As shown in (a); the classification result diagram of the classification model obtained by modeling the barley spectral data after preprocessing using an automatic spectral data preprocessing method of the present invention (after preprocessing, based on the barley spectral data set) is shown in Figure 5 As shown in (b). Figure 5As shown, the two confusion matrices were calculated based on the test set of the barley spectral dataset. The confusion matrix results for the original spectral data were poor, with an accuracy of 34.2% on the test set. Some categories (such as HB-1965) were completely incorrectly predicted. On the other hand, the classification model derived from the spectral data preprocessed by the automatic spectral data preprocessing method of the present invention easily performed classification, with an accuracy of 83.1% on the training set and 87.1% on the test set, which is very close to the best-performing model reported by Torniainen et al. For details, please refer to the paper "Open-source python module for automated preprocessing of near infrared spectroscopic data" at the following link: https: / / www.sciencedirect.com / science / article / pii / S000326702030204X.
[0044] Figure 6(a) shows a principal component analysis of barley spectral data before preprocessing using the automatic spectral data preprocessing method of the present invention (based on a barley spectral dataset), and Figure 6(b) shows a principal component analysis of barley spectral data after preprocessing using the automatic spectral data preprocessing method of the present invention (based on a barley spectral dataset). For clarity, four of the 24 categories are labeled in Figures 6(a) and 6(b) to illustrate the distribution of these spectral data. As shown in Figure 6(a), the first principal component (PC 1) of the original spectral data explains 98.53% of the variance, while the second principal component (PC 2) explains 1.38%. These two principal components appear to be insufficiently informative, as all categories are intermingled, making classification difficult. The spectral data after preprocessing using the automatic spectral data preprocessing method of the present invention is much easier to classify. As shown in Figure 6(b), the preprocessed spectral data form four clusters, making them easier to separate. Although these four categories are still intermingled with the other spectral data, it should be noted that the first two principal components only explain 88.89% of the variance. Therefore, there is still a lot of potential information that may help with classification.
[0045] In summary, the automatic preprocessing method for spectral data of the present invention can effectively eliminate irrelevant information in the data and improve the signal-to-noise ratio of the data, thereby significantly improving the accuracy of the regression and classification models, and helping to expand the application capabilities of spectral technology.
Claims
1. A method for automatic preprocessing of spectral data, characterized in that First, an algorithm library containing six preprocessing algorithms is constructed. The six preprocessing algorithms are: smoothing algorithm, baseline correction algorithm, scattering correction algorithm, data centering algorithm, data normalization algorithm and variable selection algorithm. Each preprocessing algorithm has multiple corresponding preprocessing functions. Then, a preprocessing process is constructed, which consists of five steps: processing with a smoothing algorithm, processing with a baseline correction algorithm, processing with a scattering correction algorithm, processing with a data centering algorithm and processing with a data normalization algorithm. Then, a genetic algorithm is used to optimize the preprocessing process, determine the optimal preprocessing function of each preprocessing algorithm in the preprocessing process, and obtain the optimal preprocessing process. Then, the spectral data is preliminarily preprocessed using the optimal preprocessing process to obtain the preliminarily preprocessed spectral data, and then the variable selection algorithm is used to preprocess the preliminarily preprocessed spectral data under each preprocessing function of the variable selection algorithm to determine the optimal preprocessing function of the variable selection algorithm; the variable selection algorithm preprocesses the preliminarily preprocessed spectral data under its optimal preprocessing function, and the obtained spectral data is the final preprocessed spectral data.
2. The method for automatic preprocessing of spectral data according to claim 1, characterized in that The genetic algorithm is used to optimize the preprocessing process and determine the optimal preprocessing function of the preprocessing algorithm in each step. The specific steps of the optimized preprocessing process are as follows: Step S1, randomly generating a plurality of different preprocessing processes to constitute a contemporary seed, wherein the preprocessing algorithm of each step of each preprocessing process has a preprocessing function; Step S2: preprocess the spectral data that need to be preprocessed using each preprocessing process in the contemporary seed, and evaluate each preprocessing process in the contemporary seed using the fitness function to screen out the optimal contemporary preprocessing process; Step S3: Perform crossover and mutation operations on the current optimal preprocessing process to generate a new preprocessing process to form a new seed, and update the current seed to the new seed; Step S4: Repeat steps S2 and S3 until the genetic algorithm converges to obtain the optimal preprocessing process.
3. The automatic preprocessing method for spectral data according to claim 2, characterized in that: In step S2, the fitness function models the spectral data processed by each preprocessing process in the contemporary seeds, and the model accuracy is calculated through cross-validation to evaluate the performance of each preprocessing process.