A multi-sensor spectral data processing method based on multi-component samples

Abstract

The present invention relates to near-infrared spectral analysis technology, and discloses a multi-sensor spectral data processing method based on multi-component samples, which improves the prediction and analysis capabilities of portable multi-sensor spectral equipment. The method includes: collecting original multi-sensor spectral data of multi-component samples, classifying and decomposing the data according to sensor models, and obtaining single-band data sets; performing secondary diversity on each single-band data set according to the number of samples to obtain a single-band training set and a verification set; combining the calibration values ​​of the content of each component of the multi-component sample to perform spectral modeling on each single-band training set, and using the established spectral model to predict the corresponding verification set, and calculating the prediction accuracy of the single-band spectral model; combining the prediction accuracy of each single-band spectral model for different components, calculating the characterization coefficient of the single-band spectral model for different component contents; for different components of unknown samples to be tested, combining the characterization coefficients of each single-band spectral model to characterize their corresponding prediction values, and obtaining the final prediction value.

Description

Technical Field

[0001] The invention relates to near-infrared spectral analysis technology, and in particular to a multi-sensor spectral data processing method based on multi-component samples. Background Art

[0002] With the development of micro-electromechanical technology, portable near-infrared spectrometers have been widely used in various fields such as food, beverages, medicine, coal, etc. in recent years. Portable near-infrared spectrometers can not only detect multiple components in parallel, but also have the advantages of simple structure, low cost, easy to carry, and real-time visible results compared to traditional large-scale Fourier transform spectroscopy analysis systems. At present, they have become a popular product in the field of spectroscopy.

[0003] Currently, the portable near-infrared spectrometers on the market are mainly single-sensor devices. Single-sensor devices are limited by sensor technology and cover a very limited wavelength range. The collected spectral data is less stable and prone to deviation, which can easily lead to unstable prediction results and low accuracy.

[0004] In order to increase the coverage of sensor bands and improve the application scenarios of portable near-infrared spectrometers, multi-sensor portable near-infrared spectrometers came into being. Although multi-sensor portable near-infrared spectrometers can solve various problems of single-sensor equipment, if the traditional single-sensor spectral data processing method is used to process multi-sensor spectral data, due to the different characterization capabilities of each sensor spectral data on the content of each component of the sample to be tested, it is easy to cause a large difference between the predicted value and the actual calibration value, thereby reducing the portable near-infrared device's predictive analysis ability for each component of the sample to be tested. Summary of the invention

[0005] The technical problem to be solved by the present invention is to propose a multi-sensor spectral data processing method based on multi-component samples to improve the predictive analysis capability of portable multi-sensor spectral equipment.

[0006] The technical solution adopted by the present invention to solve the above technical problems is:

[0007] A multi-sensor spectral data processing method based on multi-component samples comprises the following steps:

[0008] S1, collect the original multi-sensor spectral data of multi-component samples, classify and decompose the spectral data according to the sensor model, and obtain a single-band data set;

[0009] S2, performing secondary division on each single-band data set according to the number of samples to obtain single-band training sets and validation sets under different division modes;

[0010] S3. Spectral modeling is performed on each single-band training set under different diversity modes in combination with the calibration values ​​of each component content of the multi-component sample, and the established spectral model is used to predict the corresponding validation set, and the prediction accuracy of the single-band spectral model under different diversity modes is calculated;

[0011] S4. Combining the prediction accuracy of each single-band spectral model for different components under different diversity modes, calculate the characterization coefficient of the single-band spectral model for the content of different components;

[0012] S5. For different components of the unknown sample to be tested, the corresponding prediction values ​​are characterized by combining the characterization coefficients of each single-band spectral model under different diversity modes to obtain the final prediction value.

[0013] Furthermore, in step S1, the number of the obtained single-band data sets corresponds to the number of sensor models in the multi-sensor spectrometer, that is, each sensor model corresponds to one single-band data set.

[0014] Furthermore, in step S2, each single-band data set is divided into two sub-groups according to the number of samples to obtain a single-band training set and a validation set in different sub-grouping modes, which specifically includes:

[0015] The single-band data set is divided into a compromise training set and a compromise validation set in a time-series manner, that is, according to the data collection timeline, a certain amount of spectral data collected in the early stage is divided into a compromise training set, and the remaining certain amount of spectral data is divided into a compromise validation set; thus, a single-band training set and a validation set are obtained in a time-series diversity manner;

[0016] In addition, the single-band data set is divided into a random training set and a random validation set by random partitioning, that is, a certain amount of spectral data is randomly selected from the sample data in the single-band data set and divided into a random training set, and the remaining certain amount of spectral data is divided into a random validation set; thereby obtaining a single-band training set and a validation set under the random partitioning method.

[0017] Furthermore, in step S2, after secondary grouping, the data ratio between the compromise training set and the compromise validation set and the data ratio between the random training set and the random validation set are both 4:1.

[0018] Furthermore, in step S3, the established spectral model is used to predict the corresponding validation set, and the prediction accuracy of the single-band spectral model under different diversity modes is calculated, which specifically includes:

[0019] The spectral model established in a single-band training set under a certain diversity mode is used to predict the spectral data in the validation set under the diversity mode. If the deviation between the predicted value and the calibrated value is within the tolerance range, the prediction is judged to be correct. By counting the number of correctly predicted spectral data, the statistical value is obtained, then:

[0020] The prediction accuracy of the single-band spectral model under this diversity mode = statistical value / number of spectral data in the validation set*100%.

[0021] Furthermore, in step S4, the prediction accuracy of each single-band spectral model for different components under different diversity modes is combined to calculate the characterization coefficient of the single-band spectral model for the content of different components, which specifically includes:

[0022]

[0023]

[0024] Among them, T i is the prediction accuracy of the i-th single-band spectrum model under the time series diversity mode; a i is the characterization coefficient of the i-th single-band spectral model for a certain component content under the time series diversity mode; T i ′ is the prediction accuracy of the i-th single-band spectrum model under the random partitioning diversity mode; a i ′ is the characterization coefficient of the i-th single-band spectral model for a certain component content under the random division diversity mode; n is the number of single-band spectral models (also the number of sensors).

[0025] Furthermore, in step S5, for different components of the unknown sample to be tested, characterization processing is performed on the corresponding predicted values ​​in combination with the characterization coefficients of each single-band spectral model under different diversity modes to obtain the final predicted value, which specifically includes:

[0026]

[0027] Among them, A is the final predicted value of a component of the unknown sample to be tested, A i is the predicted value of the component in the unknown sample to be tested by the i-th single-band spectrum model under the time series diversity mode; A i ′ is the predicted value of the component in the unknown sample to be tested by the i-th single-band spectral model under the random division diversity mode.

[0028] The beneficial effects of the present invention are:

[0029] This method targets multi-sensor data of multi-component samples, uses prediction accuracy as an indicator, and further calculates the characterization coefficient corresponding to the characterization ability of different sensor spectral data for samples of different components, and then characterizes the prediction values ​​of different sensors. Since the characterization ability of different sensor spectral data for samples of different components is taken into account, the predictive analysis ability of portable multi-sensor spectral equipment is greatly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] Figure 1 is a flow chart of a multi-sensor spectral data processing method based on a multi-component sample in an embodiment of the present invention;

[0031] Figure 2 Schematic diagram of the arrangement of the sensors of the portable multi-sensor near-infrared spectrometer in an embodiment of the present invention. DETAILED DESCRIPTION

[0032] The present invention aims to propose a multi-sensor spectral data processing method based on multi-component samples to improve the prediction and analysis capabilities of portable multi-sensor spectral equipment. The method first collects the original multi-sensor spectral data of the multi-component sample, classifies and decomposes the spectral data according to the sensor model, and obtains a single-band data set; then, each single-band data set is secondary-divided according to the number of samples to obtain a single-band training set and a validation set; then, each single-band training set is spectrally modeled in combination with the calibration value of each component content of the multi-component sample, and the established spectral model is used to predict the corresponding validation set, and the prediction accuracy of the single-band spectral model is calculated; then, the prediction accuracy of each single-band spectral model for different components is combined, and the characterization coefficient of the single-band spectral model for different component contents is calculated; finally, for different components of the unknown sample to be tested, the corresponding prediction value is characterized in combination with the characterization coefficient of each single-band spectral model to obtain the final prediction value.

[0033] The scheme of the present invention adopts prediction accuracy as an indicator, corresponds to the characterization ability of different sensor spectral data for samples of different components, further calculates the characterization coefficient and then characterizes the prediction values ​​of different sensors. Since the characterization ability of different sensor spectral data for samples of different components is taken into consideration, the prediction and analysis ability of the portable multi-sensor spectral equipment is greatly improved.

[0034] Example:

[0035] like Figure 1 As shown, the multi-sensor spectral data processing method based on multi-component samples in this embodiment includes the following steps:

[0036] S101, collecting original multi-sensor spectral data of multi-component samples, classifying and decomposing the spectral data according to sensor models, and obtaining a single-band data set;

[0037] In this step, the original multi-sensor spectral data of the multi-component sample is essentially a collection of single-sensor spectral data, which are classified and decomposed into multiple single-band data according to the sensor model, so as to facilitate the subsequent calculation of the characterization ability of each single-sensor spectral data for the sample to be tested.

[0038] In a specific example, the portable multi-sensor near-infrared spectrometer used includes three sensors of different models, wherein the three sensor band ranges are 1255nm-1550nm, 1555nm-1850nm, 1855nm-2150nm, respectively, the resolution is 5nm, and the wavelength point distribution mode is wavelength averaging. The specific structure is as follows: Figure 2 As shown. The wavelength range of sensor 1 is 1255nm~1550nm, and the number of wavelength points included is M1=1+(1550-1255) / 5=60; the wavelength range of sensor 2 is 1555nm~1850nm, and the number of wavelength points included is M2=1+(1850-1555) / 5=60; the wavelength range of sensor 3 is 1855nm~2150nm, and the number of wavelength points included is M3=1+(2150-1855) / 5=60. From the above, it can be seen that the original spectral data of multiple sensors is spliced ​​by 3 segments of spectral data, with a specific wavelength range of 1255nm~2150nm, and a total of M4=M1+M2+M3=180 wavelength points. The multi-sensor spectral data are classified according to the sensor model to obtain three single-band data. For the convenience of description, they are set as single-band data P1, P2, and P3, corresponding to the spectral data of sensors 1, 2, and 3, respectively.

[0039] S102, performing secondary grouping on each single-band data set according to the number of samples to obtain a single-band training set and a verification set;

[0040] In this step, in the portable near-infrared spectroscopy analysis technology, in order to achieve the most representative prediction effect, when the number of samples to be tested is fixed, too few training set samples will cause the spectral model to overfit, and too few validation set samples will make the prediction effect unreliable. When the ratio of the number of training set samples to the number of validation set samples is maintained at 4:1, the prediction effect is most representative.

[0041] In the practical application of portable near-infrared spectroscopy analysis technology, when the number of samples reaches 400, that is, when the spectral model established corresponds to 400 spectral data, it has good stability and prediction ability. According to this characteristic, when the ratio of the number of training set samples to the number of validation set samples is maintained at 4:1, its prediction effect is most representative.

[0042] Therefore, in a specific example, the number of original multi-sensor spectral data collected for multi-component samples is 500. Since collecting spectral data of the samples to be tested is often a continuous process, 500 samples to be tested require a certain collection period, and the time span is large. The portable near-infrared spectroscopy equipment is prone to poor consistency of spectral data collected at different time spans due to its own equipment aging or other factors. Therefore, when the single-band data is divided, two different division methods are used to reduce the prediction error:

[0043] Diversity mode 1 is time series diversity mode: it is divided into compromise training set and compromise validation set in time sequence, that is, according to the acquisition timeline, the 400 spectral data collected in the early stage are divided into the compromise training set, and the remaining 100 spectral data are divided into the compromise validation set;

[0044] The second diversity method is the random division diversity method: the random division method is used to divide the data into a random training set and a random validation set. That is, 400 spectral data are randomly selected from the 500 spectral data collected as the random training set, and the remaining 100 spectral data are divided into a random validation set.

[0045] S103, performing spectral modeling on each single-band training set in combination with the calibrated values ​​of the content of each component of the multi-component sample, and using the established spectral model to predict the corresponding validation set, and calculating the prediction accuracy of the single-band spectral model;

[0046] In this step, although the spectral model itself contains some model discrimination indicators, such as root mean square error, determination coefficient, etc., such indicators can only reflect the effectiveness of the spectral model. In the actual blind test process, the prediction accuracy of unknown samples is often an intuitive indicator for judging the near-infrared spectral analysis ability. Therefore, when calculating the characterization coefficient, using prediction accuracy as a calculation indicator can better represent the prediction ability of the spectral model.

[0047] In a specific example, the multi-component sample is selected as a pepper sample, and its component content is selected from two components: volatile oil and hemp. Taking the component volatile oil as an example, the traditional classical analysis method is used to obtain the volatile oil content calibration values ​​of 500 pepper samples, and these content values ​​are matched one by one with the spectral data of the pepper samples. For the first diversity method, each single-band compromise training set is used for spectral modeling, and the corresponding single-band compromise validation set is predicted, and the predicted value is compared with the calibration value to obtain the prediction accuracy T1, T2, T3 of different single-band data P1, P2, P3. Similarly, for the hemp content, the same processing method is used to obtain the prediction accuracy K1, K2, K3 of the single-band data P1, P2, P3;

[0048] For the second diversity method, each single-band random training set is used for spectral modeling, and the corresponding single-band random validation set is predicted. The predicted value is compared with the calibration value to obtain the prediction accuracy T4, T5, and T6 of different single-band data P1, P2, and P3. Similarly, for the cannabinoid content, the same processing method is used to obtain the prediction accuracy K4, K5, and K6 of the single-band data P1, P2, and P3.

[0049] The prediction accuracy is calculated as follows: predict a total of 100 validation set spectral data. If the deviation between the predicted value and the calibrated value is within the tolerance range, the prediction is considered correct. The total number of samples with correct predictions is S, and the prediction accuracy T is:

[0050] T=100%*S / 100

[0051] S104, combining the prediction accuracy of each single-band spectral model for different components, calculating the characterization coefficient of the single-band spectral model for the content of different components;

[0052] In this step, for a certain component of the sample to be tested, the higher the prediction accuracy of the single-band training set, the stronger the sample characterization ability of the spectral data of this band is. The characterization coefficient of each single-band spectral data of this component can be further calculated through the ratio relationship between the prediction accuracies of different single-band spectral data. For the remaining different components, the characterization coefficient can be calculated in the same way to obtain the characterization coefficient of each single-band data for different component contents.

[0053] In a specific example, taking the volatile oil component content of pepper samples as an example, there are differences in the prediction accuracy of different single-band data. The higher the prediction accuracy, the stronger the characterization ability of the single-band data for the component. At the same time, two diversity methods are used to minimize the prediction error to the greatest extent. For diversity method 1, the characterization coefficients a1, a2, and a3 of the single-band data P1, P2, and P3 are calculated as follows:

[0054]

[0055] For diversity mode 2, the characterization coefficients a4, a5, and a6 of the single-band data P1, P2, and P3 are calculated as follows:

[0056]

[0057] Similarly, for the content of the anesthetic components in the pepper sample, the same calculation method is used to calculate the characterization coefficients β1, β2, β3, β4, β5, and β6 of the single-band data P1, P2, and P3 under the two diversity modes:

[0058]

[0059] S105. For different components of the unknown sample to be tested, characterization processing is performed on the corresponding predicted values ​​in combination with the characterization coefficients of each single-band spectral model to obtain a final predicted value.

[0060] In this step, due to different sensor models, the band ranges covered by their spectral data are also different, and thus their ability to characterize samples will also be different. If the same characterization coefficient is used for characterization processing of different component contents of the sample to be tested, the differential characterization ability of the spectral data for different components is likely to cause unstable prediction results, with excellent prediction results for some components and poor prediction results for other components. The characterization coefficient is calculated based on the prediction accuracy of single-band data of different components, and further characterization processing can effectively solve this problem and improve the spectral analysis capability.

[0061] In a specific example, a blind test is performed on a Zanthoxylum bungeanum sample to be tested with an unknown component calibration value, multi-sensor spectral data of the Zanthoxylum bungeanum sample to be tested is collected, and single-band data is divided according to the sensor model, and the single-band spectral model established in step 103 is further used to predict the single-band data of the Zanthoxylum bungeanum sample to be tested. For the volatile oil content of the Zanthoxylum bungeanum sample to be tested, it is assumed that the prediction values ​​corresponding to the three single-band spectral models in the diversity mode one are set to A1, A2, and A3, and the prediction values ​​corresponding to the three single-band spectral models in the diversity mode two are set to A4, A5, and A6, and the prediction values ​​of each single-band data are characterized in combination with the characterization coefficient, and the final prediction value A of the volatile oil content of the Zanthoxylum bungeanum sample to be tested can be obtained as follows:

[0062]

[0063] Similarly, for the cannabinoid content of the pepper sample to be tested, the prediction values ​​corresponding to the three single-band spectral models in the first diversity mode are set as B1, B2, and B3, and the prediction values ​​corresponding to the three single-band spectral models in the second diversity mode are set as B4, B5, and B6. The predicted values ​​of each single-band data are characterized by combining the characterization coefficient, and the final predicted value B of the cannabinoid content of the pepper sample to be tested can be obtained as follows:

[0064]

[0065] Finally, it should be noted that the above embodiments are only preferred implementations and are not intended to limit the present invention. It should be pointed out that for those skilled in the art, several modifications, equivalent replacements, improvements, etc. can be made without departing from the scope of the present invention and the scope of protection of the claims, and all of these should be included in the protection scope of the present invention.

Claims

1. A multi-sensor spectral data processing method based on multi-component samples, It is characterized in that The following steps are involved: S1, collect the original multi-sensor spectral data of multi-component samples, classify and decompose the spectral data according to the sensor model, and obtain a single-band data set; S2, performing secondary division on each single-band data set according to the number of samples to obtain single-band training sets and validation sets under different division modes; S3. Spectral modeling is performed on each single-band training set under different diversity modes in combination with the calibration values ​​of each component content of the multi-component sample, and the established spectral model is used to predict the corresponding validation set, and the prediction accuracy of the single-band spectral model under different diversity modes is calculated; S4. Combining the prediction accuracy of each single-band spectral model for different components under different diversity modes, calculate the characterization coefficient of the single-band spectral model for the content of different components; S5. For different components of the unknown sample to be tested, characterization processing is performed on the corresponding predicted values ​​by combining the characterization coefficients of each single-band spectral model under different diversity modes to obtain the final predicted value; In step S4, the prediction accuracy of each single-band spectral model for different components under different diversity modes is combined to calculate the characterization coefficient of the single-band spectral model for the content of different components, which specifically includes: Among them, T i is the prediction accuracy of the i-th single-band spectrum model under the time series diversity mode; a i is the characterization coefficient of the i-th single-band spectral model for a certain component content under the time series diversity mode; T i ' is the prediction accuracy of the i-th single-band spectrum model under the random partitioning method; a i ' is the characterization coefficient of the i-th single-band spectral model for a certain component content under the random division diversity mode; n is the number of single-band spectral models; In step S5, for different components of the unknown sample to be tested, characterization processing is performed on the corresponding predicted values ​​in combination with the characterization coefficients of each single-band spectral model under different diversity modes to obtain the final predicted value, which specifically includes: Among them, A is the final predicted value of a component of the unknown sample to be tested, A i is the predicted value of the component in the unknown sample to be tested by the i-th single-band spectrum model under the time series diversity mode; A i ' is the predicted value of the component in the unknown sample to be tested by the i-th single-band spectral model under the random partitioning diversity mode.

2. A multi-sensor spectral data processing method based on multi-component samples as claimed in claim 1, It is characterized in that In step S1 , the number of obtained single-band data sets corresponds to the number of sensor models in the multi-sensor spectrometer, that is, each sensor model corresponds to one single-band data set.

3. A multi-sensor spectral data processing method based on multi-component samples as claimed in claim 1, It is characterized in that In step S2, each single-band data set is divided into two sub-groups according to the number of samples to obtain a single-band training set and a validation set in different sub-grouping modes, which specifically includes: The single-band data set is divided into a compromise training set and a compromise validation set in a time-series manner, that is, according to the data collection timeline, a certain amount of spectral data collected in the early stage is divided into a compromise training set, and the remaining certain amount of spectral data is divided into a compromise validation set; thus, a single-band training set and a validation set are obtained in a time-series diversity manner; In addition, the single-band data set is divided into a random training set and a random validation set by random partitioning, that is, a certain amount of spectral data is randomly selected from the sample data in the single-band data set and divided into a random training set, and the remaining certain amount of spectral data is divided into a random validation set; thereby obtaining a single-band training set and a validation set under the random partitioning method.

4. A multi-sensor spectral data processing method based on multi-component samples as claimed in claim 3, It is characterized in that In step S2, after the second grouping, the data ratio between the compromise training set and the compromise validation set and the data ratio between the random training set and the random validation set are both 4:

1.

5. A multi-sensor spectral data processing method based on multi-component samples as claimed in claim 3, It is characterized in that In step S3, the established spectral model is used to predict the corresponding validation set, and the prediction accuracy of the single-band spectral model under different diversity modes is calculated, which specifically includes: The spectral model established in a single-band training set under a certain diversity mode is used to predict the spectral data in the validation set under the diversity mode. If the deviation between the predicted value and the calibrated value is within the tolerance range, the prediction is judged to be correct. By counting the number of correctly predicted spectral data, the statistical value is obtained, then: The prediction accuracy of the single-band spectral model under this diversity mode = statistical value / number of spectral data in the validation set*100%.

Images