Near infrared spectroscopy sample generation method based on model transfer under small sample conditions

Abstract

The present invention discloses a near-infrared spectrum sample generation method based on model transfer under small sample conditions. Before model transfer, a large amount of labeled Master data needs to be screened for samples, and samples whose spectrum data range seriously exceeds the Master spectrum data range when establishing model transfer are found and eliminated. After the sample screening work is completed, the screened labeled Master data can be model transferred to realize near-infrared spectrum sample generation on Slave. The present invention generates more near-infrared spectrum samples that can be used for modeling through model transfer, so as to solve the problem of feed nutrient content detection under small sample conditions.

Description

Technical Field

[0001] The invention relates to the technical field of near infrared spectrum analysis, in particular to a near infrared spectrum sample generation method based on model transfer under small sample conditions. Background Art

[0002] Near-infrared light is a high-energy vibration electromagnetic wave with a wavelength range of 780 to 2526 nm. Since the vibration of atoms or molecules in matter does not resonate, near-infrared radiation can cause some high-energy transitions of electrons, causing atoms or molecules to have symmetrical, curved, asymmetrical vibration states. If the vibration transition of a molecule from the ground state to the excited state is exactly equal to the energy of a photon of a certain wavelength in the near-infrared wavelength range, near-infrared spectral absorption will occur [5].

[0003] Under the action of near-infrared light, light and groups with the same frequency will resonate, and the energy of light will be transferred to the molecule through the change of molecular dipole moment; however, the frequency of near-infrared light is different from the vibration frequency of the sample, so it cannot absorb infrared light of this frequency. Since different organic compounds contain groups with different energy levels, different groups and the same group have significant differences in the absorption wavelength of near-infrared light in different physical and chemical environments. By analyzing the near-infrared spectrum of the sample, the characteristic information of the hydrogen-containing groups of the organic molecules it contains can be obtained, so the near-infrared spectrum can be used as an effective carrier for obtaining information.

[0004] Model transfer transfers the spectrum from one instrument to another near-infrared spectrometer, which will greatly save manpower and material resources and speed up real-time analysis and detection while ensuring analysis accuracy. The model transfer method establishes a mathematical function relationship between the master and slave instruments, thereby converting the master instrument spectrum or prediction results into the slave instrument spectrum or prediction results through a clear mathematical relationship.

[0005] Spectral model transfer can generally be divided into two types of problems: one is the model transfer of homologous samples under different circumstances, and the other is the automatic transfer and matching application of spectral models between different samples.

[0006] The main methods based on spectral signal correction in model transfer include direct standardization (DS), piecewise direct standardization (PDS), canonical correlation analysis (CCA), etc. DS algorithm and PDS algorithm are the most commonly used model transfer algorithms. DS algorithm establishes the transfer matrix through the full wavelength spectrum [1], while PDS algorithm is an improvement on the DS algorithm, selecting a part of the window data to calculate the transfer matrix [2]. Peng Yunfa et al. [3] transferred the near-infrared spectrum of jujube trees collected by a portable near-infrared spectrometer to the model established by the laboratory Antaris II near-infrared spectrometer using the DS algorithm and the PDS algorithm. The results showed that the prediction results of the model after transfer were comparable to those of the laboratory Antaris II near-infrared spectrometer model. Zheng Kaiyi et al. [4] took the three-component system of toluene, chlorobenzene and n-heptane as the research object, and applied the CCA algorithm and the PDS algorithm to transfer the differences between the near-infrared spectra caused by different instruments and different measurement times. The results showed that the CCA algorithm and the PDS algorithm can effectively improve the universality of the model.

[0007] 1. Direct standardization / segmented direct standardization:

[0008] Direct standardization, that is, the DS algorithm directly uses the transformation matrix F DS The spectral matrix X measured by the sample on the master instrument (hereinafter referred to as Master) and the slave instrument (hereinafter referred to as Slave) m , X s The relationship is as follows:

[0009] X s =X m ·F DS

[0010] F DS =X m -1 ·X s

[0011] F DS is the conversion matrix, for the spectrum of the sample to be tested measured on the Master Model transfer can be completed with the transformation matrix:

[0012]

[0013] The DS algorithm generally requires that the number of samples should be greater than the number of wavelengths, which is difficult to meet in actual situations. It is based on this consideration that the DS algorithm is improved to obtain the piecewise direct standardization (PDS) algorithm.

[0014] Unlike the DS algorithm, the PDS algorithm believes that: the changes in spectral data are regional and confined to certain related wavelengths. Therefore, a wavelength point on the Slave is only closely correlated with the wavelength near the corresponding wavelength point on the Master, and has almost no correlation with other farther wavelength points. Based on the above ideas, a window sliding from left to right is added to the Master spectral matrix in the PDS algorithm. Simply put, the DS algorithm is performed on this sliding window and the corresponding Slave spectral column. The PDS algorithm needs to use the j-th wavelength point in the Master spectral matrix as the center to establish a PDS window. The length of the left side of the window is denoted as a, and the length of the right side of the window is denoted as b. The window matrix with a total length of a+b+1 and a width of the sample size n is (a new matrix consisting of the jth column to the j+bth column in the total spectrum matrix) and the jth wavelength data column X of the slave s,j Calculate the conversion coefficient vector for the i-th wavelength point Then, the center column j of the window is moved point by point (j = 1, 2, ..., J, J is the number of spectral wavelength points) to obtain the conversion coefficient vectors of all wavelengths, and then the conversion coefficient vectors are assembled into a transfer matrix F PDS .

[0015] The specific steps of the PDS algorithm are:

[0016] Corresponding to the jth column data X on the Slave spectrum matrix s,j , in the Master spectrum matrix X m Select the spectrum segment with window length a+b+1 (0≤a,b≤J-1; a+b≤J-1) to form a matrix

[0017] Through the relation X s,j and The correlation is performed, where the conversion coefficient vector It can be obtained by PLSR algorithm. This step can refer to Figure 1 Schematic diagram of PDS steps;

[0018] Move the window center j from 1 to J and find all J is the number of spectral wavelength points;

[0019] All the conversion coefficient vectors Transpose in the transformation matrix F PDS On the main diagonal of PDS All other elements are 0, and we get the transformation matrix F PDS ;

[0020] Master spectrum matrix to be transferred By the transformation matrix F PDS Get the corresponding spectral matrix

[0021] Note: For the Master spectrum matrix to be transferred The spectra with wavelengths in the range of 1~a and (Jb)~J at both ends cannot be converted. The spectrum matrix after model transfer is This part of the spectrum is usually discarded, or the corresponding spectrum can be obtained by extrapolation.

[0022] 2. Canonical Correlation Analysis:

[0023] In order to grasp the correlation between the two groups of variables as a whole, canonical correlation analysis (Canonical Correlation Analysis) extracts representative comprehensive variables U1 and V1 (which are linear combinations of the variables in the two variable groups respectively) from the two groups of variables, and uses the correlation between these two comprehensive variables to reflect the overall correlation between the two groups of variables, and solves the linear transformation between the two.

[0024] Assume that the original spectral data on the Master and Slave are represented by X m , X s Indicates that canonical correlation analysis first converts the multidimensional spectral data X m , X s Linear transformation into one-dimensional standard variable L m , L s , and then use the correlation coefficient to check L m , L s The correlation between the two is calculated and the conversion matrix between them is solved. The Master spectral data completes the model transfer through a series of linear transformations.

[0025] Assume that the projection vector during CCA dimensionality reduction is w m 、w s Indicates that the standard variable L m , L s It can be expressed as:

[0026] L m =X m × m

[0027] L s =X s × s

[0028] L m , L s The relationship can be expressed by the following formula:

[0029] Lm =L s ×F1

[0030] The conversion matrix F1 can be calculated by partial least squares method. At the same time, the conversion matrix F2 can be obtained by the following formula:

[0031] X m =L m ×F2

[0032] Master spectrum to be transferred to the model Then you can use w m , F1, F2 to transfer the model:

[0033]

[0034] Ding Ke et al. [6] established a good near-infrared spectroscopy quantitative analysis model for protein feed and realized model sharing between different instruments, which greatly improved the utilization efficiency of the model and met the needs of the rapid development of the feed industry. In order to solve the problem of the applicability of the near-infrared analysis model for crude protein content in protein feed raw materials, the model transfer between three different types of near-infrared spectrometers was studied for the first time using spectral difference transfer, direct correction and segmented direct correction methods. The experimental samples were four protein feed raw materials: corn gluten meal, rapeseed meal, distiller's grains and fish meal. The experimental instruments included MATRIX-I Fourier transform near-infrared spectrometer (master instrument), Spectrum400 Fourier transform near-infrared spectrometer (slave instrument 1) and SupNIR-2750 grating scanning near-infrared spectrometer (slave instrument 2). The study showed that the difference in spectral data obtained from the master instrument and slave instrument 2 for the same sample system was relatively small, and the difference was relatively large from the spectral data obtained from slave instrument 1. Except for the segmented direct correction method, which had no effect on the prediction results of corn gluten meal from instrument 2, the prediction root mean square error and systematic deviation of other models were significantly lower than before the transfer. The relative analysis error (RPD) of the model prediction of corn protein meal, rapeseed meal and distiller's grains after the three methods were transferred was greater than 3.0, and the prediction effect was good. The predicted RPD of fish meal samples after the model transfer was greater than 2.5, and the prediction effect was good. The three methods effectively corrected the spectral differences between different instruments for protein feed raw materials. The research results are of great significance for the widespread application of near-infrared rapid analysis models for protein feed quality.

[0035] There are many excellent models in the detection of common feed nutrient content. At present, the open source data sets commonly used in the detection field of near-infrared spectroscopy include corn data sets (80 records), wheat data sets (248 records), soil data sets (3793 records), etc., among which the minimum sample size of gasoline octane prediction data sets is 60. Existing research on near-infrared spectroscopy detection technology is generally conducted on the above open source data sets, or with sufficient resources to collect sufficient samples for experiments, neither of which involves the case of small samples. However, in the actual feed industry, due to the high economic and time costs of obtaining the true moisture content through chemical experiments, only a small number of samples can be used to establish prediction models. The near-infrared spectroscopy detection technology for small samples needs further research.

[0036] On the other hand, in the near-infrared spectroscopy model transfer technology, the existing research is to transfer the near-infrared spectrum on the slave instrument to the master instrument, and then use the existing model on the master instrument for prediction. It has not yet taken into account that in practice, it may be possible to only obtain experimental data on the master instrument, but not know the model used by the master instrument. In this case, it is necessary to use a large amount of data on the master instrument, convert it into slave instrument data, and then build a model on the slave instrument.

[0037] [1] Chu Xiaoli, Yuan Hongfu, Lu Wanzhen. Model transfer in spectral multivariate calibration[J]. Spectroscopy and Spectral Analysis, 2001, 21(6):5.

[0038] [2] Chu Xiaoli, Xu Yupeng, Lu Wanzhen. Research and application progress of chemometric methods for near infrared spectroscopy analysis [J]. Analytical Chemistry, 2008, 36(5):8.

[0039] [3] Peng Yunfa, Peng Haigen, Zhan Ying, et al. Study on non-destructive detection of moisture content in red dates from southern Xinjiang using near infrared spectroscopy [J]. Food Science and Technology, 2013, 38(11): 4.

[0040] [4] Zheng Kaiyi, Fan Wei, Wu Ting, et al. Application of canonical correlation analysis to calibrate near infrared spectra measured by different instruments and at different times [J]. Computers and Applied Chemistry, 2013(3):5.

[0041] [5] Yan Yanlu, Chen Bin, Zhu Dazhou. Principles, techniques and applications of near infrared spectroscopy analysis[M]. China Light Industry Press, 2013.

[0042] [6] Ding Ke, Zhang Yuejing, Shen Guanghui, et al. Study on near infrared spectroscopy model transfer of crude protein content in protein feed raw materials[J]. Spectroscopy and Spectral Analysis, 2016, 36(5):7. DOI:10.3964 / j.issn.1000-0593(2016)05-1334-06. Summary of the invention

[0043] In order to solve the problems existing in the prior art, the purpose of the present invention is to provide a near-infrared spectral sample generation method based on model transfer under small samples. The present invention generates more near-infrared spectral samples that can be used for modeling through model transfer to solve the problem of feed nutrient content detection under small samples.

[0044] To achieve the above object, the technical solution adopted by the present invention is: a near infrared spectrum sample generation method based on model transfer under small sample conditions, comprising the following steps:

[0045] Step 1: Scan a small amount of feed sample with a sample size of n1 on the secondary spectrometer, i.e., the spectrometer for which the model is to be established, to obtain a spectral matrix As the test set of the prediction model; let the spectral band scanned by the spectrometer be q, then is a spectral matrix of n1×q; at the same time, chemical experiments are carried out on these feed samples to detect their true nutrient content, that is, the label value;

[0046] Step 2: Scan the feed sample with a sample size of n2 on the spectrometer to obtain a spectrum matrix X s , and then perform spectrum scanning on the main spectrometer to obtain the spectrum matrix X m , X m , X s That is, a set of n2×q spectral matrices for model transfer is used as unlabeled data;

[0047] Step 3: Use X m , X s Perform model transfer and find the transformation relationship g(·) from the master spectrometer to the slave spectrometer;

[0048] Step 4: For a large amount of spectral data with label y on the main spectrometer Perform spectral sample screening to find and remove spectral ranges far exceeding X m The sample after filtering is express; also Remove the corresponding label based on the nutrient content label value;

[0049] Step 5: The spectral matrix of the main spectrometer after sample screening by g(·) obtained in step 3 Perform model transfer, and use the spectral matrix after model transfer express, That is, the near-infrared spectrum sample generated from the spectrometer;

[0050] Step 6: is the training set, Spectral preprocessing was performed for the test set, and a suitable nutrient content prediction model was established.

[0051] As a further improvement of the present invention, in step 3, the transformation relationship g(·) represents the PDS algorithm or the CCA algorithm, and the leave-one-out method is used, each pair of spectral data is used as a test set exactly once, and the obtained t transfer matrices are averaged to obtain the final transfer matrix.

[0052] As a further improvement of the present invention, before using the PDS algorithm for model transfer, it is necessary to adjust its parameters, set the range of the PDS left and right window lengths to 0-(0.05*q), and make any combination under the condition that the sum of the left and right lengths does not exceed (0.05*q); and use RMSE as an evaluation indicator, and select the left and right window lengths with the smallest RMSE as the standard.

[0053] As a further improvement of the present invention, in the CCA algorithm, the number of components n_components retained by the CCA algorithm is gridded and adjusted before use, and the RMSE of the training set is used as an evaluation indicator, and the n_components with the smallest RMSE is taken.

[0054] As a further improvement of the present invention, in step 4, the main spectrometer carries the labeled spectral data The screening criteria are as follows:

[0055] Calculate the unlabeled spectral data X of the master spectrometer involved in the model transfer m The maximum and minimum spectral values ​​at each wavelength are traversed Check the spectral values ​​of all spectral wavelengths in a single sample to see if they are greater than X m The maximum spectral value of this wavelength is less than X m The minimum spectral value of the wavelength is calculated. If one of the two conditions is met, the number of wavelengths out of range is counted. When the total number of wavelengths out of range of a single sample is less than a certain threshold of the total number of wavelengths, it can be retained, otherwise the sample is discarded.

[0056] The sample screening condition is expressed as follows:

[0057]

[0058] in, is the jth wavelength of a single sample of labeled spectral data of the main spectrometer, is a vector consisting of the jth wavelength of all samples in the unlabeled spectral data of the main spectrometer, and q is the number of spectral bands.

[0059] The beneficial effects of the present invention are:

[0060] The present invention performs model transfer on different instruments for homologous samples, generates more near-infrared spectrum samples that can be used for modeling through model transfer, and solves the problem of feed nutrient content detection under small samples. BRIEF DESCRIPTION OF THE DRAWINGS

[0061] Figure 1 It is a schematic diagram of the steps of the PDS algorithm;

[0062] Figure 2 A spectrum diagram drawn by 33 pairs of spectrum data related to model transfer in Example 1 of the present invention;

[0063] Figure 3 Spectra of two randomly selected samples drawn respectively in Example 1 of the present invention;

[0064] Figure 4 This is a spectrum comparison diagram of 33 unlabeled Master spectral data Master_X_33 before and after the PDS algorithm model transfer in Example 1 of the present invention;

[0065] Figure 5 The comparison diagram of two samples randomly selected in Example 1 of the present invention before and after the single sample PDS algorithm spectrum transfer is performed;

[0066] Figure 6 This is a spectrum comparison diagram of 33 unlabeled Master spectral data Master_X_33 before and after the CCA algorithm model transfer in Example 1 of the present invention;

[0067] Figure 7 The comparison of two samples randomly selected in Example 1 of the present invention before and after the single sample CCA algorithm spectrum transfer is shown in the figure;

[0068] Figure 8 Spectral comparison diagram of Master_X before and after sample screening in Example 1 of the present invention;

[0069] Fig. 9 The transfer matrix obtained by the PDS algorithm in Example 1 of the present invention is used to perform model transfer on Master_X after sample screening to obtain Master_X_trans, and 100 samples are randomly selected from them to draw comparison diagrams before and after the model transfer;

[0070] Fig.10 This is a schematic diagram of the process of Embodiment 2 of the present invention;

[0071] Fig.11 This is a flow chart of the prediction process in Example 2 of the present invention. DETAILED DESCRIPTION

[0072] The embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0073] Example 1

[0074] In the actual feed industry, before establishing a content prediction model, it is necessary to collect experimental data for modeling. Experimental data collection is divided into two steps: one is to perform spectral scanning on the feed sample to obtain spectral data, which is the independent variable in the model; the other is to perform chemical experiments on the same feed sample to detect its precise content and obtain label data, which is the dependent variable in the model.

[0075] In the collection of experimental data, since chemical experiments for testing nutrient content in feed are expensive and time-consuming, feed industry manufacturers are unable to conduct a large number of chemical experiments when collecting experimental data, resulting in a small sample size of experimental data that can be used to establish predictive models.

[0076] This embodiment generates more near-infrared spectral samples that can be used for modeling through model transfer to solve the problem of feed nutrient content detection under small samples.

[0077] Model transfer is to construct a related mathematical function relationship through the spectral data scanned by the same group of feed samples in the Master and Slave, so as to transfer the Master spectral data to the Slave through a certain mathematical relationship to complete the model transfer. Since segmented direct standardization (PDS) and canonical correlation analysis (CCA) are currently recognized as better model transfer methods, this embodiment uses the PDS algorithm and the CCA algorithm to study model transfer. And STER, RMSE and MAE are used as the evaluation criteria for model transfer, and the algorithm with good model transfer effect is selected from the PDS algorithm and the CCA algorithm as the standard, and a large amount of labeled Master experimental data is model transferred by this algorithm.

[0078] Prior to this, considering that the amount of data to be transferred to the model is large, while the amount of data to be transferred to establish the model is small, if the range of the Master spectral data to be transferred seriously exceeds the range of the Master spectral data used for establishing the model transfer, the established model transfer may not be applicable to the samples that seriously exceed the spectral data range, or the model transfer effect may be poor. Therefore, before the model transfer, it is necessary to screen a large number of labeled Master data samples to find and eliminate samples whose spectral data range seriously exceeds the Master spectral data range when establishing the model transfer. After completing the sample screening work, the model transfer can be performed on the screened labeled Master data to realize the generation of near-infrared spectral samples on the Slave.

[0079] This embodiment includes the following steps:

[0080] Step 1: Scan a small amount of feed samples (sample size is n1) on the spectrometer Slave (i.e. the spectrometer that needs to build the model) to obtain the spectral matrix As the test set of the prediction model. Assume that the spectral band scanned by the spectrometer is q, then is a spectral matrix of n1×q. At the same time, chemical experiments are carried out on these feed samples to detect their true nutrient content (i.e., label value);

[0081] Step 2: Scan a batch of feed samples (sample size is n2) on the slave to obtain the spectrum matrix X s , and then perform spectrum scanning on the Master to obtain the spectrum matrix X m .X m , X s That is, a set of n2×q spectral matrices for model transfer (note that no chemical experimental detection is required here, so this set of data is unlabeled data);

[0082] Step 3: Through X m , X s Perform model transfer and find the transformation relationship g(·) from Master to Slave. In this embodiment, g(·) represents the PDS algorithm or the CCA algorithm. m , X s The sample size is small, and dividing the training set and the test set may have a great impact on the model transfer effect. Therefore, it is necessary to use the leave-one-out method. Each pair of spectral data is used as a test set exactly once, and the t transfer matrices obtained are averaged to obtain the final transfer matrix. This can avoid overfitting of the model transfer and achieve better results.

[0083] In addition to the division of the training set, the left and right lengths of the PDS sliding window as a hyperparameter are also one of the important factors affecting the model effect. It not only affects the final transfer matrix, but also determines the wavelength of the spectral data after the transfer. Therefore, it is necessary to adjust the parameters before using PDS for model transfer. Considering the problem that the number of spectral data variables will be reduced after PDS, the range of the left and right window lengths of PDS is set to 0~(0.05*q), and any combination can be made under the condition that the sum of the left and right lengths does not exceed (0.05*q). Taking RMSE as the evaluation index, the left and right window lengths with the minimum RMSE are selected as the standard. For the CCA algorithm, n_components is an important parameter. This parameter indicates the number of components retained by the CCA algorithm. It needs to be gridded and adjusted before use. Taking the RMSE of the training set as the evaluation index, take the n_components with the minimum RMSE.

[0084] Step 4: In order to ensure the model transfer effect, for a large amount of spectral data with label y on the Master Perform spectral sample screening to find and remove spectral ranges far exceeding X m The filtered spectral data are used express; also The nutrient content label value also needs to be removed from the corresponding label.

[0085] Master with labeled spectral data Filtering conditions: Calculate the Master unlabeled spectral data X involved in model transfer m The maximum and minimum spectral values ​​at each wavelength are traversed Check the spectral values ​​of all spectral wavelengths in a single sample to see if they are greater than X m The maximum spectral value of this wavelength is less than X m The minimum spectral value of the wavelength is calculated. If one of the two conditions is met, the number of wavelengths out of range is counted. If the total number of wavelengths out of range for a single sample is less than 10% of the total number of wavelengths, it can be retained, otherwise the sample needs to be eliminated.

[0086] The sample screening condition is expressed as follows:

[0087]

[0088]

[0089] in, is the jth wavelength of a single sample in the Master labeled spectral data. is the vector composed of the j-th wavelength of all samples in the Master unlabeled spectral data, and q is the number of spectral bands.

[0090] Step 5: Master spectrum matrix after sample screening using g(·) obtained in step 3 Perform model transfer, and use the spectral matrix after model transfer express, This is the near-infrared spectrum sample generated on the Slave.

[0091] Step 6: is the training set, Spectral preprocessing was performed for the test set, and a suitable nutrient content prediction model was established.

[0092] Sample model transfer evaluation indicators:

[0093] 1. Spectral transfer error rate STER

[0094] For the model transfer of the same sample between different spectrometers, the goal is to make the spectrum of the same sample Slave as consistent as possible with the spectrum of the Master after model transfer. In order to quantitatively describe this consistency, the spectra transfer error rate (STER) is used as an evaluation index for the model transfer of a single sample.

[0095] X s,ij (i=1,2,…,n;j=1,2,…,J) is the spectrum measurement value of the i-th sample at the j-th wavelength scanned by the Slave, X m,ij (i=1,2,…,n;j=1,2,…,J) is the spectrum measurement value of the i-th sample at the j-th wavelength scanned by the Master, For X m,ij The spectral value of the i-th sample at the j-th wavelength after model transfer, where n is the number of model transfer samples and J is the number of spectral wavelength points.

[0096] The transmission error rate for the i-th sample is defined as:

[0097]

[0098] For the entire model transfer training set, the average transfer error rate is used as the model transfer evaluation index, which is defined as:

[0099]

[0100] When STER<10%, the model transfer of a single sample / the entire training set is considered effective; when STER<5%, the model transfer effect of a single sample / the entire training set is considered good; when STER<1%, the model transfer effect of a single sample / the entire training set is considered very good.

[0101] 2. Root Mean Square Error (RMSE)

[0102] RMSE is the square root of the average of the squared differences between the slave spectral data and the master spectral data after transfer through the model, and is defined as follows:

[0103]

[0104] 3. Mean Absolute Error (MAE)

[0105] MAE is the average absolute error between the Slave spectral data and the Master spectral data after transfer through the model, and is defined as follows:

[0106]

[0107] The present embodiment is further described below:

[0108] Spectral model transfer is based on the difference of spectra scanned by different spectrometers to explore the mathematical relationship between the two. Therefore, before model transfer, it is necessary to ensure that there is a clear difference between the Master spectral data and the Slave spectral data in this embodiment. If the 33 pairs of spectral data involved in model transfer are basically the same, they can be directly regarded as spectral data scanned by the same spectrometer for modeling without model transfer.

[0109] Drawing a spectrum is the simplest and most effective way to explore spectral data. The spectrum can not only directly illustrate the differences and connections between Master and Slave, but also help determine whether there are outliers in the 33 pairs of spectral data involved in model transfer. By drawing a spectrum through 33 pairs of spectral data involved in model transfer, as shown in the figure Figure 2 as shown.

[0110] Figure 2 The differences and connections between the Master and Slave of the same group of samples can be viewed as a whole. In addition, two samples are randomly selected and the spectra are plotted separately. Figure 3 As shown, it is easy to view the relationship between the scanning results of a single sample on the Master and Slave: (the horizontal axis is the wavelength (nm), the vertical axis is the spectrum value, the upper line is the Master spectrum data, and the lower line is the Slave spectrum data)

[0111] from Figure 3 The following information can be seen:

[0112] 1. There are no outliers in the Master spectral data and the Slave spectral data;

[0113] 2. There are obvious differences between the spectral data of the same group of samples scanned on the Master and the spectral data scanned on the Slave, and they cannot be regarded as spectral data scanned by the same spectral instrument for modeling;

[0114] 3. The spectral data of the same sample scanned on the Master and the spectral data scanned on the Slave show the same trend, and there is a certain correlation between the two.

[0115] According to the information conveyed by the above spectrum comparison diagram, it can be determined that the spectrum data in this embodiment is suitable for model transfer and it is necessary to perform model transfer.

[0116] Then, the leave-one-out method is used to average the 33 transfer matrices to obtain the final transfer matrix. Here, the PDS left and right window length range is set to 0-18, and any combination can be performed under the condition that the sum of the left and right lengths does not exceed 18. RMSE is used as the evaluation index, and the left and right window lengths with the smallest RMSE are selected as the standard.

[0117] After adjusting the parameters, the left length of the PDS window is 1, and the right length is 17. The minimum RMSE is 0.0046. The spectral comparison of 33 unlabeled Master spectral data Master_X_33 before and after the PDS algorithm model transfer is shown in the figure below. Figure 4 shown.

[0118] In addition to the overall comparison of the model before and after the transfer, two samples were randomly selected to compare the single sample PDS algorithm spectrum before and after the transfer. Figure 5 As shown: (the horizontal axis is wavelength (nm), the vertical axis is spectrum value, from top to bottom are Master spectrum data, Slave spectrum data, and spectrum data after Master model transfer).

[0119] Depend on Figure 5 It can be seen that no matter it is the whole sample or a single sample, the PDS algorithm can establish an effective model transfer between the Master and the Slave. The spectral data of the Master's unlabeled spectral data after the PDS algorithm model transfer is very similar to the Slave's unlabeled spectral data, and the model transfer effect is good.

[0120] Before using the CCA algorithm, it is necessary to adjust the parameters of the grid. The RMSE of the training set is used as the evaluation index, and n_components = 31 is selected when the RMSE is the smallest. At this time, the spectral comparison of 33 unlabeled Master spectral data Master_X_33 before and after the CCA algorithm model transfer is shown in the figure below. Figure 6 shown.

[0121] In order to check the effect of single sample model transfer, two samples are randomly selected from the unlabeled spectral dataset, and the single sample CCA algorithm spectrum before and after transfer is compared. Figure 7 As shown: (the horizontal axis is wavelength (nm), the vertical axis is spectrum value, from top to bottom are Master spectrum data, Slave spectrum data, and spectrum data after Master model transfer).

[0122] Depend on Figure 7It can be seen that the CCA algorithm can also establish an effective model transfer on the unlabeled spectral data of the Master and Slave. After the model transfer, the unlabeled spectral data of the Master is closer to the unlabeled spectral data of the Slave. However, overall, the model transfer effect of the spectral data with a wavelength range of 950-1150 is average, and the problem of spectral data homogeneity occurs after the CCA algorithm performs model transfer.

[0123] The original Master unlabeled spectral data, the Master unlabeled spectral data after the PDS algorithm model transfer, and the Master unlabeled spectral data after the CCA algorithm model transfer were compared with the Slave unlabeled spectral data, and the STER, RMSE, and MAE in the three cases were calculated, and these three indicators were used as the difference measurement indicators between the two. The following table lists the detailed results of the model transfer of soybean meal data:

[0124]

[0125]

[0126] From the table data we can see that:

[0127] (1) Without model transfer, the STER of 33 pairs of unlabeled data Master_33 and Slave_33 is directly calculated to be 0.3137, the RMSE is 0.0551, and the MAE is 0.0548, indicating that there are serious differences between the spectral data of the same group of samples scanned on the Master and the spectral data scanned on the Slave. If the model transfer is not performed, the spectral data scanned by the same spectral instrument cannot be regarded as modeling;

[0128] (2) After the spectral data model was transferred by the PDS algorithm and the CCA algorithm, the spectral data difference was significantly reduced: the RMSE after the CCA algorithm model transfer was 0.0131, while the RMSE after the PDS algorithm conversion was 0.0046. In addition, the STER, RMSE, and MAE obtained by the PDS algorithm were all smaller than those of the CCA algorithm. This result shows that both the PDS algorithm and the CCA algorithm can establish an effective model transfer and improve the error of the spectral data obtained by different instruments, but the PDS algorithm model transfer effect is better. Therefore, the PDS algorithm will be used to transfer the model of the labeled Master spectral data Master_X, and then the subsequent research work will be carried out on the transferred spectral data.

[0129] The following formula is used to filter the Master labeled spectrum data Master_X:

[0130] num{Master_Xj >max(Master_33 j )or Master_X j <min(Master_33 j )}≤35

[0131] (j=1,2,…,351)

[0132] Among them, Master_X j The jth wavelength of a single sample on the Master labeled spectral data Master_X, Master_33 j is a vector consisting of the j-th wavelength of all samples on the Master unlabeled spectral data Master_33, and 35 is obtained by multiplying the total number of wavelengths 351 by 10%.

[0133] The sample screening operation can not only remove the samples whose spectral range in Master_X seriously exceeds the spectral data range of Master_33, but also remove the spectral abnormal samples in Master_X. The spectral comparison of Master_X before and after sample screening is shown in the figure below. Figure 8 as shown.

[0134] After sample screening, the number of samples of Master_X and its corresponding label Mater_Y changes from 2545 to 1558, and the new label set is represented by Mater_Y_PLS. Figure 8 It can be seen that the distribution of samples after screening is closer to the Master unlabeled dataset Master_33 used to establish the model transfer.

[0135] Next, the transfer matrix obtained by the PDS algorithm is used to transfer the model of Master_X after sample screening to obtain Master_X_trans. 100 samples are randomly selected from them to draw the comparison chart before and after the model transfer as shown below: Fig. 9 shown.

[0136] from Fig. 9 It can be seen that after the model transfer, the Master spectral data will move downward on average, and the model transfer through the PDS algorithm will shorten the front and back wavelengths of the spectral data. Therefore, new near-infrared spectral samples are generated on the Slave based on the model transfer. The subsequent research has 1558 training set samples and 17 test set samples.

[0137] Example 2

[0138] Take the soybean meal moisture content data provided by a manufacturer as an example: Currently, the manufacturer needs to build a model on its own spectral instrument (target machine Slave: Aura Handheld NIR). However, due to the high cost of obtaining the real label y, a total of 17 sample spectra were scanned on the Slave and 17 chemical experiments were conducted. That is, the Slave has 17 labeled experimental data. The main difficulty in modeling on the Slave is the insufficient amount of experimental data.

[0139] In addition to the spectrometer Slave, the manufacturer has currently obtained experimental data from another spectrometer (source machine Master: bruker matrix-i) with a large amount of label y. The Master has 2545 pieces of soybean meal moisture content data with label y, and the manufacturer hopes to make full use of this data to build a model on the Slave.

[0140] However, there are significant differences between different spectrometers. The spectral data of the same soybean meal sample scanned on the Master is quite different from that scanned on the Slave. Therefore, these 2545 labeled Master experimental data cannot be used directly on the Slave.

[0141] To solve this problem, 33 soybean meal samples were spectrally scanned on both the Slave and the Master. However, these 33 soybean meal samples were not subjected to chemical experiments to detect the precise moisture content, so they were unlabeled data. This method established the connection between the Master and the Slave through these 33 pairs of unlabeled spectral data, and realized the model transfer of 2545 labeled spectral data from the Master to the Slave.

[0142] The data used in the specific implementation involves two spectral instruments (source machine Master and target machine Slave), with a total of 6 data tables, as shown in the following table:

[0143]

[0144] The four data tables Master_33, Slave_33, Master_X and Slave_X are all spectral data, and their spectral wavelength range is uniformly 950nm-1650nm, and the spectral resolution is 2nm, that is, the original spectral data involved in this study are all composed of 351-dimensional variables.

[0145] Model transfer is to construct relevant mathematical function relationships through the spectral data scanned by the same group of soybean meal samples in the Master and Slave, so as to transfer the Master spectral data to the Slave through the determined mathematical relationship to complete the model transfer. Since segmented direct standardization (PDS) and canonical correlation analysis (CCA) are currently recognized as better model transfer methods, the PDS algorithm and CCA algorithm were used in this study to study the soybean meal model transfer through 33 unlabeled Master and Slave spectral data. STER, RMSE and MAE were used as the evaluation criteria for model transfer, and the algorithm with good model transfer effect was selected from the PDS algorithm and CCA algorithm as the standard, and the model transfer was performed on 2545 labeled Master experimental data using this algorithm.

[0146] Prior to this, considering that the amount of data to be transferred to the model is large, while the amount of data to be transferred to establish the model is small, if the range of the Master spectral data to be transferred seriously exceeds the range of the Master spectral data used for establishing the model transfer, the established model transfer may not be applicable to the samples that seriously exceed the spectral data range, or the model transfer effect may be poor. Therefore, before the model transfer, it is necessary to screen the 2545 labeled Master data, find and eliminate the samples whose spectral data range seriously exceeds the range of the 33 Master spectral data when establishing the model transfer. After completing the sample screening work, the model transfer can be performed on the screened labeled Master data.

[0147] Symbols involved in the study:

[0148]

[0149]

[0150] like Fig.10 As shown, the specific steps include:

[0151] Step 1: 33 pairs of raw spectral data X of soybean meal without label values m , X s (from Master and Slave, respectively) perform model transfer, and find the transformation relationship g(·) from Master to Slave. In this embodiment, g(·) represents the PDS algorithm or the CCA algorithm;

[0152] Step 2: In order to ensure the model transfer effect, it is necessary to collect 2545 original spectral data of soybean meal on the Master. Perform spectral sample screening to find and remove spectral ranges far exceeding X m The filtered spectral data are used express; also Label value of moisture content of soybean meal 2545 Also remove the corresponding label and get y 1558 .

[0153] Step 3: Master spectrum matrix after sample screening by g(·) obtained in step 1 Perform model transfer, and use the spectral matrix after model transfer express;

[0154] Step 4: The spectrum is preprocessed in the order of standard normal transformation SNV, trend correction DT, Savitzky-Golay smoothing, and forward derivative to obtain the processed spectral data. And and 1558 The KS algorithm is used to divide the training set and validation set, establish the prediction model f(·), and adjust the important parameters in the model in a grid-based manner;

[0155] Step 5: The original spectral data of soybean meal collected on 17 slaves Perform spectral preprocessing in the same order as before to obtain processed spectral data The corresponding predicted label value is obtained through the prediction model f(·)

[0156] like Fig.11 As shown, the prediction process:

[0157] When we get the new Slave spectrum data X to be detected, we first need to perform spectrum preprocessing on it to obtain the processed spectrum data X'. The preprocessing operation is also carried out in the order of standard normal transformation SNV, trend correction DT, Savitzky-Golay smoothing, and forward derivative. After completing the spectrum preprocessing, we use the preliminary prediction model f(·) established by the PLSR algorithm to obtain the final prediction value

[0158] The above-mentioned embodiments only express the specific implementation of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that, for ordinary technicians in this field, several variations and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.

Claims

1. A near infrared spectroscopy sample generation method based on model transfer under small sample conditions, characterized in that: The following steps are involved: Step 1: Scan a small amount of feed sample with a sample size of n1 on the secondary spectrometer, i.e., the spectrometer for which the model is to be established, to obtain a spectral matrix As the test set of the prediction model; let the spectral band scanned by the spectrometer be q, then is a spectral matrix of n1×q; at the same time, chemical experiments are carried out on these feed samples to detect their true nutrient content, that is, the label value; Step 2: Scan the feed sample with a sample size of n2 on the spectrometer to obtain a spectrum matrix X s , and then perform spectrum scanning on the main spectrometer to obtain the spectrum matrix X m , X m , X s That is, a set of n2×q spectral matrices for model transfer is used as unlabeled data; Step 3: Use X m , X s Perform model transfer and find the transformation relationship g(·) from the master spectrometer to the slave spectrometer; In step 3, the transformation relationship g(·) represents the PDS algorithm or the CCA algorithm, and the leave-one-out method is used, each pair of spectral data is used as a test set exactly once, and the obtained t transfer matrices are averaged to obtain the final transfer matrix; Step 4: For a large amount of spectral data with label y on the main spectrometer Perform spectral sample screening to find and remove spectral ranges far exceeding X m The sample after filtering is express; also Remove the corresponding label based on the nutrient content label value; In step 4, the main spectrometer with labeled spectral data The screening criteria are as follows: Calculate the unlabeled spectral data X of the master spectrometer involved in the model transfer m The maximum and minimum spectral values ​​at each wavelength are traversed Check the spectral values ​​of all spectral wavelengths in a single sample to see if they are greater than X m The maximum spectral value of this wavelength is less than X m The minimum spectral value of the wavelength is calculated. If one of the two conditions is met, the number of wavelengths out of range is counted. When the total number of wavelengths out of range of a single sample is less than a certain threshold of the total number of wavelengths, it can be retained, otherwise the sample is discarded. The sample screening condition is expressed as follows: (j=1,2,…,q) in, is the jth wavelength of a single sample of labeled spectral data of the main spectrometer, is the vector composed of the jth wavelength of all samples on the unlabeled spectral data of the main spectrometer, and q is the number of spectral bands; Step 5: The spectral matrix of the main spectrometer after sample screening by g(·) obtained in step 3 Perform model transfer, and use the spectral matrix after model transfer express, That is, the near-infrared spectrum sample generated from the spectrometer; Step 6: is the training set, Spectral preprocessing was performed for the test set, and a suitable nutrient content prediction model was established; The nutrient content prediction model, i.e., the sample model transfer evaluation index specifically includes: Spectral transfer error rate STER: Use the spectral transfer error rate STER as the evaluation indicator for single sample model transfer: X s,ij (i=1,2,…,n;j=1,2,…,J) is the spectrum measurement value of the i-th sample at the j-th wavelength scanned by the spectrometer, X m,ij (i=1,2,…,n;j=1,2,…,J) is the spectrum measurement value of the i-th sample at the j-th wavelength scanned by the main spectrometer, For X m,ij The spectral value of the i-th sample at the j-th wavelength after model transfer, where n is the number of model transfer samples and J is the number of spectral wavelength points; The transmission error rate for the i-th sample is defined as: For the entire model transfer training set, the average transfer error rate is used as the model transfer evaluation index, which is defined as: When STER < 10%, the model transfer of a single sample / the entire training set is considered effective; when STER < 5%, the model transfer effect of a single sample / the entire training set is considered good; when STER < 1%, the model transfer effect of a single sample / the entire training set is considered very good; Root mean square error RMSE: RMSE is the square root of the mean of the squared differences between the spectral data from the spectrometer and the spectral data from the master spectrometer after being transferred through the model, and is defined as follows: Mean absolute error (MAE): MAE is the average absolute error between the spectral data from the spectrometer and the spectral data from the master spectrometer after being transferred through the model, and is defined as follows:

2. The near infrared spectrum sample generation method based on model transfer under small sample conditions according to claim 1 is characterized in that: Before using the PDS algorithm for model transfer, it is necessary to adjust its parameters. Set the range of the PDS left and right window lengths to 0-(0.05*q), and make any combination under the condition that the sum of the left and right lengths does not exceed (0.05*q); and use RMSE as the evaluation indicator, and select the left and right window lengths with the smallest RMSE as the standard.

3. The near infrared spectrum sample generation method based on model transfer under small sample conditions according to claim 1, characterized in that: In the CCA algorithm, the number of components n_components retained by the CCA algorithm is gridded and adjusted before use, and the RMSE of the training set is used as the evaluation indicator, and the n_components with the minimum RMSE is taken.

Images