Sample screening and modeling method based on spectral similarity and chemical index difference

Abstract

The present invention discloses a sample screening and modeling method based on spectral similarity and chemical index difference, by examining the relationship between the most similar sample pairs and the chemical index difference in a sample set. By measuring the sample similarity, the difference range of the chemical values ​​of two samples with high similarity is obtained. As the similarity value decreases, the maximum chemical index difference increases. Samples with such a trend are valid samples, and the relationship between the spectrum and the chemical index can be well established regardless of the modeling method used. If this trend cannot be reflected, it means that the batch of data lacks the spectral distinction between chemical indicators, and therefore cannot be successfully modeled. Therefore, the method helps users to promptly eliminate factors such as improper modeling methods and fewer modeling samples, and more clearly gives whether the batch of data can be modeled, thereby saving necessary manpower and material resources, and is particularly suitable for users with low modeling and data analysis levels.

Description

Technical Field

[0001] The present invention belongs to the field of chemometrics, and in particular relates to a sample screening and modeling method based on spectral similarity and chemical index difference. Background Art

[0002] In the field of chemometrics, spectral analysis techniques, including near-infrared spectroscopy, mid-infrared spectroscopy, ultraviolet-visible spectroscopy, Raman spectroscopy, etc., are usually used to model the content of the substance of interest in the analyte. Generally speaking, the spectrum of the analyte is X, and the content of the substance of interest measured by analytical chemistry methods is Y. The correlation between the spectrum and the substance content is established through algorithms such as Partial Least Squares (PLS) to obtain a prediction model. For similar analytes with unknown content, by measuring their spectra, the content of the substance of interest in the sample can be obtained through the prediction model.

[0003] For example, the existing patent document CN110085286A discloses a method for evaluating the chemical suitability of tobacco strips for maintaining cigarette formulas, including the following steps: collecting tobacco strip samples and testing the chemical indicators of the tobacco strip samples; determining the confidence intervals of the chemical indicators of the currently used tobacco strips; establishing a tobacco strip chemical suitability function model; determining the suitability scores of the chemical indicators of the tobacco strip samples to be replaced; determining the weights of the chemical indicators; determining the chemical suitability evaluation values ​​of the tobacco strip samples to be replaced, and determining the selection and deployment order according to the chemical suitability evaluation values ​​of the tobacco strip samples to be replaced. The maintenance of the tobacco strip chemical suitability function model recorded in this method still requires the addition of a certain number of new samples, and may also eliminate the same number of oldest modeling samples at the same time to update the model, resulting in low detection efficiency.

[0004] Since this type of chemometric evaluation method achieves the effect of the original analytical method through modeling and prediction, it is a secondary method or indirect method. For a long time, whether a substance can be modeled and predicted in this way requires the accumulation of a certain number of samples, including spectra and corresponding analytical chemistry method detection values, and the use of PLS ​​and other methods to try to establish a model. If the modeling effect is good, for example, the modeling error is in line with expectations, independent samples are used for verification. If the model can still be verified by several independent samples, it means that the prediction model meets the practical standard. If the modeling effect is poor, for example, the modeling error is far beyond expectations, it means that the model is unsuccessful. But the cause of failure cannot be determined. That is, it cannot be determined that the reason for the failure of modeling is insufficient number of modeling samples, incorrect use of modeling methods, and the inability of the indicator to be modeled using the selected secondary measurement method (such as spectroscopy).

[0005] Since multiple indicators can be modeled separately by collecting the spectrum X of the object to be tested only once, users often hope to continue to expand the indicator range and model some substances with lower concentrations or lower detection accuracy after achieving the modeling of the main chemical components, such as protein and moisture in feed, nicotine and sugar in tobacco. For similar attempts, there is currently no method to estimate the experimental data and the model to be established, that is, to clearly obtain a conclusion on whether the batch of data can be successfully modeled. Summary of the invention

[0006] In view of the above, the purpose of the present invention is to provide a sample screening and modeling method based on spectral similarity and chemical indicator difference, which comprehensively considers the samples based on spectral similarity and chemical indicator difference to screen samples to obtain valid samples, and uses valid sample modeling to improve the prediction accuracy of the model.

[0007] In order to achieve the above-mentioned invention object, the present invention provides the following technical solutions:

[0008] In a first aspect, the embodiment provides a sample screening method based on spectral similarity and chemical index difference, comprising the following steps:

[0009] Obtain multiple samples, collect spectra of each sample, and analyze the chemical indicators of each sample;

[0010] After preprocessing the spectra of the samples, the spectral similarity between every two samples is calculated, and the difference value of the chemical index between every two samples is calculated;

[0011] Randomly extract any two samples from all samples to form a sample pair, forming a sample pair set, and each sample pair is characterized by spectral similarity and chemical index difference;

[0012] After sorting the sample pairs in the sample pair set according to spectral similarity, according to the set different levels, extract the sample pairs of different levels with the greatest similarity from the sample pair set to form sample pair groups, and calculate the maximum chemical index difference value and the minimum similarity value in each sample pair group;

[0013] After normalizing the maximum chemical index difference values ​​in all sample pairs, two key maximum chemical index difference values ​​are selected as two screening indicators based on the correlation scatter plot of the normalized maximum chemical index difference value and the minimum similarity value;

[0014] When the two screening indicators meet the screening conditions, this batch of samples is considered as valid samples.

[0015] The spectrum includes near infrared spectrum, mid infrared spectrum, ultraviolet visible spectrum and terahertz spectrum. The preprocessing of the spectrum of the sample includes a combination of one or more of first-order derivative processing, second-order derivative processing, smoothing, baseline removal, standard normal transformation and multivariate scattering correction in any order.

[0016] In one embodiment, the spectral similarity between two samples is calculated using the Pearson correlation coefficient and the Euclidean distance.

[0017] In one embodiment, the set level adopts a percentage system, and the percentage value is a natural number with a fixed interval between 1-100, and the fixed interval is 1, 2, 4, 5, 10, 15, 20, 25, and the fixed interval is preferably 1, 2, 4. Further, the fixed interval is preferably 1, and the percentage value is all natural numbers between 1 and 100. When the percentage value is 1, the set level is 1%, that is, the sample pairs with the largest similarity of 1% are extracted from the sample pair set to form a sample pair group.

[0018] Preferably, the normalization process of the maximum chemical index difference values ​​in all sample pairs includes:

[0019] A correlation scatter plot is formed based on the maximum chemical indicator difference value and the minimum similarity value in each sample pair group. In the area where the minimum similarity value of the correlation scatter plot is large and the maximum chemical indicator difference value is small, the maximum chemical indicator difference value corresponding to the boundary scatter point with a high degree of dispersion is selected as the normalized benchmark data, and the ratio of the maximum chemical indicator difference value in all sample pairs to the benchmark data is calculated as the normalization processing result.

[0020] Preferably, when the set level adopts a percentage system and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the level with a percentage value of 10-20 (that is, the level 10-20%) is used as the normalized reference data. Further preferably, the maximum chemical index difference value corresponding to the sample pair group corresponding to the 10% level is used as the normalized reference data.

[0021] In one embodiment, the method of selecting two key maximum chemical index difference values ​​as two screening indicators based on the correlation scatter plot of the normalized maximum chemical index difference value and the minimum similarity value includes:

[0022] In a correlation scatter plot formed according to the normalized maximum chemical index difference value and minimum similarity value, in an area where the minimum similarity value of the correlation scatter plot is small and the maximum chemical index difference value is large, the maximum chemical index difference value corresponding to the boundary scatter point with a high degree of dispersion is screened as the second screening indicator; in the correlation scatter plot, the maximum chemical index difference value corresponding to the central scatter point in the concentrated dispersion area is screened as the first screening indicator, thereby obtaining two screening indicators.

[0023] Preferably, when the set grade adopts a percentage system and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 80-90 is used as the second screening index, and the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 50 is used as the first screening index. Further preferably, the maximum chemical index difference value corresponding to the pair group corresponding to the grade of 90% and the grade of 50% is used as the second grade index and the first screening index respectively.

[0024] In one embodiment, when the two screening indicators meet the screening conditions, the samples are considered as valid samples, including:

[0025] The independent threshold and the correlation threshold of the screening indicators are set. When the two screening indicators are respectively greater than their respective independent thresholds and the ratio of the two screening indicators is greater than the correlation threshold, this batch of samples is considered to be valid samples and can be used for modeling.

[0026] Preferably, the independent threshold of the first screening indicator is 1.1-1.8, the independent threshold of the second screening indicator is 1.4-2.6, and the related threshold is 1.1-1.5. Preferably, when the set grade adopts the percentage system, and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 80 to 90 is used as the second screening indicator, and the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 50 is used as the first screening indicator, the independent threshold of the first screening indicator is 1.1-1.3, the independent threshold of the second screening indicator is 1.4-1.6, and the related threshold is 1.15-1.35. Further preferably, the independent threshold of the first screening indicator is 1.2, the independent threshold of the second screening indicator is 1.5, and the related threshold is 1.25.

[0027] The sample screening method based on spectral similarity and chemical index difference provided in the embodiment is by investigating the relationship between the most similar sample pairs and chemical index difference in the sample set. By measuring the sample similarity, the difference range of the chemical values ​​of the two samples under high similarity is obtained. As the similarity value decreases, the maximum chemical index difference increases, and the sample with such a trend is the effective sample, and the relationship between the spectrum and the chemical index can be well established no matter which modeling method is adopted. If this trend cannot be reflected, it means that the batch of data lacks the distinction between the spectrum and the chemical index, and therefore cannot be successfully modeled. Therefore, the method helps the user to timely eliminate factors such as improper modeling methods and fewer modeling samples, and more clearly gives whether the batch of data can be modeled, thereby saving necessary manpower and material resources, and is particularly suitable for users with low modeling and data analysis levels.

[0028] In a second aspect, the embodiment further provides a modeling method based on spectral similarity and chemical index difference, comprising the following steps:

[0029] The above-mentioned sample screening method based on spectral similarity and chemical index difference is used to screen and obtain valid data;

[0030] A prediction model between spectra and chemical indicators is established using valid data. The prediction model is used to predict chemical indicators based on the input spectra.

[0031] The modeling method based on spectral similarity and chemical index difference provided in the embodiment has screened valid data, and the prediction model established using valid data is more robust and can achieve accurate prediction of chemical indicators based on the spectrum. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0033] Figure 1 is the relative difference value of the spectral similarity and chemical value between two samples of total sugar data provided in the embodiment;

[0034] Figure 2 The embodiment provides Figure 1 The result diagram obtained by normalizing the data in the figure;

[0035] Figure 3 It is a rendering of modeling four indicators of training data using partial least squares method on the basis of obtaining valid data provided by the embodiment;

[0036] Figure 4 is the relative difference between the spectral similarity value and the chemical value between the two samples of starch data provided in the embodiment;

[0037] Figure 5 The embodiment provides Figure 2 The result diagram obtained by normalizing the data in the figure;

[0038] Figure 6 It is a diagram showing the effect of using the partial least squares method to model starch indicators, as provided in the embodiment. DETAILED DESCRIPTION

[0039] To make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific implementation methods described herein are only used to explain the present invention and do not limit the scope of protection of the present invention.

[0040] Example 1

[0041] Take the prediction of tobacco components by near infrared spectroscopy as an example. 180 flue-cured tobacco samples from a certain enterprise were selected. After sampling, the samples were prepared into powder samples according to the tobacco industry standard "YC / T 31-1996 Preparation and moisture determination of tobacco and tobacco products samples - oven method" (the tobacco leaves were placed in an oven, dried at 40°C for 4 hours, and ground with a cyclone mill (FOSS) to pass through a 40-mesh sieve), and sealed and balanced for 1 day before near infrared spectroscopy measurement. At the same time, the total sugar, nicotine, and reducing sugar contents of the flue-cured tobacco samples were measured using analytical chemistry methods as chemical values ​​of different chemical indicators.

[0042] Then, after the first-order derivative and segmented transformation of the obtained near-infrared spectrum, the similarity between every two flue-cured tobacco samples in all flue-cured tobacco samples is calculated. In the embodiment, the Pearson correlation coefficient and the Euclidean distance are used to calculate the similarity. The specific process is: Similarity s = -log 10 ((1-p)*d). Where p is the Pearson correlation coefficient, p=corr(x1,x2), x1 x2 represents the target spectrum and the comparison spectrum after preprocessing. d is the Euclidean distance, d=sum(abs(x1-x2)).

[0043] At the same time, for each of the three chemical indicators of sugar, nicotine and reducing sugar, the relative difference value of the chemical value of the two flue-cured tobacco samples for each chemical indicator (that is, the chemical indicator difference) is calculated respectively. Then, the sample pairs consisting of all two samples are used to construct a sample pair set, and the similarity between the two samples is matched with the relative difference value of the chemical value. That is, the sample pairs consisting of two samples are characterized by the spectral similarity and the relative difference value of the chemical value to form the following: Figure 1 The two-dimensional display shown in the figure, Figure 1 In the figure, the horizontal axis shows the spectral similarity between samples, and the vertical axis shows the relative difference of chemical values ​​between samples.

[0044] Next, the sample pairs in the sample pair set are sorted from high to low according to the spectral similarity, and the top 1%, 2%, ..., 100% sample pairs with the largest similarity are selected to form 100 sample pair groups. For each sample pair group, the relative difference c of the maximum chemical value is calculated. max and the minimum spectral similarity s min , let Q = [c max1 c max2 …c max100], in the embodiment, the relative difference c of the maximum chemical value of the sample pair group corresponding to 10% is selected max10 is the normalization basis. Using this normalization basis, the normalization result Q1=Q / c max10 ; The normalized result and the corresponding minimum spectral similarity form a related scatter plot as shown Figure 2 As shown in Figure 2 There are 100 points in total, representing the sample s when the sampling is the first 1%, 2%, ..., 100%. min and c max .analyze Figure 2 , respectively select the relative differences Q1(90) and Q1(50) of the normalized maximum chemical values ​​corresponding to 90% and 50% as the first screening index P1 and the second screening index P2. Figure 2 It can be seen that when data modeling is possible, as the similarity between samples decreases, the difference between the chemical values ​​corresponding to the samples increases. That is, in the sample set, there is an obvious negative correlation between the minimum similarity value and the maximum chemical value difference, which is a sign that the data set can be modeled.

[0045] Table 1 is the calculated values ​​of various indicators of the data in Example 1

[0046] index Q(10) Q(50) Q(90) p1 p2 Total Sugar 0.18 0.30 0.44 1.68 2.49 Nicotine 0.29 0.49 0.74 1.68 2.51 Reducing sugar 0.22 0.29 0.39 1.31 1.77

[0047] For numerical purposes, in the embodiment, the independent threshold of the first screening indicator is set to 1.2, the independent threshold of the second screening indicator is set to 1.5, and the correlation threshold of the second screening indicator and the first screening indicator is set to 1.25. The screening indicators of each of the above chemical indicators are judged based on two independent thresholds and one correlation threshold, and it is found that the two screening indicators P1 and P2 are both greater than their respective independent thresholds, and the ratio of P2 to P1 is greater than the correlation indicator. Therefore, the chemical values ​​and spectra of each indicator are valid data and can be used for modeling. Table 2 shows the modeling effect of using PLS for modeling in the data of Example 1.

[0048] Table 2

[0049] index Number of PLS ​​latent variables RMSEC RMSECV Total Sugar 19 0.01 0.02 Nicotine 20 0.02 0.03 Reducing sugar 16 0.02 0.03

[0050] It can be seen that after PLS modeling, the predicted results of chemical values ​​are very close to the true values.

[0051] Example 2

[0052] The starch test for a batch of data gave incorrect starch chemical test values ​​due to lack of training of test personnel and incorrect standard sample preparation. This method confirms this problem.

[0053] Example 2 uses the same method as Example 1. After the chemical value and infrared spectrum of starch are detected, the infrared spectrum is subjected to first-order derivative and piecewise transformation processing, and the spectral similarity and relative difference of chemical values ​​between the two samples are calculated. Then, the two samples are matched to form sample pairs, and a sample pair set is formed and presented in a two-dimensional display diagram as shown in 3.

[0054] Similarly, after sorting the sample pairs in the sample pair set from high to low according to the spectral similarity, the top 1%, 2%, ..., 100% sample pairs with the largest similarity are selected to form 100 sample pair groups, and the relative difference c of the maximum chemical value is calculated for each sample pair group. max and the minimum spectral similarity s min , we get Q = [c max1 c max2 …c max100 ], in the embodiment, the relative difference c of the maximum chemical value of the sample pair group corresponding to 10% is selected max10 is the normalization basis. Using this normalization basis, the normalization result Q1=Q / c max10 ; The normalized result and the corresponding minimum spectral similarity form a related scatter plot as shown Figure 4 As shown in . Figure 5 It can be seen that the Q1 curve does not show a good linear relationship with a negative slope. It is calculated that p1=Q1(50)=0.973, p2=Q1(90)=0.986, p2 / p1=1.01. All three indicators fail to meet the standard for successful modeling.

[0055] The partial least squares method is used to model the data. The modeling results are: number of latent variables: 1, RMSEC = 0.16, RMSECV = 0.17, and the modeling effect is shown in Figure 5 It can be seen that the PLS method was not successful in modeling this batch of data, which is consistent with the results of this method.

[0056] Therefore, the method provided by the present invention provides a new perspective on whether data can be modeled, and provides a powerful tool for modeling trace chemical indicators and artificial sensory evaluation indicators through spectral data.

[0057] The specific implementation methods described above provide a detailed description of the technical solutions and beneficial effects of the present invention. It should be understood that the above is only the most preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, supplements and equivalent substitutions made within the scope of the principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A sample screening method based on spectral similarity and chemical index difference, comprising the following steps: Obtain multiple samples, collect spectra of each sample, and analyze the chemical indicators of each sample; After preprocessing the spectra of the samples, the spectral similarity between every two samples is calculated, and the difference value of the chemical index between every two samples is calculated; Randomly extract any two samples from all samples to form a sample pair, forming a sample pair set, and each sample pair is characterized by spectral similarity and chemical index difference; After sorting the sample pairs in the sample pair set according to spectral similarity, according to the set different levels, extract the sample pairs of different levels with the greatest similarity from the sample pair set to form sample pair groups, and calculate the maximum chemical index difference value and the minimum similarity value in each sample pair group; After normalizing the maximum chemical index difference values ​​in all sample pairs, two key maximum chemical index difference values ​​are selected as two screening indicators based on the correlation scatter plot of the normalized maximum chemical index difference value and the minimum similarity value, including: in a correlation scatter plot formed based on the normalized maximum chemical index difference value and the minimum similarity value, in an area where the minimum similarity value of the correlation scatter plot is small and the maximum chemical index difference value is large, the maximum chemical index difference value corresponding to the boundary scatter point with a high degree of dispersion is selected as the second screening indicator; in the correlation scatter plot, the maximum chemical index difference value corresponding to the central scatter point of the dispersed and concentrated area is selected as the first screening indicator, thereby obtaining two screening indicators; When two screening indicators meet the screening conditions, this batch of samples is considered to be valid samples, including: setting independent thresholds and correlation thresholds for the screening indicators. When the two screening indicators are respectively greater than their respective independent thresholds, and the ratio of the two screening indicators is greater than the correlation threshold, this batch of samples is considered to be valid samples and can be used for modeling.

2. The sample screening method based on spectral similarity and chemical index difference according to claim 1, characterized in that: The spectrum includes near infrared spectrum, mid infrared spectrum, ultraviolet visible spectrum, and terahertz spectrum; The preprocessing of the spectrum of the sample includes a combination of one or more of first-order derivative processing, second-order derivative processing, smoothing, baseline removal, standard normal transformation, and multivariate scatter correction in any order.

3. The sample screening method based on spectral similarity and chemical index difference according to claim 1, characterized in that: The Pearson correlation coefficient and Euclidean distance were used to calculate the spectral similarity between two samples.

4. The sample screening method based on spectral similarity and chemical index difference according to claim 1, characterized in that: The set level is based on a percentage system, and the percentage value is a natural number with a fixed interval between 1-100, and the fixed interval values ​​are 1, 2, 4, 5, 10, 15, 20, 25.

5. The sample screening method based on spectral similarity and chemical index difference according to claim 4, characterized in that: The fixed intervals are 1,2,4.

6. The sample screening method based on spectral similarity and chemical index difference according to claim 1 or 4, characterized in that: The normalization process of the maximum chemical index difference value in all sample pairs includes: A correlation scatter plot is formed based on the maximum chemical indicator difference value and the minimum similarity value in each sample pair group. In the area where the minimum similarity value of the correlation scatter plot is large and the maximum chemical indicator difference value is small, the maximum chemical indicator difference value corresponding to the boundary scatter point with a high degree of dispersion is selected as the normalized benchmark data, and the ratio of the maximum chemical indicator difference value in all sample pairs to the benchmark data is calculated as the normalization processing result.

7. The sample screening method based on spectral similarity and chemical index difference according to claim 6, characterized in that: When the set grade adopts the percentage system and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 10-20 is used as the normalized benchmark data.

8. The sample screening method based on spectral similarity and chemical index difference according to claim 1, characterized in that: The method of selecting two key maximum chemical index difference values ​​as two screening indicators based on the correlation scatter plot of the normalized maximum chemical index difference value and the minimum similarity value includes: When the set grade adopts the percentage system and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 80~90 is used as the second screening index, and the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 50 is used as the first screening index.

9. The sample screening method based on spectral similarity and chemical index difference according to claim 1 or 8, characterized in that: The independent threshold of the first screening indicator is 1.1-1.8, the independent threshold of the second screening indicator is 1.4-2.6, and the correlation threshold is 1.1-1.

5.

10. The sample screening method based on spectral similarity and chemical index difference according to claim 1 or 8, characterized in that: When the set grade adopts the percentage system and the percentage value is all natural numbers between 1-100, the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 80~90 is used as the second screening index, and the maximum chemical index difference value corresponding to the sample pair group corresponding to the grade with a percentage value of 50 is used as the first screening index, the independent threshold value of the first screening index is 1.1-1.3, the independent threshold value of the second screening index is 1.4-1.6, and the correlation threshold value is 1.15-1.

35.

11. A modeling method based on spectral similarity and chemical index difference, comprising the following steps: Utilize the sample screening method based on spectral similarity and chemical index difference as described in any one of claims 1 to 10 to screen and obtain valid data; A prediction model between spectra and chemical indicators is established using valid data. The prediction model is used to predict chemical indicators based on the input spectra.

Images