Biomass spectrum self-correlation analysis method and system and application of biomass spectrum self-correlation analysis method and system

Abstract

The invention provides a biomass spectrum self-correlation analysis method and system and application thereof, and belongs to the technical field of spectrum self-correlation analysis. The analysis method comprises the following steps: acquiring sample spectrum data, acquiring probability distribution characteristics of each wavelength point of a spectrum, calculating pairwise correlation of the probability distribution characteristics among different wavelength points, drawing a probability distribution characteristic autocorrelation result graph of the absorbance of each wavelength point according to a correlation coefficient R or a significance P value obtained by a statistical method, and calculating a probability distribution characteristic autocorrelation result graph of the absorbance of each wavelength point according to the autocorrelation result graph. And setting a threshold value to judge the probability distribution characteristic correlation degree of the wavelength point absorbance. According to the method, the similarity of probability distribution of absorbance data of different wavelengths of a spectrum is taken as a judgment basis of a spectrum autocorrelation rule, nonparametric analysis of biomass spectrum autocorrelation is realized, an autocorrelation characteristic region of biomass under a certain wavelength is effectively judged, and a foundation is laid for selection of a detection absorption wavelength for subsequent detection of the biomass.

Description

Technical Field

[0001] The present invention belongs to the technical field of spectral autocorrelation analysis, and in particular relates to a biomass spectral autocorrelation analysis method, system and application thereof. Background Art

[0002] Biomass is abundant in production, widely distributed, and rich in wood fiber, nutrients and other ingredients. It can be used as raw material for products in a variety of industries such as energy, feed, fertilizer, and chemicals. Rapidly and efficiently obtaining the composition and content of biomass raw materials is an important link in the high-value, low-carbon industrial application of biomass.

[0003] Spectral analysis technology is a green and environmentally friendly detection method that can be used in the industrial chain and production and processing of agriculture, petrochemicals, pharmaceuticals, food, feed, etc. It can quickly and efficiently determine the physical and chemical composition and characteristics of samples, becoming one of the indispensable analytical methods for enterprises and scientific research institutions, and can be effectively applied to the detection of the composition and content of biomass raw materials. However, due to the particularity of the spectral band and the complexity of biomass composition, biomass spectral data has the characteristics of large volatility and the prevalence of multicollinearity, which makes spectral data difficult to analyze and is not conducive to predicting the composition characteristics of biomass. Therefore, it is urgent to deeply explore the autocorrelation relationship of spectral data. However, the existing autocorrelation analysis is based on parameter analysis, and mainly uses Pearson correlation analysis or covariance analysis methods to explore the linear correlation between spectral wavelengths, hoping to analyze the relationship between spectra with an accurate linear data model, which is very effective for pure substances. When the research object is a complex polymer represented by biomass, the above existing methods are prone to overfitting, and the results are not robust, resulting in the correlation results being false, coincidental, and non-reproducible. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a biomass spectral autocorrelation analysis method, system and application thereof. The method uses the similarity of the probability distribution of absorbance data at different wavelengths of the spectrum as the basis for distinguishing the spectral autocorrelation law, realizes the non-parametric analysis of biomass spectral autocorrelation, effectively judges the autocorrelation characteristic area of ​​biomass at a certain wavelength, and lays the foundation for selecting the detection absorption wavelength for subsequent detection of biomass.

[0005] The present invention provides a biomass spectrum autocorrelation analysis method, comprising the following steps: obtaining sample spectrum data; obtaining probability distribution characteristics of each wavelength point of the spectrum by using a tool for analyzing data distribution based on the spectrum data; calculating pairwise correlations of probability distribution characteristics between different wavelength points by using a statistical method based on the probability distribution characteristics of each wavelength point of the spectrum; drawing an autocorrelation result diagram of probability distribution characteristics of absorbance at each wavelength point according to the correlation coefficient |R| or significance P value obtained by the statistical method, and setting a threshold value to judge the correlation degree of probability distribution characteristics of absorbance at the wavelength point.

[0006] Preferably, the number of samples is no less than 30.

[0007] Preferably, the tools for analyzing data distribution include: a histogram method, a classical probability distribution function method or a kernel density curve fitting method.

[0008] Preferably, the statistical method includes: correlation analysis, univariate linear fitting, variance analysis, chi-square test or Kolmogorov-Smirnov test.

[0009] Preferably, when the Kolmogorov-Smirnov test is used for statistical analysis, the probability distribution characteristics of each wavelength point of the spectrum are standardized, and the method used for standardization includes: first calculating the average value and standard deviation of the absorbance at each wavelength point of the spectrum, and then obtaining the standardized value according to the formula (absorbance-average value) / standard deviation.

[0010] Preferably, correlation analysis is used as a statistical method, and the correlation coefficient |R|≥0.8 is used as a threshold value, that is, when |R|≥0.8, there is a correlation between the two wavelength points and there is autocorrelation, and when |R|<0.8, there is a difference between the two wavelength points and there is no autocorrelation; the Kolmogorov-Smirnov test is used as a statistical means, and the significance P≥0.05 is used as a threshold value, that is, when P≥0.05, there is a correlation between the two wavelength points and there is autocorrelation, and when P<0.05, there is a difference between the two wavelength points and there is no autocorrelation.

[0011] The present invention provides a biomass spectrum autocorrelation analysis system, which is used to execute the biomass spectrum autocorrelation analysis method, and comprises: a spectrum data acquisition module, a spectrum data processing module and a visualization module; the spectrum data acquisition module adopts a spectrometer to collect data; the spectrum data processing module comprises a tool for analyzing data distribution to obtain the probability distribution characteristics of each wavelength point of the spectrum and adopts a statistical method to calculate the pairwise correlation of the probability distribution characteristics between different wavelength points; the visualization module draws a probability distribution characteristic autocorrelation result diagram of the absorbance of each wavelength point according to the correlation coefficient |R| or the significance P value obtained by the statistical method.

[0012] Preferably, the data processing module is connected to the visualization module.

[0013] Preferably, the data processing module includes tools for analyzing data distribution and data statistics methods.

[0014] The present invention provides the application of the biomass spectrum autocorrelation analysis method and the biomass spectrum autocorrelation analysis system in analyzing the autocorrelation of the biomass spectrum of a sample.

[0015] Compared with the prior art, the present invention has the following beneficial effects:

[0016] The biomass spectrum autocorrelation analysis method of the present invention uses the similarity of probability distribution of absorbance data of different wavelengths of the spectrum as the basis for judging the spectrum autocorrelation law, realizes the non-parametric analysis of biomass spectrum autocorrelation, effectively judges the autocorrelation characteristic area of ​​biomass at a certain wavelength, and lays the foundation for selecting the detection absorption wavelength for subsequent detection of sample biomass. The method belongs to a non-parametric method, has high robustness, does not produce overfitting phenomenon, and the results can be reproduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 This is the probability distribution fitting result of rice straw spectral data.

[0018] Figure 2 The results of autocorrelation analysis of rice straw spectral data.

[0019] Figure 3 This is the probability distribution fitting result of laying hen feces spectral data.

[0020] Figure 4 This is the result of autocorrelation analysis of spectral data of laying hen feces.

[0021] Figure 5 This is the probability distribution fitting result of straw spectral data.

[0022] Figure 6 This is the result of autocorrelation analysis of straw spectral data. DETAILED DESCRIPTION

[0023] The present invention provides a biomass spectrum autocorrelation analysis method, comprising the following steps: obtaining sample spectrum data; obtaining the probability distribution characteristics of each wavelength (wave number) point of the spectrum using a tool for analyzing data distribution based on the spectrum data; using a statistical method to calculate the pairwise correlation of the probability distribution characteristics between different wavelength (wave number) points based on the probability distribution characteristics of each wavelength (wave number) point of the spectrum; using a two-dimensional heat map, density map or contour map drawing method to draw the probability distribution characteristics autocorrelation result map of the absorbance of each wavelength (wave number) point according to the correlation coefficient |R| or the significant P value obtained by the statistical method, setting a threshold, and judging the correlation degree of the probability distribution characteristics of the absorbance of the wavelength (wave number) point. The method of the present invention uses the similarity of the probability distribution of the absorbance data of different wavelengths of the spectrum as the basis for distinguishing the spectrum autocorrelation law, and realizes the non-parametric analysis of the biomass spectrum autocorrelation.

[0024] In the present invention, the number of samples is not less than 30, and more preferably not less than 50. The present invention collects spectral data for each sample at least 3 times and takes the average value. The samples of the present invention need to be dried and crushed in the early stage, preferably dried in an oven at 35-55°C to constant weight, in order to simulate the natural drying state; the samples are crushed to less than 1 mm, the finer the better, preferably 20 mesh-80 mesh (about 0.83-0.18 mm). The present invention uses a spectrometer to collect sample spectral data, and can collect molecular spectral or atomic spectral data such as near-infrared, mid-infrared, far-infrared, Raman, ultraviolet-visible light, fluorescence, laser-induced breakdown spectroscopy, etc., preferably collecting near-infrared spectral data. The method for collecting near-infrared spectral data of the present invention includes transmission, diffuse reflection, integrating sphere, total reflection, etc.

[0025] In the present invention, the tool for analyzing data distribution includes: a histogram method, a classical probability distribution function method or a kernel density curve fitting method. The present invention adopts one or more methods of a histogram, a classical probability distribution function or a kernel density curve fitting to obtain the probability distribution characteristics (frequency, probability distribution density curve, probability distribution density function or cumulative probability distribution model) of the absorbance data at each wavelength (wave number) point of the spectrum. The tool for analyzing data distribution described in the present invention is an analysis tool well known in the art.

[0026] In the present invention, the statistical method includes but is not limited to the following methods: correlation analysis, univariate linear fitting, variance analysis, chi-square test or Kolmogorov-Smirnov test. The probability distribution characteristics (frequency) obtained by the present invention with a histogram are preferably obtained by correlation analysis, univariate linear fitting, and variance analysis methods to obtain the degree of correlation of the probability distribution characteristics; the probability distribution characteristics (probability distribution density function) obtained by the classical probability distribution function method are preferably calculated by chi-square test and Kolmogorov-Smirnov test methods to calculate the autocorrelation of the probability distribution characteristics; the probability distribution characteristics obtained by the kernel density curve fitting method are preferably calculated by correlation analysis, chi-square test, and Kolmogorov-Smirnov test methods to calculate the autocorrelation characteristics of the probability distribution characteristics. The present invention needs to select a suitable statistical method according to the method for obtaining the probability distribution characteristics. The statistical method described in the present invention is a statistical method well known in the art.

[0027] In the present invention, when the Kolmogorov-Smirnov test is used for statistical analysis, the probability distribution characteristics of each wavelength point of the spectrum are standardized, and the method used for standardization includes: first calculating the average value and standard deviation of the absorbance of each wavelength (wave number) point of the spectrum, and then obtaining the standardized value according to the formula (absorbance-average value) / standard deviation. The present invention is based on the standardized value obtained by standardizing the data of each wavelength (wave number) point of the spectrum, and uses a statistical method to calculate the autocorrelation characteristics of the probability distribution characteristics; the statistical method is consistent with the statistical method for calculating the pairwise correlation between the probability distribution characteristics of the original data of each wavelength (wave number) point of the spectrum. The present invention needs to compare the autocorrelation results obtained by using the probability distribution characteristics of the original data of the absorbance of each wavelength (wave number) point of the spectrum with the autocorrelation results obtained after standardization of the probability distribution characteristics of the original data of the absorbance of each wavelength (wave number) point of the spectrum, in order to exclude the lack of autocorrelation of the spectrum due to the data dimension, or to analyze whether the spectrum is different due to the dimension.

[0028] The statistical methods of the present invention include but are not limited to the following methods: correlation analysis, univariate linear fitting, variance analysis, chi-square test or Kolmogorov-Smirnov test.

[0029] In the present invention, correlation analysis is used as a statistical method, and the correlation coefficient |R|≥0.7 is used as a threshold value, that is, when |R|≥0.7, there is correlation between two wavelength (wave number) points, there is autocorrelation, and when |R|<0.7, there is difference between two wavelength points, and there is no autocorrelation; Kolmogorov-Smirnov test is used as a statistical means, and significance P≥0.05 is used as a threshold value, that is, when P≥0.05, there is correlation between two wavelength points, there is autocorrelation, and when P<0.05, there is difference between two wavelength points, and there is no autocorrelation.

[0030] The present invention also provides a biomass spectrum autocorrelation analysis system for executing the biomass spectrum autocorrelation analysis method, comprising: a spectrum data acquisition module, a spectrum data processing module and a visualization module; the spectrum data acquisition module uses a spectrometer to collect data; the spectrum data processing module includes a tool for analyzing data distribution to obtain the probability distribution characteristics of each wavelength (wave number) point of the spectrum and uses a statistical method to calculate the pairwise correlation of the probability distribution characteristics between different wavelength (wave number) points; the visualization module draws a probability distribution characteristic autocorrelation result diagram of the absorbance at each wavelength point based on the correlation coefficient |R| or the significance P value obtained by the statistical method. The data processing module of the present invention is connected to the visualization module. The data processing module of the present invention includes a tool for analyzing data distribution and a data statistical method.

[0031] The present invention also provides the application of the biomass spectrum autocorrelation analysis method and the biomass spectrum autocorrelation analysis system in analyzing the autocorrelation of the biomass spectrum of a sample.

[0032] The technical solutions provided by the present invention are described in detail below in conjunction with the embodiments, but they should not be construed as limiting the protection scope of the present invention.

[0033] Example 1

[0034] (1) Fifty rice straw samples were collected, dried at 45±5°C, and crushed to less than 0.5 mm. About 20 g of the sample was placed in a test sample cup, and the sample cup was placed on the scanning platform of a Sup-NIR 2750 diffuse reflectance infrared spectrometer (Juguang Technology, China). The sample spectral data was obtained and saved in a readable data format. The spectral wavelength range was 1000-2500 nm, and the resolution was 1 nm.

[0035] (2) Based on the above spectral data, the histogram analysis method was used to obtain the probability distribution characteristics of each wave number point of the rice straw spectrum ( Figure 1 ), the specific method is as follows: first determine the maximum and minimum absorbance of each wavelength point in the spectrum, then obtain the number of grids k of the absorbance of each wavelength point according to formula (1-3) and take the average, and get the average number of grids as 7, that is, each wavelength point interval [starting value (SV), ending value (EV)] is divided into 7 parts on average; finally, count the number of absorbances of each wavelength point of all spectral data in the 7 intervals, and obtain the probability distribution characteristics of the absorbance of each wavelength point.

[0036]

[0037] Among them, SV represents the starting value, EV represents the ending value, k represents the number of grids, and x min 、x max They represent the minimum and maximum absorbance values ​​in the sorted absorbance value sequence, IQR represents the interquartile range, and m represents the number of absorbance values ​​in the sorted absorbance value sequence. Indicates rounding down. Indicates rounding up.

[0038] (3) The probability distribution characteristics of the absorbance at each wavelength point of the spectrum obtained above are subjected to pairwise Pearson correlation analysis (Formula 4). Then, according to the results obtained from the above analysis and the correlation coefficient |R| value, a contour map of the autocorrelation results of the probability distribution characteristics of the absorbance at each wavelength point is drawn. The correlation coefficient |R| ≥ 0.8 is selected as the threshold value. The autocorrelation results of the rice straw spectrum are as follows: Figure 2As shown in the figure, the black area represents the autocorrelation characteristics between the two wavelengths of the spectrum. The results show that the rice straw spectrum has two obvious autocorrelation characteristic areas at 1000-1400mm wavelength and 1400-2500mm, which means that the spectrum in this area contains similarities. In the subsequent spectrum prediction and wavelength screening, the spectral points in the autocorrelation characteristic area should be avoided to be selected simultaneously and concentratedly.

[0039]

[0040] Among them, R ij Pairwise Pearson correlation coefficient, p, represents the probability distribution characteristics of absorbance at wavelength points i 、p j They represent the probability distribution characteristics of absorbance at two wavelength points, cov represents covariance, σp i ,σp j Respectively represent the standard deviation of the absorbance probability distribution characteristics of the two wavelength points.

[0041] Example 2

[0042] (1) 500 laying hen feces samples were collected, dried at 70±5℃, and crushed to less than 1 mm. About 10 g of the sample was placed in a test sample cup, and the sample cup was placed on the scanning platform of a Bruker MPAII spectrometer (Bruker, Germany). The sample spectral data was obtained and saved in a readable data format. The wave number range was 10000-4000 cm -1 , resolution 2cm -1 .

[0043] (2) Based on the spectral data collected above, the kernel density curve fitting method was used to obtain the probability distribution characteristics of each wave number point of the laying hen feces spectrum ( Figure 3 ), the specific method is as follows: first, select the Gaussian function as the kernel function, calculate the kernel density bandwidth of each wavelength point according to the Silverman empirical method, then sort each wavelength point separately, use the kernel function and bandwidth fitting to calculate the Gaussian distribution of the absorbance data of each wavelength point, and finally superimpose the Gaussian distribution of the absorbance of each wavelength point to form the probability distribution characteristics of the absorbance of each wavelength point.

[0044] (3) The probability distribution characteristics of the absorbance at each wavelength point obtained in step (2) are subjected to a pairwise Kolmogorov-Smirnov test (KS test) to obtain the autocorrelation KS test significance P matrix of the absorbance at each wavelength point of the spectrum.

[0045] (4) Further standardize the probability distribution characteristic data of the absorbance at each wavelength point obtained in step (2), first calculate the mean value and standard deviation of the absorbance at each wavelength point of the spectrum, and then obtain the standardized value according to the formula (absorbance-mean value) / standard deviation; then perform kernel density curve fitting and KS test on the standardized value in turn to obtain the autocorrelation KS test significance P matrix of the absorbance at each wavelength point of the spectrum after standardization.

[0046] (5) Then, according to the analysis results obtained in step (3) and step (4) and the significance P value, a two-dimensional heat map of the probability distribution characteristic autocorrelation results of the absorbance at each wavelength point is drawn. Taking the significance P ≥ 0.05 as the threshold, the autocorrelation results of the laying hen feces spectrum are obtained as follows: Figure 4 As shown in the figure, the lower left part is the result of the KS test of the significance P matrix of the absorbance autocorrelation of each wavelength point of the unstandardized spectrum, and the upper right part is the result of the KS test of the significance P matrix of the absorbance autocorrelation of each wavelength point of the standardized spectrum. The white area represents the autocorrelation characteristics between the two wavenumbers of the spectrum. Comparing the results of the KS test of the significance P matrix of the absorbance autocorrelation of each wavelength point of the spectrum before and after standardization, the red color before standardization and the white color after standardization represent the correlation between the two wavenumbers of the spectrum, and the difference between the two wavenumbers of the spectrum is mainly reflected in the mean or standard deviation (size or ratio); before standardization, there are wide differences between the wavelength points of the laying hen feces spectrum, 7000cm -1 10000-6000cm -1 There is a correlation between the 7000cm -1 Contains more chemical value information; after standardization, the spectral data can be roughly divided into 10000-7000cm -1 , 7000-5000cm -1 4700-4000cm -1 The three autocorrelation intervals mean that the spectral difference in this area is only the mean and standard deviation of the absorbance. After scaling the data, the wavelength points in these wavelength ranges may contain similar information. When using spectra to predict chemical values, we should try to avoid selecting spectral wavelength points in the same area to avoid overfitting the model.

[0047] Example 3

[0048] (1) 2900 straw samples were collected, including rice straw, wheat straw, corn straw, rape straw, cotton straw, etc. The samples were dried naturally and crushed to less than 20 mesh. About 20 grams of sample was placed in the test sample cup, and the sample cup was placed on the scanning platform of PerkinElmer Spectrum400 Fourier transform spectrometer (PerkinElmer, USA). The sample spectral data was obtained and saved in a readable data format. The spectral wavenumber range was 10000-4000cm-1 , resolution 2cm -1 .

[0049] (2) Based on the spectral data collected above, the kernel density curve fitting method was used to obtain the probability distribution characteristics of each wave number point of the straw spectrum ( Figure 5 ). The experimental method is the same as step (2) of Example 2.

[0050] (3) The probability distribution characteristics of each wave number point of the above spectrum are subjected to Spearman correlation analysis (Formula 5) to obtain the Spearman autocorrelation matrix of each wave number point of the spectrum. Then, according to the results obtained from the above analysis and the correlation coefficient |R| value, a contour map of the probability distribution characteristics of the absorbance at each wavelength point is drawn. The correlation coefficient |R| ≥ 0.9 is used as the threshold value. The autocorrelation result of the straw sample spectrum is as follows: Figure 6 As shown in the figure, the black area represents the autocorrelation characteristics between the two wave numbers of the spectrum. The spectral data can be roughly divided into 10000-7000cm -1 、7000-5300cm -1 、5300-4800cm -1 4800-4000cm -1 The four autocorrelation intervals mean that the wavelength points within these wavelength ranges may contain similar information. When using spectra to predict chemical values, we should try to avoid selecting spectral wavelength points in the same area to avoid overfitting the model.

[0051]

[0052] In the formula, r s represents the Spearman correlation coefficient, R(p i )、R(p j ) represent the rank of the probability distribution characteristics of absorbance at each wavelength, and n is the number of spectra.

[0053] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A biomass spectrum autocorrelation analysis method, characterized in that: The steps include: Obtain sample spectrum data; Based on the spectral data, the probability distribution characteristics of each wavelength point of the spectrum are obtained by using the tool for analyzing data distribution; Based on the probability distribution characteristics of each wavelength point of the spectrum, a statistical method is used to calculate the pairwise correlation of the probability distribution characteristics between different wavelength points; According to the correlation coefficient |R| or the significant P value obtained by the statistical method, the probability distribution characteristic autocorrelation result diagram of the absorbance at each wavelength point is drawn, and the threshold is set to judge the correlation degree of the probability distribution characteristic of the absorbance at the wavelength point.

2. The biomass spectrum autocorrelation analysis method according to claim 1, characterized in that: The number of samples is no less than 30.

3. The biomass spectrum autocorrelation analysis method according to claim 1, characterized in that: The tools for analyzing data distribution include: a histogram method, a classical probability distribution function method or a kernel density curve fitting method.

4. The biomass spectrum autocorrelation analysis method according to claim 1, characterized in that: The statistical methods include: correlation analysis, univariate linear fitting, variance analysis, chi-square test or Kolmogorov-Smirnov test.

5. The biomass spectrum autocorrelation analysis method according to claim 1, characterized in that: When the Kolmogorov-Smirnov test is used for statistical analysis, the probability distribution characteristics of each wavelength point of the spectrum are standardized. The method used for standardization includes: first calculating the average value and standard deviation of the absorbance of each wavelength point of the spectrum, and then obtaining the standardized value according to the formula (absorbance-average value) / standard deviation.

6. The biomass spectrum autocorrelation analysis method according to claim 1, characterized in that: Correlation analysis was used as a statistical method, and the correlation coefficient |R|≥0.7 was used as the threshold. When |R|≥0.7, the two wavelength points were correlated and autocorrelation existed. When |R|<0.7, the two wavelength points were different and there was no autocorrelation. Kolmogorov-Smirnov test was used as a statistical means, and significance P≥0.05 was used as the threshold. When P≥0.05, the two wavelength points were correlated and autocorrelation existed. When P<0.05, the two wavelength points were different and there was no autocorrelation.

7. A biomass spectrum autocorrelation analysis system, characterized in that: Used to perform the biomass spectrum autocorrelation analysis method according to any one of claims 1 to 6, comprising: a spectrum data acquisition module, a spectrum data processing module and a visualization module; The spectrum data acquisition module uses a spectrometer to collect data; The spectrum data processing module includes a tool for analyzing data distribution to obtain the probability distribution characteristics of each wavelength point of the spectrum and to calculate the pairwise correlation of the probability distribution characteristics between different wavelength points using a statistical method; The visualization module draws a probability distribution characteristic autocorrelation result diagram of the absorbance at each wavelength point according to the correlation coefficient |R| or the significance P value obtained by the statistical method.

8. The biomass spectrum autocorrelation analysis system according to claim 7, characterized in that: The data processing module is connected to the visualization module.

9. The biomass spectrum autocorrelation analysis system according to claim 7, characterized in that: The data processing module includes tools for analyzing data distribution and data statistics methods.

10. Use of the biomass spectrum autocorrelation analysis method according to any one of claims 1 to 6 and the biomass spectrum autocorrelation analysis system according to any one of claims 7 to 9 in analyzing the autocorrelation of biomass spectrum of a sample.

Images