A wavelength selection method in near infrared spectroscopy quantitative analysis

Abstract

The wavelength selection method in near-infrared quantitative analysis provided by the application adopts a hyperspectral imaging system to obtain a hyperspectral image of a standard reference object, and adopts a micro spectral sensor to obtain a visible-near-infrared reflection spectrum of an analysis object. For the hyperspectral image of the standard reference object, second-order difference operation and smoothing denoising are performed on the spectral data to obtain a standard reference object data set; for the visible-near-infrared reflection spectrum of the analysis object, abnormal data is removed, a Monte Carlo sampling method is used for analysis, then a wavelet packet decomposition and reconstruction method is executed for data smoothing to obtain an analysis object data set. In the modeling process, the classification criteria of different data sets are fully mined, and the Euclidean distance similarity is judged, so that different classes of data sets are objectively obtained, and the largest and smallest data sets of the two classes are selected as the calibration set and the prediction set, respectively, so that better modeling effect can be achieved.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of spectral analysis, and particularly relates to a wavelength selection method in near-infrared spectral quantitative analysis, a device for implementing the method and a computer system. BACKGROUND

[0002] Infrared light and near-infrared light are electromagnetic radiation waves between visible light (Vis) and mid-infrared (MIR). The American Society for Testing and Materials (ASTM) defines the near-infrared spectral region as the region of 780-2526 nm, which is the first non-visible light region discovered in the absorption spectrum. The near-infrared spectral region is the absorption of the combination frequency and the multiple frequency of the hydrogen-containing groups (O-H, N-H, C-H) in molecules. By scanning the near-infrared spectrum of a sample, the characteristic information of the hydrogen-containing groups in the sample can be obtained. The use of near-infrared spectral technology to analyze samples has the advantages of convenience, rapidity, high efficiency, accuracy, low cost, no sample destruction, no consumption of chemical reagents, and no environmental pollution, and therefore the technology is favored by more and more people.

[0003] After nearly half a century of development, near-infrared spectral analysis technology has become one of the most promising analysis technologies in the new century. Many countries have established specialized research forces to develop related application field instruments and equipment. Reducing the cost of instruments while maintaining sufficient analysis performance has become the dominant direction of current near-infrared instrument development. Many developed countries in Europe have already used this technology as a standard technology for evaluating the quality of industry products in many fields, almost completely replacing the previously widely used chemical analysis method, and achieving good results in production efficiency and product quality.

[0004] China started late in the research of near-infrared spectral analysis technology. In the late 1980s, Changchun Institute of Optics, Fine Mechanics and Physics undertook the "Eight-Five" scientific and technological research project of the State Grain Bureau and successfully developed a filter-type feed near-infrared analyzer. In the following decade, filter-type near-infrared analyzers for analyzing corn, wheat, and soybeans were developed. At present, research and instrument development are being carried out in the fields of ginseng, human blood sugar, coal, honey, and tea.

[0005] Chinese invention patent application CN201810912233.7 proposes a near-infrared spectral data analysis method based on support vector machines, and Chinese invention patent application CN201610824094.3 proposes a handheld near-infrared spectral detection system and method for fruit and vegetable quality. Both utilize a near-infrared spectrometer designed with digital micromirror devices and configured with a single-point detector to obtain a miniaturized, low-cost near-infrared spectral system for fruit and vegetable quality. This avoids the need for expensive linear detector arrays while achieving high-performance spectral information acquisition. Regarding the establishment of the fruit and vegetable quality detection model, firstly, characteristic bands are selected, and bands with no informational variables and low correlation are removed. Then, a characteristic wavelength selection method is used to optimize a small number of characteristic wavelengths, eliminating collinearity within the spectral data, reducing the computational load of the model, simplifying the model, and improving its quality.

[0006] However, although near-infrared analyzers have achieved satisfactory results in agricultural applications by using multiple linear regression to establish calibration models, there is still a lack of effective solutions to problems such as how to perform correlation calculations between near-infrared spectral absorbance data of the analyte and reference chemical data under specific combinations of variables, the characteristic relationships between each spectral variable and the analyte, and the instability caused by sample particle size and scattering effects. Summary of the Invention

[0007] This invention proposes a wavelength selection method for near-infrared spectroscopy quantitative analysis, an apparatus for implementing the method, and a computer system, which can solve the above-mentioned problems existing in the prior art.

[0008] In a first aspect of the invention, a wavelength selection method for near-infrared spectroscopy quantitative analysis is provided, wherein the method employs a hyperspectral imaging system to acquire a hyperspectral image of a standard reference object and a miniature spectroscopic sensor to acquire the visible-near-infrared reflectance spectrum of the object to be analyzed.

[0009] Unlike existing near-infrared spectroscopy quantitative analysis, which only models and analyzes the sample to be tested, the technical solution of this invention first requires the use of a hyperspectral imaging system to acquire hyperspectral images of standard reference objects; here, the standard reference objects are a pre-established library of standard reference objects relative to the sample to be tested, i.e., the object to be analyzed.

[0010] Therefore, as the first innovation of this invention, for the hyperspectral image of the standard reference object acquired by the hyperspectral imaging system, after performing second-order difference operation on its spectral data, the standard normalization transformation formula combined with the polynomial smoothing method is used to smooth and denoise the data to obtain the standard reference object dataset.

[0011] For the visible-near-infrared reflectance spectra of the analyzed object acquired by the micro-miniature spectroscopic sensor, the Monte Carlo sampling method is used to analyze the spectral dataset to remove outliers, and then wavelet packet decomposition and reconstruction methods are performed to smooth the data to obtain the dataset of the analyzed object.

[0012] Based on this, the second innovative aspect of the present invention includes the following steps in implementing the method:

[0013] S101: The standard reference object dataset is classified using principal component analysis to obtain at least three categories of standard reference object data subsets Refi, i = 1, 2, 3, ...;

[0014] S103: For each type of standard reference object data subset Refi, select multiple analysis object data sets whose similarity satisfies the first predetermined condition from the analysis object dataset to form at least three types of analysis object data subsets Stdj, j = 1, 2, 3, ...;

[0015] S105: Select max{Stdj, j = 1, 2, 3, ...} as the calibration set and min{Stdj, j = 1, 2, 3, ...} as the prediction set. Use a competitive adaptive weighting algorithm to select the optimal wavelength for the characteristic wavelength.

[0016] It should be noted that this method of dividing the calibration set and the prediction set is completely different from the existing technology and can achieve better modeling results (see the data comparison in the implementation examples for details).

[0017] The selection of multiple analysis object data sets whose similarity satisfies a first predetermined condition from the analysis object dataset includes:

[0018] Calculate the least-squares distance (LSP) between the spectral data x of the analysis object dataset and the data subset Refi of each class of standard reference objects:

[0019]

[0020] Among them, ref center The center value of the standard reference object data subset Refi;

[0021] If the LSP is less than the predetermined value, then the spectral data is selected as an element of the subset Stdj of the analysis object data.

[0022] Unlike existing technologies that simply preprocess the original samples and then divide them into training (calibration) and prediction sets based on experience for modeling, the modeling process of this invention fully explores the classification criteria of the dataset itself, uses similarity judgment, and thus objectively derives datasets of different classes. The datasets with the largest and smallest values ​​of the two classes are selected as the calibration and prediction sets, respectively, which can achieve better modeling results.

[0023] In specific implementation, the micro-miniature spectroscopic sensor includes an optical sensor, a data storage and transmission module and a controller. The optical sensor is externally connected to a fiber optic probe, which is perpendicular to the surface of the object being analyzed. After acquiring the hyperspectral image data of the standard reference object using a hyperspectral imaging system, a matrix-style spectral database of the standard reference object is generated.

[0024] In one aspect, a competitive adaptive weighted algorithm is used to select the optimal wavelength for the characteristic wavelength, including:

[0025] A competitive adaptive weighting algorithm combined with a continuous projection algorithm was used to select characteristic wavelengths, and a partial least squares (PLS) model was established and validated.

[0026] At this point, the wavelength points with larger absolute values ​​of regression coefficients (preferably the largest) in the PLS model can be screened using an adaptive weighted sampling method, and the wavelength points with smaller weights (preferably the smallest) can be removed. The spectral variables corresponding to the subset with the smallest root mean square error in the PLS model are then cross-validated as the optimal variable subset.

[0027] On the other hand, a competitive adaptive weighted algorithm is used to select the optimal wavelength for the characteristic wavelength, including:

[0028] A competitive adaptive weighted algorithm combined with a random frog jumping algorithm is used to extract feature bands and establish a partial least squares regression (PLSR) model. The random frog jumping algorithm combines the advantages of meme algorithms based on genetic characteristics and particle swarm optimization based on behavior, making it suitable for solving various combinatorial optimization problems. The random frog jumping algorithm is simple to understand and highly robust.

[0029] At this point, after each PLSR model is established, the regression coefficient of each feature band in the model is recorded. The feature band with the smallest regression coefficient is removed, the range of feature bands is narrowed, and the model is built again. Finally, the importance of each feature band is statistically analyzed, and the optimal feature band is selected from them.

[0030] Furthermore, the above-described method of the present invention can be implemented by a computer program, which is stored in a readable media, a computer-readable medium, a readable optical disc, etc. Therefore, a computer-readable storage medium is also provided, which stores computer-executable instructions, and the instructions are executed by a processor to implement the aforementioned method.

[0031] In another aspect of the invention, a computer system is provided for automatically performing wavelength selection in near-infrared spectral quantitative analysis. The computer system is connected to a hyperspectral imaging system and a miniature spectroscopic sensor. The hyperspectral imaging system, the miniature spectroscopic sensor, and the data processing engine of the computer system communicate with each other. After receiving spectral data acquired by the hyperspectral imaging system and the miniature spectroscopic sensor, the data processing engine executes the aforementioned method.

[0032] In another aspect of the present invention, a wavelength selection device for near-infrared spectroscopy quantitative analysis is also disclosed. The device includes a hyperspectral imaging system and a miniature spectroscopic sensor, wherein the miniature spectroscopic sensor is a portable and mobile device. The wavelength selection device further includes a data smoothing and noise reduction engine, a wavelet packet decomposition and reconstruction engine, and an optimal wavelength selection engine for implementing the aforementioned method.

[0033] Further advantages of the present invention will be further demonstrated in conjunction with the accompanying drawings in the specific embodiments section. Attached Figure Description

[0034] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0035] Figure 1 This is a block diagram showing the overall implementation of the wavelength selection method in this application;

[0036] Figure 2 This is the main flowchart of the wavelength selection method of this application;

[0037] Figure 3 This is a schematic diagram of the structure of the micro-miniature spectroscopic sensor of this application;

[0038] Figure 4 This is a structural diagram of the wavelength selection device of this application;

[0039] Figure 5 This is a classification diagram of a subset of standard reference object data according to an embodiment of this application;

[0040] Figure 6 This is a schematic diagram illustrating the determination of the correction set and prediction set of the data subset analyzed in one embodiment of this application;

[0041] Figure 7 This is a comparison diagram of the effects of one embodiment of this application and the prior art. Specific Implementation

[0042] To better understand the above-mentioned objectives, features, and advantages of the present invention, the present invention will be further described below in conjunction with the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features described in these embodiments can be combined with each other.

[0043] See Figure 1 This is a wavelength selection method in near-infrared spectroscopy quantitative analysis according to an embodiment of the present invention.

[0044] In this embodiment, a hyperspectral imaging system is first used to acquire a hyperspectral image of a standard reference object, and a miniature spectroscopic sensor is used to acquire the visible-near-infrared reflectance spectrum of the object to be analyzed.

[0045] As a non-limiting embodiment, the standard reference object is based on the object to be analyzed, but not on a pre-established standard object database;

[0046] Taking the analysis of crop moisture content as an example, the standard reference object can be a crop cultivated under laboratory conditions according to set environmental parameters, and the physicochemical parameters of the crop (including moisture content, sugar content, etc.) can be obtained through chemical determination methods;

[0047] Correspondingly, the object of analysis is the sample to be tested; taking crops as an example, it is crops growing in the wild. Therefore, this embodiment uses a portable micro-miniature spectroscopic sensor to acquire the visible-near-infrared reflectance spectrum of the object of analysis.

[0048] Figure 3 A schematic diagram of a miniature spectroscopic sensor is provided. The miniature spectroscopic sensor includes an optical sensor, a data storage and transmission module, and a controller. The optical sensor is externally connected to an optical fiber probe, which is perpendicular to the surface of the object being analyzed.

[0049] More specifically, the controller is connected to spectral control software, which includes modules for setting acquisition parameters, acquisition control, and data management, each implementing its corresponding function. See details below. Figure 3 .

[0050] Therefore, unlike the existing near-infrared spectroscopy quantitative analysis which only models and analyzes the sample to be tested, this embodiment first needs to use a hyperspectral imaging system to obtain hyperspectral images of standard reference objects; here, the standard reference objects are a pre-established standard reference object library relative to the sample to be tested, that is, the object to be analyzed.

[0051] Furthermore, in this embodiment, a data preprocessing method suitable for the characteristics of the spectral data obtained from the standard reference object and the analysis object is adopted.

[0052] Specifically, for the hyperspectral image of the standard reference object acquired by the hyperspectral imaging system, after performing second-order difference operations on its spectral data, a standard normalization transformation formula combined with a polynomial smoothing method is used to smooth and remove noise, resulting in a standard reference object dataset; for the visible-near-infrared reflectance spectrum of the analysis object acquired by the micro-miniature spectroscopic sensor, the spectral dataset is analyzed using the Monte Carlo sampling method to remove outlier data, and then wavelet packet decomposition and reconstruction methods are performed to smooth the data, resulting in an analysis object dataset.

[0053] It should be noted that the data preprocessing methods used for the different types of spectral data are different. For example, the spectral data of the hyperspectral image of the standard reference object does not require sampling, while the visible-near-infrared reflectance spectrum of the analysis object acquired by the miniature spectroscopic sensor requires sampling analysis.

[0054] See further Figure 2 The method includes the following steps:

[0055] S101: The standard reference object dataset is classified using principal component analysis to obtain at least three categories of standard reference object data subsets Refi, i = 1, 2, 3, ...;

[0056] In this embodiment, to achieve better results, the standard reference object data subset is divided into four categories, see [link to relevant documentation]. Figure 5 As stated above.

[0057] S103: For each type of standard reference object data subset Refi, select multiple analysis object data that satisfy the first predetermined condition from the analysis object dataset to form at least three types of analysis object data subsets Stdj, j = 1, 2, 3...;

[0058] In this embodiment, the corresponding subset of the analysis object data is divided into 4 categories;

[0059] S105: Select max{Stdj, j = 1, 2, 3, ...;} as the calibration set and min{Stdj, j = 1, 2, 3, ...;} as the prediction set, and use a competitive adaptive weighting algorithm to select the optimal wavelength for the characteristic wavelength;

[0060] In this embodiment, correspondingly, after the data subsets of the analysis object are divided into 4 categories, the subset with the most elements in the subsets Stdj,j=1,2,3... is selected as the calibration set, and the subset with the fewest elements in the subsets Stdj,j=1,2,3... is selected as the prediction set.

[0061] Taking a classification into 4 categories as an example (i = 1, 2, 3, 4, j = 1, 2, 3, 4), max{Stdj, j = 1, 2, 3, 4) refers to the subset with the most elements in the four subsets Std1, Std2, Std3, and Std4, and min{Stdj, j = 1, 2, 3, 4} refers to the subset with the fewest elements in the four subsets Std1, Std2, Std3, and Std4.

[0062] Figure 6 The diagram shows four subsets of the data set being analyzed.

[0063] In this embodiment, selecting multiple analysis object data sets whose similarity satisfies a first predetermined condition from the analysis object dataset includes:

[0064] Calculate the least-squares distance (LSP) between the spectral data x of the analysis object dataset and the data subset Refi of each class of standard reference objects:

[0065]

[0066] Among them, ref center The center value of the standard reference object data subset Refi;

[0067] If the LSP is less than the predetermined value, then the spectral data is selected as an element of the subset Stdj of the analysis object data.

[0068] The center value of the standard reference object data subset Refi can be the geometric center value or the algebraic average of all elements in the subset; this embodiment does not impose any restrictions on this.

[0069] Furthermore, selecting multiple data points from the dataset of analyzed objects whose similarity satisfies the first predetermined condition can also be achieved in other ways, and is not limited to the least squares distance (LSP) method described above. All methods well-known to those skilled in the art for determining data similarity can be used.

[0070] More specifically, in the above embodiments, a competitive adaptive weighted algorithm is used to screen feature wavelengths, establish a partial least squares regression (PLSR) model, and verify it.

[0071] Specifically, this involves using an adaptive weighted sampling method to select the wavelength points with the largest absolute values ​​of regression coefficients in the PLSR model, removing the wavelength points with the smallest weights, and cross-validating the subset of spectral variables corresponding to the subset with the smallest root mean square error in the PLS model as the optimal subset of variables.

[0072] Competitive Adaptive Weighted Algorithm (CARS) is a variable selection method that simulates Darwin's theory of evolution, specifically the "survival of the fittest." It is well-known to those skilled in the art and will not be elaborated upon here.

[0073] In another embodiment, a competitive adaptive weighted algorithm combined with a random frog-jumping algorithm is used to extract feature bands and establish a partial least squares regression (PLSR) model. After each PLSR model is established, the regression coefficient of each feature band in the model is recorded. The feature band with the smallest regression coefficient is removed, the range of feature bands is narrowed, and the model is re-built. Finally, the importance of each feature band is statistically analyzed, and the optimal feature band is selected.

[0074] The random frog algorithm is a feature variable selection method proposed by Li HD et al. It can perform modeling using a small number of variables iteratively and is a very effective method for selecting variables in high-dimensional data (for specific implementation, see frog Random: an efficient reversible jump Markov Chain Monte Carlo-like approach for visible selection with applications to gene selection and disease classification).

[0075] To better demonstrate the technical improvements of this invention compared to existing methods, near-infrared spectral analysis of the sugar content of a certain food is used as an example. Figure 7 Comparative data on the effects of this invention and existing technologies (other models under all variables) are provided.

[0076] Existing technologies include Signal Vector Machine (SVM), simple CARS algorithm, relatively new variable selection methods such as Monte Carlo elimination of uninformed variables (mcuve), Particle Swarm Optimization (PSO), and Genetic Algorithm (GA). These methods all employ full variables in their modeling process, performing simple preprocessing on the original samples before empirically dividing them into training (calibration) and prediction sets for modeling. In contrast, the technical solution in this application fully leverages the classification criteria of the dataset itself, using similarity judgment to objectively derive different classes of datasets. It then selects the datasets with the largest and smallest values ​​from the two classes as the calibration and prediction sets, respectively, achieving better modeling results.

[0077] As can be seen, the model index (RMSEP, root mean square error) of this invention is significantly better than that of existing technology models.

[0078] Through the specific embodiments described above, those skilled in the art can easily implement the present invention. However, it should be understood that the present invention is not limited to the specific embodiments described above. Based on the disclosed embodiments, those skilled in the art can arbitrarily combine different technical features to achieve different technical solutions.

Claims

1. A wavelength selection method in near-infrared quantitative analysis, the method comprising: acquiring a hyperspectral image of a standard reference object by using a hyperspectral imaging system, and acquiring a visible-near-infrared reflectance spectrum of an analysis object by using a miniature spectroscopy sensor; wherein: the hyperspectral image of the standard reference object is subjected to a second-order difference operation on spectral data, and then is smoothed and denoised by using a standard normalization transformation formula combined with a polynomial smoothing method to obtain a standard reference object data set; the visible-near-infrared reflectance spectrum of the analysis object is subjected to a Monte Carlo sampling method to analyze and remove abnormal data, and then is subjected to a wavelet packet decomposition and reconstruction method to smooth the data, thereby obtaining an analysis object data set; the method comprises the following steps: S101: classifying the standard reference object data set by using a principal component analysis method to obtain at least three standard reference object data subsets Refi, i = 1, 2, 3,...; S103: for each standard reference object data subset Refi, selecting a plurality of analysis object data from the analysis object data set that satisfy a first predetermined condition of similarity to form at least three analysis object data subsets Stdj, j = 1, 2, 3,...; S105: selecting max{Stdj, j = 1, 2, 3,...} as a calibration set and selecting min{Stdj, j = 1, 2, 3,...} as a prediction set, and performing optimal wavelength selection of characteristic wavelengths by using a competitive adaptive reweighted algorithm; wherein, the selecting of the plurality of analysis object data from the analysis object data set that satisfy the first predetermined condition of similarity comprises: calculating a least square distance LSP between spectral data x of the analysis object data set and each standard reference object data subset Refi; and if the LSP is less than a predetermined value, selecting the spectral data as an element of the analysis object data subset Stdj; wherein, the performing of the optimal wavelength selection of the characteristic wavelengths by using the competitive adaptive reweighted algorithm comprises: screening the characteristic wavelengths by using the competitive adaptive reweighted algorithm combined with a successive projections algorithm, establishing a partial least squares regression PLSR model, and verifying the PLSR model; wherein, the performing of the optimal wavelength selection of the characteristic wavelengths by using the competitive adaptive reweighted algorithm comprises: extracting the characteristic wavelengths by using the competitive adaptive reweighted algorithm combined with a random frog algorithm, and establishing a partial least squares regression PLSR model. The miniature spectroscopy sensor comprises an optical sensor, a data storage and transmission module, and a controller, the optical sensor is circumscribed by a fiber probe, and the fiber probe is perpendicular to a surface of the analysis object. After acquiring the hyperspectral image data of the standard reference object by using the hyperspectral imaging system, a matrix spectral database of the standard reference object is generated. The wavelength point with the largest absolute value of a regression coefficient in a PLS model is screened by using an adaptive weighted sampling method, the wavelength point with the smallest weight is removed, and a spectral variable corresponding to a subset with the smallest root mean square error in the PLS model is determined as an optimal variable subset by cross-validation. ​ ​ ​ ​ wherein ref center is the central value of the standard reference object data subset Refi; ​ ​ ​ ​ ​ 2. The method of claim 1, wherein, ​ 3. The method of claim 1, wherein, ​ 4. The method of claim 1, wherein, ​ 5. The method of claim 1, wherein, After the PLSR model is established each time, the regression coefficient of each characteristic wave band in the model is recorded, the characteristic wave band with the smallest regression coefficient is removed, the range of the characteristic wave band is reduced, and modeling is performed again, the importance of each characteristic wave band is finally counted, and the optimal characteristic wave band is screened from the characteristic wave bands.

6. A wavelength selection device in near infrared spectroscopy quantitative analysis, the device comprising a hyperspectral imaging system, a micro-spectroscopy sensor, characterized in that, The micro-spectroscopy sensor is a portable mobile device; the wavelength selection device further comprises a data smoothing and noise reduction engine, a wavelet packet decomposition and reconstruction engine, and an optimal wavelength selection engine, which are used to realize the wavelength selection method in the near-infrared spectroscopy quantitative analysis according to any one of claims 1-5. 7.A computer system for automatically realizing wavelength selection in near-infrared spectroscopy quantitative analysis, wherein the computer system is connected with a hyperspectral imaging system and a micro-spectroscopy sensor, the hyperspectral imaging system, the micro-spectroscopy sensor, and a data processing engine of the computer system are in communication, and after the data processing engine receives spectral data acquired by the hyperspectral imaging system and the micro-spectroscopy sensor, the data processing engine executes the wavelength selection method in the near-infrared spectroscopy quantitative analysis according to any one of claims 1-5.

Images