A method for identifying the origin of rehmannia based on a table-based model

Abstract

The application discloses a hyperspectral radix scrophulariae origin identification method based on a tabular foundation model. In order to improve the feature analysis accuracy and reasoning efficiency of the hyperspectral identification task under limited samples, the application comprises the following steps: first, collecting the hyperspectral data of radix scrophulariae and extracting the ROI average spectrum; second, preprocessing the spectrum, automatically identifying and removing the abnormal spectrum by calculating the Mahalanobis distance in the sample feature space; then, performing SG filtering smoothing and extreme value normalization, continuously calculating the first and second derivatives and the relative intensity reflecting the global proportion; then, splicing the absolute intensity, relative intensity and spectral derivative (first and second) of the full waveband spectrum to reconstruct a high-dimensional structured matrix containing multi-dimensional spectral features, and adapting the input format of the tabular foundation model; finally, using the excellent context learning mechanism of the tabular model, the reference sample set is combined to realize fast reasoning. The application can fully mine and utilize high-dimensional spectral features, without complicated gradient iteration steps, and realizes low sample dependence, extremely fast and high-precision intelligent nondestructive identification of radix scrophulariae origin.

Description

Technical Field

[0001] This invention relates to the fields of non-destructive intelligent detection of agricultural products and Chinese medicinal materials, spectral analysis and machine learning, specifically to a hyperspectral method for identifying the origin of Rehmannia glutinosa based on tabular foundation models and in-context learning. Background Technology

[0002] Rehmannia glutinosa (Libosch.), one of the four traditional Chinese medicinal herbs from Huai, possesses extremely high pharmacological activity and clinical application value. Due to differences in soil trace elements, climate, and hydrology, the content of its core active ingredients, such as catalpol, rehmannia glycoside, and verbascoside, varies significantly depending on the origin and cultivation method of the rehmannia. For example, genuine rehmannia from Jiaozuo, Henan (formerly known as Huaiqing Prefecture) often has far higher medicinal efficacy and market value than those from other origins or counterfeit substitutes. Therefore, rapid, non-destructive, and accurate identification of the true origin of rehmannia is of great significance for regulating the market order of traditional Chinese medicine, ensuring the safety of clinical medication, and maintaining the reputation of geographical indication products.

[0003] Currently, traditional methods for identifying the origin of Rehmannia glutinosa mainly include sensory identification and physicochemical analysis (such as high-performance liquid chromatography, mass spectrometry, etc.). Sensory identification relies heavily on expert experience, is highly subjective, and is difficult to quantify; while physicochemical analysis has high detection accuracy, it generally suffers from bottlenecks such as time-consuming sample pretreatment, sample destruction, and high detection costs, making it difficult to meet the needs of large-scale, real-time sampling in the modern Chinese medicinal materials industry chain.

[0004] In recent years, hyperspectral imaging (HSI) technology has been widely used in the identification of traditional Chinese medicinal materials due to its advantages of "image-spectrum integration" and non-destructive testing. Traditional hyperspectral data analysis usually relies on partial least squares analysis (PLS-DA) or support vector machines (SVM), and often requires feature reduction using algorithms such as continuous projection (SPA). However, forced dimensionality reduction can easily lead to the loss of weak key chemical absorption peak information. On the other hand, although deep learning models such as convolutional neural networks (CNN) introduced in recent years have improved performance, these models heavily rely on massive amounts of spectral data with precise origin labels for lengthy gradient iteration training. In practical applications, they face extremely high sample collection and labeling costs, and are prone to overfitting in small sample scenarios.

[0005] In recent years, tabular foundation models, as a novel type of pre-trained structured data classification and regression model, have demonstrated remarkable few-shot context learning capabilities and rapid inference potential. However, when directly applying hyperspectral waveforms to this type of model, there is a significant input mechanism mismatch problem. The core of this invention lies in achieving morphological adaptation between continuous waveforms and tabular inputs, reconstructing thousands of dimensions of continuous spectral signals into high-dimensional structured feature vectors that fit the model.

[0006] Based on this, the purpose of this invention is to provide a hyperspectral intelligent identification method for the origin of Rehmannia glutinosa that can fully explore multidimensional spectral features, achieve efficient reasoning with few samples, and fully unleash the rapid classification potential of the table-based model. Summary of the Invention

[0007] To address the problems described in the background art, this invention proposes a hyperspectral method for identifying the origin of Rehmannia glutinosa based on tabular foundation models. To achieve the above objective, this invention adopts the following technical solution: A hyperspectral method for identifying the origin of Rehmannia glutinosa based on tabular foundation models is designed, comprising the following steps:

[0008] S1. Samples of Rehmannia glutinosa from different origins were collected in accordance with the relevant standards of the Pharmacopoeia of the People's Republic of China, and hyperspectral data of the Rehmannia glutinosa samples were acquired using a hyperspectral imaging system.

[0009] S2, Use RAD correction software to perform black and white correction on the acquired hyperspectral data to obtain corrected hyperspectral data;

[0010] S3. Use ENVI software to extract the region of interest (ROI) from the corrected hyperspectral data, calculate the average spectral reflectance of the ROI, and establish a one-dimensional spectral dataset of Rehmannia glutinosa samples.

[0011] S4, preprocesses the one-dimensional spectral data in the Rehmannia glutinosa spectral dataset;

[0012] S5, construct a high-dimensional structured matrix, and concatenate the numerical features of the absolute intensity, relative intensity, first derivative and second derivative of the spectrum across the entire spectrum band to directly reconstruct a high-dimensional structured matrix containing multi-dimensional spectral features, which is compatible with the input format of the basic table model.

[0013] S6 constructs a table-based model with a few-shot reasoning mechanism based on global context input, performs end-to-end classification reasoning, and outputs the final identification result of Rehmannia glutinosa.

[0014] Compared with the prior art, the present invention has the following beneficial effects:

[0015] Multidimensional representation fusion reduces information loss during dimensionality reduction: This invention introduces a multidimensional collaborative mapping mechanism for all features, horizontally fusing the physical representations of four core dimensions: absolute intensity, relative intensity, first derivative, and second derivative, and reshaping them into a high-dimensional structured matrix adapted to the table-based model. This mechanism enables multidimensional and refined capture of the absorption peaks of weak effective chemical components in Rehmannia glutinosa, improving the upper limit of feature expression and information completeness of the model at the front end.

[0016] End-to-end inference is achieved without massive data training and hyperparameter tuning: It eliminates the need for expending enormous computing power and time to train conventional deep neural networks from scratch. Relying solely on the meta-learning and in-context learning characteristics of the table-based model, a high-dimensional decision boundary can be quickly established by inputting only a small set of reference samples, achieving efficient intelligent identification with zero gradient updates. Attached Figure Description

[0017] The accompanying drawings are part of this invention and are primarily intended to facilitate understanding and explanation of the invention. They should not be construed as undue limitations on the invention.

[0018] Figure 1 This is a flowchart of the method described in this invention. Detailed Implementation

[0019] The specific embodiments of the present invention will be described below with reference to the accompanying drawings and examples. However, the following examples are only used to illustrate the present invention in detail and do not limit the scope of the present invention in any way.

[0020] This embodiment uses TabPFN 2.5 (Tabular Prior-Data Fitted Network v2.5, an advanced tabular basic model based on the Transformer architecture) as a specific instantiation object of the tabular basic model. Relying on a Linux high-performance computing platform equipped with NVIDIA A100 GPU and other accelerated computing hardware, it demonstrates in detail a hyperspectral method for identifying the origin of Rehmannia glutinosa. The specific steps are as follows:

[0021] S1, a total of samples of Rehmannia glutinosa slices were selected from different production areas (such as the authentic "Four Great Huai Medicines" production areas of Wenxian, Wuzhi, and Mengzhou in Henan Province, as well as non-authentic main production areas such as Shandong and Shanxi). The Rehmannia glutinosa sample was placed on the stage of the hyperspectral imaging system, and the intensity of the halogen light source and the camera exposure time were set. In this embodiment, the hyperspectral camera acquired wavelengths ranging from 400 nm to 1000 nm, with continuous sampling at the spectral resolution, acquiring a total of [number missing] samples. The original hyperspectral image data of the Rehmannia glutinosa sample was obtained by using 10 effective spectral bands.

[0022] S2. Due to the unavoidable physical noise from instrument dark current and uneven light source distribution during data acquisition, the hyperspectral camera must undergo hardware-level calibration. The instrument's built-in RAD calibration software is used to reduce noise in the original hyperspectral image. Subsequently, black-and-white correction calculations are performed. A completely black reference image (calibrated reflectance of 0) is acquired by covering the camera lens, and a completely white reference image (calibrated reflectance of 100%) is acquired using a standard Teflon white board. The relative reflectance image after eliminating environmental interference is obtained using the following formula:

[0023]

[0024] Where I O Indicates the spectral image before correction, I C I represents the corrected spectral image. D Represents a black reference image, I W This represents a reference image for the whiteboard.

[0025] S3. Using ENVI software (or an automated masking algorithm), manually or automatically select the Region of Interest (ROI) on the surface of each Rehmannia glutinosa sample in the corrected hyperspectral data to avoid edge and background interference. Calculate and export the average spectral reflectance of all effective pixels within each sample's ROI in each band. This step spatially reduces the dimensionality of the three-dimensional hyperspectral data block (containing spatial coordinates and spectral wavelengths), effectively eliminating local physical scattering errors and establishing a one-dimensional high signal-to-noise ratio Rehmannia glutinosa spectral dataset.

[0026] S4. To ensure that the table-based model acquires high signal-to-noise ratio and high-fidelity multidimensional refined features, the following preprocessing is performed on the Rehmannia glutinosa spectral data:

[0027] 1. Outlier Removal: First, calculate the Mahalanobis distance of the spectral samples in the feature space and set a tolerance threshold to automatically identify and remove outliers in the spectral dataset caused by physical sampling errors or extremely uneven slices, so as to construct a clean spectral dataset.

[0028] 2. Smoothing and Feature Extraction: Next, a Savitzky-Golay (SG) smoothing filter is used to remove high-frequency noise from the clean one-dimensional Rehmannia glutinosa spectral data (e.g., setting the sliding window length to 21 and the polynomial order to 2), effectively filtering out high-frequency electrical noise from the instrument without altering the overall spectral profile. Extremum normalization mapping is then applied to the filtered data to eliminate absolute scattering interference caused by sample thickness, and simultaneously, the relative intensity characteristics reflecting the proportion of a single band in the global spectrum are calculated.

[0029] 3. Morphological derivative calculation: Finally, the Savitzky-Golay smoothing filter is used again to perform polynomial fitting and differentiation on the normalized data to extract the first derivative morphological features that reflect the microscopic absorption rate of the spectrum and represent the absorption intensity of key chemical components, as well as the second derivative morphological features that can effectively eliminate baseline drift and highlight the curvature features of overlapping absorption peaks.

[0030] S5. To achieve deep adaptation between multidimensional continuous spectral features and the high-dimensional matrix input format of the tabular basic model, the continuous waveform features need to be tiled and reconstructed. The specific implementation process is as follows:

[0031] 1. Feature Extraction and Vector Tiling: Horizontally traverse all effective bands acquired by the hyperspectral imaging system (e.g., in this embodiment, the number of effective bands is set). For each Rehmannia glutinosa sample, the extreme value normalized absolute spectral intensity (224 dimensions), relative intensity reflecting global proportion (224 dimensions), first derivative (224 dimensions), and second derivative (224 dimensions) are precisely extracted in wavelength order. These four dimensions of feature values ​​are seamlessly concatenated to reshape a high-information-density structured feature vector. For a single sample, the mathematical structure of its reshaped and tiled feature vector V is rigorously defined as follows:

[0032]

[0033] in: to This represents the absolute intensity characteristics of the spectra across M bands; to This represents the relative intensity characteristics of the M bands; to Indicates the morphological characteristics of the first derivative of M bands; to This represents the morphological characteristics of the second derivative of M bands.

[0034] 2. Physical meaning of multidimensional features: In this embodiment, the feature vector dimension of a single Rehmannia glutinosa sample reaches... This splicing and reconstruction method not only preserves the macroscopic reflectivity profiles of each band through absolute and relative intensity, but also accurately depicts the microscopic absorption fingerprint peaks and curvature inversions of core chemical active ingredients such as catalpol and rehmannia glycosides at specific wavelengths through morphological derivative characteristics.

[0035] 3. Matrix Construction and Format Alignment: This involves preprocessing all clean samples remaining in the dataset (e.g., those after removing outliers). The feature vectors V of each sample are stacked vertically in rows. Simultaneously, the corresponding origin label column (Label) of each sample is appended after the feature columns of the matrix, ultimately constructing a matrix of shape […]. (Right now The high-dimensional structured matrix X is used to successfully transform traditional continuous spectral maps into discrete tabular forms, thus meeting the high-dimensional matrix input requirements of tabular-based model classifiers for tabular data.

[0036] S6. To achieve intelligent and rapid identification with low sample dependency and gradient-free training, this step constructs a few-shot inference mechanism for a table-based model based on global context input. Taking the pre-trained large table-based model TabPFN 2.5 based on the Transformer architecture as an example, its specific implementation process is as follows:

[0037] First, a global partitioning strategy is adopted, randomly dividing the high-dimensional structured matrix constructed in step S5 into a reference sample set and a test sample set containing known Huai Rehmannia origin labels at a ratio of 80% and 20%. Next, the pre-trained TabPFN 2.5 table-based model is loaded into the computing platform, and the network weight parameters of the model are completely frozen throughout the implementation process, without any gradient backpropagation or iterative training. Subsequently, the high-dimensional features and origin labels of the segmented reference sample set are used as global context and input into the table-based model at once. The model uses the built-in attention mechanism to automatically resolve the implicit correlation between bands and directly establish the classification decision boundary in the high-dimensional feature space without updating the physical weights. Finally, in the classification inference stage, the high-dimensional structured matrix of the test sample set is input into the model. Relying on the model's excellent few-shot context learning characteristics, the model directly performs conditional probability inference on the test samples and selects the category corresponding to the highest probability, thereby outputting the final Huai Rehmannia origin identification result.

[0038] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. The table-based model used in the present invention includes, but is not limited to, a table pre-trained model based on the Transformer architecture or other structured basic large-scale models with few-shot context learning capabilities. Without departing from the concept of the present invention, various specific parameters in the above embodiments can be changed, or relevant structural parts can be equivalently substituted, thereby forming multiple specific embodiments. These are all common variations within the scope of the present invention and will not be described in detail here.

Claims

1. A hyperspectral method for identifying the origin of Rehmannia glutinosa based on a tabular model, characterized in that, Includes the following steps: S1. Samples of Rehmannia glutinosa from different origins were collected in accordance with the relevant standards of the Pharmacopoeia of the People's Republic of China, and hyperspectral data of the Rehmannia glutinosa samples were acquired using a hyperspectral imaging system. S2, Use RAD correction software to perform black and white correction on the acquired hyperspectral data to obtain corrected hyperspectral data; S3. Use ENVI software to extract the region of interest (ROI) from the corrected hyperspectral data, calculate the average spectral reflectance of the ROI, and establish a one-dimensional spectral dataset of Rehmannia glutinosa samples. S4, preprocesses the one-dimensional spectral data in the Rehmannia glutinosa spectral dataset; S5, construct a high-dimensional structured matrix, and concatenate the numerical features of the absolute intensity, relative intensity, first derivative and second derivative of the spectrum across the entire spectrum band to directly reconstruct a high-dimensional structured matrix containing multi-dimensional spectral features, which is compatible with the input format of the basic table model. S6 constructs a table-based model with a few-shot reasoning mechanism based on global context input, performs end-to-end classification reasoning, and outputs the final identification result of Rehmannia glutinosa.

2. The method for identifying the origin of Rehmannia glutinosa based on the present invention relates to the interdisciplinary fields of non-destructive intelligent detection, spectral analysis, and machine learning of agricultural products and traditional Chinese medicinal materials, specifically, to a hyperspectral method for identifying the origin of Rehmannia glutinosa based on Tabular Foundation Models and In-Context Learning. The hyperspectral method for identifying the origin of Rehmannia glutinosa based on this model is characterized by: Samples of Rehmannia glutinosa from different origins were collected and placed in a hyperspectral imaging system for spectral scanning, acquiring raw hyperspectral image data of the Rehmannia glutinosa samples within a specific wavelength range.

3. The method for identifying the origin of Rehmannia glutinosa based on a tabular model according to claim 1, characterized in that, In step S2: The instrument's built-in RAD correction software is used to perform black and white correction on the original hyperspectral image to eliminate interference from instrument dark current and uneven light source distribution. The black and white correction calculation formula is as follows: Among them I O Indicates the spectral image before correction, I C I represents the corrected spectral image. D Represents a black reference image, I W This represents a reference image for the whiteboard.

4. The method for identifying the origin of Rehmannia glutinosa based on a tabular model according to claim 1, characterized in that: In step S3: Use ENVI software or an automated masking algorithm to select the region of interest (ROI) of each Rehmannia glutinosa sample in the corrected hyperspectral data to avoid background interference, and calculate and export the average spectral reflectance of all effective pixels in the ROI of each sample, perform spatial dimensionality reduction, and establish a one-dimensional Rehmannia glutinosa spectral dataset.

5. The method for identifying the origin of Rehmannia glutinosa based on a tabular model according to claim 1, characterized in that, In step S4, the preprocessing specifically involves: First, calculating the Mahalanobis distance of the spectral samples in the feature space and setting a tolerance threshold to automatically identify and remove outliers in the spectral data to construct a clean spectral dataset; Second, using a Savitzky-Golay smoothing filter to filter out high-frequency noise from the clean spectral dataset, performing extreme value normalization mapping on the filtered data to eliminate absolute scattering interference caused by sample thickness, and simultaneously calculating the relative intensity features reflecting the proportion of a single band in the global spectrum; Finally, again using the Savitzky-Golay smoothing filter to perform polynomial fitting and differentiation on the normalized data to extract the first derivative morphological features reflecting the spectral micro-absorption rate and the second derivative morphological features to eliminate baseline drift.

6. The method for identifying the origin of Rehmannia glutinosa based on a tabular model according to claim 1, characterized in that, In step S5: the construction of the high-dimensional structured matrix specifically involves: horizontally traversing all bands acquired by the hyperspectral imaging system, and accurately extracting the features of each dimension calculated in step S4 for each single band. Subsequently, all the traversed bands and their corresponding feature values ​​are concatenated to reshape a high-information-density structured feature vector containing thousands of features, thereby adapting to the high-dimensional matrix input requirements of the table-based model classifier for tabular data.

7. The method for identifying the origin of Rehmannia glutinosa based on a tabular model according to claim 1, characterized in that, In step S6: The mechanism for constructing a table-based model for few-shot inference based on global context input is as follows: The reconstructed high-dimensional structured matrix is ​​proportionally divided into a context support set containing prior knowledge of the origin of Rehmannia glutinosa and a test sample set. The context support set is loaded into a pre-trained table-based model classifier with frozen parameters. The internal context parsing mechanism of the model is used to capture long-distance implicit correlations between high-dimensional bands to establish decision boundaries. In the end-to-end classification inference stage, the high-dimensional structured matrix of the test sample set is input into the table-based model classifier. Finally, the table-based model classifier is used to directly perform probabilistic inference in the high-dimensional feature space and output the final Rehmannia glutinosa origin identification result.

Images