Rheum officinale age identification method based on multi-modal atlas fusion

Abstract

The application discloses a kind of based on multimodal atlas fusion's Rhubarb age identification method, collect the same origin different age Rhubarb sample, using hyperspectral imaging equipment acquisition Rhubarb sample's hyperspectral data;Using ENVI software carries out black and white correction;Extract the ROI region of corrected hyperspectral data, calculate average spectral reflectance;Build one-dimensional attention mechanism convolutional neural network to obtain the spectral recognition model of Rhubarb age;Again build two-dimensional convolutional neural network to obtain the image recognition model of Rhubarb age on image dataset;Global average pooling is used for multimodal feature fusion training.The application has the advantages that the spectral information and image information of hyperspectral image are fused and trained, based on the end-to-end deep learning model, without manual feature extraction, reduces the energy consumption, effectively improves the utilization efficiency of Rhubarb of different growth years, and provides scientific data support for scientifically and reasonably arranging the planting age of Rhubarb.

Description

Technical Field

[0001] This invention relates to the field of identification of Chinese medicinal plants, and in particular to a method for identifying the age of rhubarb based on multimodal graph fusion. Background Technology

[0002] Rhubarb is the dried root and rhizome of *Rheum palmatum*, *Rheum tanguticum*, or *Rheum palmatum* var. *tanguticum*, all perennial tall herbaceous plants of the Polygonaceae family. Also known as General Rhubarb, Huangliang, etc., it is cold in nature and bitter in taste. It possesses purgative, heat-clearing, fire-purging, blood-cooling, detoxifying, blood-activating, menstruation-regulating, astringent, and diuretic properties, and is widely used clinically. Modern research shows that it mainly contains anthraquinones, phenols, tannins, amino acids, trace elements, and other chemical components. It exhibits significant clinical efficacy in inducing diarrhea, antibacterial activity, antiviral activity, immune regulation, hemostasis, anti-inflammation, diuresis, liver protection, and the treatment of intestinal obstruction, pancreatitis, nephritis, and acute respiratory distress syndrome.

[0003] Because rhubarb roots contain medicinal components, and studies have shown that the content of anthraquinone derivatives such as rhein, chrysophanol, emodin, and aloe-chrysophanol, as well as oxalic acid, accumulates with increasing growth years. Therefore, generally speaking, the longer rhubarb has grown, the higher its medicinal value. For example, rhubarb older than 10 years typically contains more than twice the medicinal components of rhubarb younger than 6 years. Furthermore, measurements of rhubarb content at different ages indicate that rhubarb aged 3 years or older is the preferred choice for medicinal use.

[0004] However, determining the age of rhubarb relies heavily on experience, creating opportunities for unscrupulous merchants to pass off inferior products as superior ones. Therefore, a rapid, effective, and non-destructive method for identifying the age of rhubarb is needed to standardize its selection, ensure its medicinal value, and improve the efficacy of traditional Chinese medicine. Currently, there are methods for classifying Chinese herbal medicines based on hyperspectral images. However, these methods primarily use traditional machine learning to extract spectral features to train the classifier. They only focus on spectral information and require significant effort to extract effective band information, often resulting in severe loss of hyperspectral information and limiting classification performance. In recent years, deep learning has flourished in various fields, providing more possibilities for building models for identifying the age of rhubarb. Summary of the Invention

[0005] The purpose of this invention is to provide a method for identifying the age of rhubarb based on multimodal spectrum fusion.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] The method for identifying the age of rhubarb based on multimodal spectral fusion described in this invention includes the following steps:

[0008] S1, collect rhubarb samples of different ages from the same producing area, and use a hyperspectral imaging device to collect hyperspectral data of the rhubarb samples;

[0009] S2, use ENVI software to perform black and white correction on the hyperspectral data to obtain corrected hyperspectral data;

[0010] S3, extract an ROI region of the corrected hyperspectral data, calculate the average spectral reflectivity of the ROI region, and establish a spectral data set of the rhubarb samples;

[0011] S4, build a one-dimensional attention mechanism convolutional neural network to learn the spectral data set to obtain a spectral recognition model of rhubarb age;

[0012] S5, perform PCA dimensionality reduction processing on the ROI region to establish an image data set of the rhubarb samples;

[0013] S6, build a two-dimensional convolutional neural network to learn the image data set to obtain an image recognition model of rhubarb age;

[0014] S7, use global average pooling to extract features of the spectral recognition model and the image recognition model, perform multi-modal feature fusion training, and obtain a multi-modal atlas fusion recognition model of rhubarb age;

[0015] S8, use the multi-modal atlas fusion recognition model to identify the age of rhubarb.

[0016] Further, the hyperspectral data is obtained by a HySpex series hyperspectral imaging spectrometer at a distance of 20-30 cm from the rhubarb sample; and the moving speed of the platform on which the rhubarb sample is placed is 1.5 mm / s; the wavelength range of the hyperspectral data is 948.7188-2512.9722 nm, the integration time is 4500 μs, the frame time is 46928 s, and the number of bands is 288.

[0017] Further, the black and white correction calculation formula is:

[0018]

[0019] where R is the relative reflectivity, I R is the original reflectivity, I B is the blackboard reflectivity, I W is the whiteboard reflectivity.

[0020] Further, in S3, the ROI region has a size of 64*64 pixels; and the average spectrum of the ROI region is 1*1*288 spectral data.

[0021] Further, S5 specifically includes:

[0022] S5.1, unfolding the 288-dimensional spectral channel of each pixel point in the corrected hyperspectral data into a 1*288 feature matrix A;

[0023] S5.2, averaging the elements in the feature matrix A by column, and subtracting the average value of the corresponding column in the feature matrix A from each element in the feature matrix A to obtain a feature matrix B;

[0024] S5.3, calculating the covariance of every two columns of elements in the feature matrix B to construct a covariance matrix C of the feature matrix;

[0025] S5.4, using the eigen equation of the covariance matrix C to calculate all eigenvalues of the covariance matrix corresponding to the eigenvectors;

[0026] S5.5, sorting all eigenvalues from large to small, selecting the first 20 eigenvalues; and arranging the eigenvectors corresponding to the 20 eigenvalues into a feature vector matrix D by column;

[0027] S5.6, projecting the corrected hyperspectral data onto the feature vector matrix D to obtain an image data set with a size of 64*64*20.

[0028] Further, in the step S4, the one-dimensional attention mechanism convolutional neural network comprises a channel attention mechanism module and a spectral attention mechanism module, and the number of network layers is greater than or equal to 5.

[0029] Further, in the step S6, the two-dimensional convolutional neural network is a DenseNet network.

[0030] The advantage of the present application is that the spectral information and image information of the hyperspectral image are fused and trained, compared with the traditional method, not only the spectral information is used, but also the image information is used, and the model based on deep learning is end-to-end, without manual feature extraction, reducing the effort, effectively improving the utilization efficiency of different growth years of rhubarb, and providing scientific data support for scientific and reasonable arrangement of rhubarb planting years, the test proves that the accuracy of rhubarb age identification of the present application can reach 98%, which is improved by 3.0% compared with a single model. BRIEF DESCRIPTION OF DRAWINGS

[0031] Figure 1 is the method flow chart of the present application.

[0032] Figure 2 is an example of a corrected rhubarb sample in the method of the present application.

[0033] Figure 3 is a multi-modal fusion learning model structure schematic diagram constructed in the method of the present application. DETAILED DESCRIPTION

[0034] The technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0035] like Figure 1 As shown, the method for identifying the age of rhubarb based on multimodal spectral fusion according to the present invention includes the following steps:

[0036] S1. Collect rhubarb samples from the same origin but of different ages, and use a hyperspectral imaging device to acquire hyperspectral data of the rhubarb samples.

[0037] Specifically, the rhubarb herb used in this embodiment is Tangut rhubarb, and numerous samples of dried Tangut rhubarb raw materials with 1-year, 3-year, and 5-year growth periods were collected. Then, images were taken using a HySpex series hyperspectral imaging spectrometer.

[0038] The HySpex series hyperspectral imaging spectrometer consists of two halogen tungsten lamps, a CCD detector, an N3124 SWIR lens, and a moving platform. During image capture, the distance between the hyperspectral imager lens and the rhubarb sample is 20-30 cm, the platform movement speed is 1.5 mm / s, the N3124 SWIR lens integration time is 3500 μs, and the frame time is 66482 s. The spectrometer's spectral range is 948.7188–2512.9722 nm, with a band spacing of 5.43 nm, totaling 288 bands. The spectral resolution is 6 nm.

[0039] S2, Use ENVI software to perform black and white correction on the hyperspectral data to obtain corrected hyperspectral data;

[0040] ENVI software performs black-and-white correction on the raw hyperspectral data, which can effectively eliminate the influence of air and environment on the raw hyperspectral image and obtain clear, undistorted corrected hyperspectral data.

[0041] The formula for calculating black and white correction is:

[0042]

[0043] in R Relative reflectance I R Original reflectivity I B Blackboard reflectivity, I W This represents the reflectivity of the whiteboard. For example... Figure 2The shown after-correction rhubarb sample example.

[0044] S3, extracting the ROI region of the correction hyperspectral data, calculating the average spectral reflectivity of the ROI region, and establishing the spectral data set of the rhubarb sample; wherein the size of the extracted ROI region is 64*64 pixels; after calculating the average spectrum of the ROI region, the spectral data of 1*1*288 will be obtained. At the same time, it is also necessary to use python to mark the extracted spectral data for the later computer learning.

[0045] S4, building a one-dimensional attention mechanism convolutional neural network to learn the spectral data set, and obtaining a spectral recognition model of the rhubarb age limit;

[0046] As Figure 3 shown, the one-dimensional attention mechanism convolutional neural network is to integrate the attention mechanism module into the one-dimensional convolutional neural network to improve the feature learning ability and classification performance of the one-dimensional convolutional neural network.

[0047] Among them, the one-dimensional convolutional neural network in the application includes an input layer, a convolutional layer, an activation function, a pooling layer, a fully connected layer and a Softmax layer. The input layer is used to receive the spectral data; the activation function is ReLU; the pooling layer adopts global average one-dimensional pooling; the convolutional layer is one-dimensional convolution which can reduce the amount of calculation and speed up the calculation.

[0048] The attention mechanism module includes but is not limited to a channel attention mechanism module and a spectral attention mechanism module; it is used to embed a convolutional neural network of different depths, to adaptively optimize feature mapping, and to accumulate this advantage in the entire convolutional neural network by increasing multiple layers of attention modules. In addition, the attention mechanism module can learn the relationship between the target output and the spectral data input, thereby enhancing the relevant features, suppressing the irrelevant features, and ultimately improving the classification effect of the convolutional neural network.

[0049] The channel attention mechanism module in the application aggregates the feature information of the feature maps output by each channel of the convolutional layer, reweights the feature maps of each channel, distinguishes the importance of each channel feature map through different weights, obtains the channels that need to be paid attention to by the convolutional neural network, suppresses the channels with smaller effects, and adaptively changes the weights of each channel feature map.

[0050] The spectral attention mechanism module aggregates the feature information of the feature maps output by each channel of the convolutional layer to locate the relevant spectral bands, thereby optimizing the feature response of the convolutional neural network, enhancing the effective spectral band features, ignoring the invalid features, improving the efficiency of the convolutional neural network, and speeding up the training of the convolutional neural network model.

[0051] Specifically, for the multi-channel input Global temporal information is compressed into one channel by global average pooling to generate channel statistics vector where the th element of is calculated as follows:

[0052]

[0053] Then CAM (Chinese interpretation: Channel Attention Mechanism) adopts a simple gating mechanism to fully capture the channel dependency and generate channel recalibration vector . The formula is as follows:

[0054]

[0055] where represents the ReLU activation function, and represents the convolution with channel number 1 and kernel size 1x1, is the Sigmoid function, which compresses the information into the range of [0, 1]. The value of represents the importance of the th channel. Channel recalibration vector is used to recalibrate the feature The formula is as follows:

[0056]

[0057] Finally, the obtained feature fully considers the guidance of global information and can effectively highlight more discriminative feature information. The idea of residual learning is used and residual connection is introduced to optimize while reacquiring the original information. The final output of CAM is: .

[0058] The input feature is where , represents the position of the th band, and SAM obtains the input through a 1x1 convolution layer The feature map of the time domain signal is as follows:

[0059]

[0060] 1x1 convolution operation can aggregate the features of all channels, and then the Sigmoid function is used to obtain the spectral weight vector :

[0061]

[0062] The spectral weight vector can indicate the importance of the waveband, and the recalibration feature map is obtained by multiplying the spectral weight vector and the feature map

[0063]

[0064] Before recalibration, the SAM (Chinese interpretation as spectral attention mechanism) uses a convolution layer to encode the feature information between spectral bands to prevent excessive focus on related bands. Similar to the CAM, a residual connection is introduced to prevent the reduction of feature response values in the SAM. Finally, the SAM output is .

[0065] The present application will establish all spectral data sets in the spectral data set of rhubarb samples according to the ratio of 7:3 to randomly divide into training set and test set. First, the data in the training set is input into the constructed one-dimensional attention mechanism convolutional neural network for learning, and after the preliminary model is obtained, the optimal spectral recognition model of rhubarb age limit is obtained after debugging and learning through the validation set.

[0066] S5, PCA dimensionality reduction processing is carried out on the ROI region, and the image data set of rhubarb samples is established; specifically comprising the following steps:

[0067] S5.1, the 288-dimensional spectral channel of each pixel point in the corrected hyperspectral data is unfolded into a 1*288 feature matrix A;

[0068] S5.2, the elements in the feature matrix A are averaged by column, and each element in the feature matrix A is subtracted by the average value of the corresponding column in the feature matrix A to obtain a feature matrix B;

[0069] S5.3, the covariance of every two columns of elements in the feature matrix B is calculated, and the covariance matrix C of the feature matrix is constructed; the calculation formula of the covariance matrix C of the feature matrix is as follows:

[0070]

[0071]

[0072] wherein, represents and direct covariance, , m represents the number of columns of the feature matrix, E represents the expectation matrix, and C represents the covariance matrix

[0073] S5.4, all eigenvalues of the covariance matrix corresponding to the eigenvector are solved by using the eigen equation of the covariance matrix C; the solving formula is as follows: ​

[0074]

[0075] Where C is the covariance matrix, Let I be the identity matrix, representing the eigenvalues ​​obtained from the solution. 11 …a mm is an element in the covariance matrix C.

[0076] S5.5 Sort all eigenvalues ​​from largest to smallest and select the first 20 eigenvalues; arrange the eigenvectors corresponding to the 20 eigenvalues ​​into an eigenvector matrix D by columns;

[0077] S5.6 Project the corrected hyperspectral data onto the feature vector matrix D to obtain an image dataset of size 64*64*20.

[0078] S6, construct a two-dimensional convolutional neural network to learn from the image dataset and obtain an image recognition model for the age of rhubarb; the two-dimensional convolutional neural network is a DenseNet network, including but not limited to DenseNet121, DenseNet169, DenseNet201, and DenseNet264.

[0079] like Figure 3 As shown, this embodiment uses the classic DenseNet121 model as the image feature extraction network. Compared with traditional CNN (Convolutional Neural Network), it has two differences: (1) It establishes dense connections between all preceding layers and subsequent layers, and achieves feature reuse through the connection of features on the channel, thus achieving better performance with fewer parameters and computational costs. (2) It uses batch normalization layers, which solves the gradient vanishing or exploding gradient and degradation problems of traditional convolutional networks.

[0080] The network structure of DenseNet121 mainly consists of DenseBlock and Transition. In the DenseBlock, the feature map sizes of each layer are consistent, and can be connected in the channel dimension. The non-linear combination function in the DenseBlock adopts the structure of BN+ReLU+3x3 Conv. The DenseBlock layer numbers are set to 6, 12, 24 and 16 in sequence. For the Transition layer, it mainly connects two adjacent DenseBlocks and reduces the feature map size. The Transition layer includes a 1x1 convolution and a 2x2 AvgPooling, and the structure is BN+ReLU+1x1Conv+2x2 AvgPooling.

[0081] S7, extracting features of the spectrum recognition model and the image recognition model by using global average pooling, performing multi-modal feature fusion training, and obtaining a multi-modal spectrum fusion recognition model for the rhubarb age;

[0082] S8, identifying the age of the rhubarb by using the multi-modal spectrum fusion recognition model.

[0083] Through experiment verification, the model accuracy for identifying the age of the rhubarb can reach 98.10%. Compared with single, it has great improvement.

Claims

1. A method for identifying the age of rhubarb based on multimodal spectrum fusion, characterized in that: Includes the following steps: S1. Collect rhubarb samples from the same origin but of different ages, and acquire hyperspectral data of the rhubarb samples using a hyperspectral imaging device; the hyperspectral data is acquired by a HySpex series hyperspectral imaging spectrometer at a distance of 20-30 cm from the rhubarb samples; the platform on which the rhubarb samples are placed moves at a speed of 1.5 mm / s; the wavelength range of the hyperspectral data is 948.7188-2512.9722 nm, the integration time is 4500 μs, the frame time is 46928 s, and the number of bands is 288; S2, Use ENVI software to perform black and white correction on the hyperspectral data to obtain corrected hyperspectral data; S3, extract the ROI region of the corrected hyperspectral data, calculate the average spectral reflectance of the ROI region, and establish a spectral dataset of rhubarb samples; S4. Construct a one-dimensional attention mechanism convolutional neural network to learn the spectral dataset and obtain a spectral recognition model for the age of rhubarb. S5, perform PCA dimensionality reduction on the ROI region to establish an image dataset of rhubarb samples; specifically including: S5.1, expand the 288-dimensional spectral channels of each pixel in the corrected hyperspectral data into a 1*288 feature matrix A; S5.2, calculate the average value of each column of the elements in the feature matrix A, and subtract the average value of the corresponding column of the feature matrix A from each element of the feature matrix A to obtain the feature matrix B; S5.3, calculate the covariance of every two columns of the characteristic matrix B, and construct the covariance matrix C of the characteristic matrix; S5.4, Using the characteristic equation of the covariance matrix C, find the eigenvalues ​​of all covariance matrices that correspond one-to-one with the eigenvectors; S5.5 Sort all eigenvalues ​​from largest to smallest and select the first 20 eigenvalues; arrange the eigenvectors corresponding to the 20 eigenvalues ​​into an eigenvector matrix D by columns; S5.6 Project the corrected hyperspectral data onto the feature vector matrix D to obtain an image dataset of size 64*64*20; S6, a two-dimensional convolutional neural network is constructed to learn from the image dataset to obtain an image recognition model for the age of rhubarb; the one-dimensional attention mechanism convolutional neural network includes a channel attention mechanism module and a spectral attention mechanism module, and the number of network layers is greater than or equal to 5; S7. Use global average pooling to extract features from the spectral recognition model and the image recognition model, perform multimodal feature fusion training, and obtain a multimodal spectral fusion recognition model for rhubarb age. S8. The age of rhubarb is identified using the multimodal graph fusion recognition model.

2. The method for identifying the age of rhubarb based on multimodal spectrum fusion according to claim 1, characterized in that: The formula for calculating black and white correction is: Where R is the relative reflectance, and I R I represents the original reflectivity. B I is the blackboard reflectivity. W The reflectance is the whiteboard's reflectivity.

3. The method for identifying the age of rhubarb based on multimodal spectrum fusion according to claim 1, characterized in that: In step S3, the size of the ROI region is 64*64 pixels; the average spectrum of the ROI region is 1*1*288 spectral data.

4. The method for identifying the age of rhubarb based on multimodal spectrum fusion according to claim 1, characterized in that: In step S6, the two-dimensional convolutional neural network is the DenseNet network.

Images