A traditional Chinese medicinal material recognition method based on terahertz spectrum multi-modal feature enhancement
By constructing a three-branch neural network model and using conditional generative adversarial networks for data augmentation, and combining it with an attention mechanism for feature fusion, the problem of insufficient utilization of multimodal features in the identification of Chinese medicinal materials is solved, and the identification of subtle differences in Chinese medicinal materials with similar chemical compositions and the robustness of the model are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- JIAXING UNIV
- Filing Date
- 2026-04-17
- Publication Date
- 2026-07-14
AI Technical Summary
Existing technologies struggle to reliably and accurately identify Chinese medicinal herbs with similar chemical compositions and molecular structures. Terahertz signal acquisition is costly and has a limited number of samples, leading to overfitting of deep learning models and a decline in recognition capabilities. In multimodal recognition tasks, the consistency between generated samples and real samples is insufficient, and the feature fusion method is not adaptive enough, affecting the stability and accuracy of recognition results.
A three-branch neural network model is constructed to process time-domain, frequency-domain, and absorbance features respectively. A conditional generative adversarial network is used for data augmentation. The model is combined with discriminator confidence screening and FID index verification. Feature fusion is performed through an attention mechanism, and the modal feature weights are adaptively adjusted to achieve comprehensive utilization and robust recognition of multimodal features.
It enables the identification of subtle differences in Chinese medicinal materials with similar chemical compositions or structures, improving the model's generalization ability and recognition accuracy, and is applicable to Chinese medicinal material identification and other terahertz application scenarios.
Smart Images
Figure CN122385533A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of terahertz spectroscopy detection technology, specifically to a method for identifying Chinese medicinal materials based on the enhancement of multimodal features of terahertz spectroscopy. Background Technology
[0002] Terahertz (THz) radiation lies between the microwave and infrared regions of the electromagnetic spectrum, corresponding to wavelengths of 3 μm to 0.3 mm, typically covering a frequency range of 0.1–10 THz. It possesses both electromagnetic and optical properties and advantages, enabling it to penetrate many non-metallic materials that are opaque to infrared and visible light. Since the vibrational and rotational frequencies of many biological macromolecules are in the THz band, terahertz waves can be used to obtain rich biological and material information. Terahertz detection offers advantages such as non-destructive operation, low photon energy, and minimal sample damage, and has shown promising application prospects in fields such as biology, medicine, agriculture, and materials testing.
[0003] In the identification of traditional Chinese medicinal materials, different types of medicinal materials often have similar chemical compositions and molecular structures, resulting in subtle differences in their terahertz responses. Furthermore, terahertz signals are easily affected by sample preparation conditions, environmental factors, and system noise during acquisition, further weakening the already limited differences between samples. Under these circumstances, relying solely on time-domain waveforms, frequency-domain spectral peaks, or traditional peak analysis and principal component analysis methods is insufficient to stably and accurately represent the subtle differences between different traditional Chinese medicinal materials.
[0004] With the development of deep learning, research has begun to explore using structures such as convolutional neural networks and recurrent neural networks to process terahertz spectra, replacing manual feature engineering. Currently, common methods primarily focus on single-modal features, such as training using only time-domain signals or frequency-domain information. While these methods can improve classification performance to some extent, they have limited utilization of the multimodal characteristics of terahertz signals, resulting in insufficient feature coverage and a tendency to under-discriminate when dealing with fine-grained differences.
[0005] Furthermore, terahertz data acquisition is relatively expensive and has a long experimental cycle, resulting in a limited number of available Chinese medicinal herb samples. With insufficient samples, deep learning models are prone to over-reliance on existing samples during training, leading to overfitting and a decline in the model's recognition ability on unknown samples. To address this issue, existing technologies employ generative adversarial networks (GANs) to expand the sample pool by generating new training samples. However, most existing solutions only build generative models for single-type features, focusing primarily on approximation within a single feature space. For multimodal recognition tasks, the consistency between generated and real samples across different feature representations remains insufficient, and differences in generation quality and stability also exist, making it difficult to directly meet the training needs of multimodal recognition models.
[0006] Existing multi-branch or multi-modal recognition methods typically employ direct splicing, fixed-weight addition, or simple combination in the feature fusion stage. While these fusion methods can synthesize different feature information to some extent, they often struggle to dynamically adjust the importance of each branch feature based on the specific circumstances of different samples. When the signal is subject to noise interference, local features are not prominent, or the information of a certain modality is insufficiently expressed, simple fusion methods fail to fully reflect the complementary effects between different modalities, thus affecting the stability and accuracy of the final recognition results. Therefore, for the task of identifying traditional Chinese medicine using terahertz spectroscopy, it is still necessary to provide a recognition method that can simultaneously handle multi-modal feature extraction, sample enhancement, and adaptive fusion to improve the recognition performance of complex traditional Chinese medicine samples. Summary of the Invention
[0007] The purpose of this invention is to provide a method for identifying Chinese medicinal materials based on terahertz spectral multimodal feature enhancement, aiming to solve the above-mentioned problems.
[0008] To address the aforementioned technical problems, this invention provides a method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement, characterized by the following steps:
[0009] 1. Data preprocessing: Obtain the original terahertz time-domain signal of the Chinese medicinal materials, and preprocess the original time-domain signal; divide the preprocessed original time-domain signal into three paths: the first path is directly used as the time-domain input signal, the second path is converted into the frequency domain amplitude spectrum through Fast Fourier Transform (FFT) as the frequency domain input signal, and the third path is used as the absorbance input signal after calculating the absorbance characteristics and standardizing the signal.
[0010] The formula for calculating absorbance characteristics is: Where A(ν) represents the absorbance at frequency ν, I(ν) is the intensity of the terahertz signal after the sample is transmitted, and I0(ν) is the reference signal.
[0011] 2. Data Augmentation: A conditional generative adversarial network (GAN) is used to generate synthetic samples. The generator takes random noise and class labels as input and generates synthetic features through a fully connected network. The discriminator receives real sample features and class labels and outputs the probability of sample authenticity. A discriminator confidence screening strategy is used to retain only synthetic samples whose confidence scores in all three modality branches are higher than a set threshold. Simultaneously, the Fréchet distance (FID) metric is used to verify the distribution difference between synthetic and real samples. The FID formula is:
[0012] μr and μg represent the feature means of the real sample and the generated sample, respectively, and Σr and Σg represent their covariance matrices. The first term reflects the difference in means, and the second term reflects the difference in covariance structure. When the FID is small, it indicates that the generated sample is closer to the real sample in terms of overall distribution, indicating higher data augmentation quality. The conditional generative adversarial network projects the three types of features into a low-dimensional space, and then trains independent generators and discriminators for each, making the generation process closer to the features of each modality.
[0013] 3. Three-branch feature extraction: A three-branch neural network model is constructed, comprising a time-domain feature extraction branch, a frequency-domain feature extraction branch, and an absorbance feature extraction branch. These three branches process the corresponding input signals in parallel.
[0014] The temporal feature extraction branch adopts a CNN-LSTM combined structure, extracting temporal dynamic features through multiple layers of one-dimensional convolution, batch normalization, pooling operations, Dropout layers, LSTM layers, and fully connected layers, outputting a 64-dimensional temporal feature vector. Among them, the first convolutional layer uses a large convolutional kernel to capture the overall shape of the pulse and the position of the main peaks, while the second convolutional layer uses a smaller convolutional kernel to identify subtle local changes in the waveform. Global average pooling (GAP) is used after the convolutional layers instead of traditional flattening operations to avoid stacking too many parameters on the high-dimensional convolutional output.
[0015] The frequency domain feature extraction branch selects the main components of the frequency domain amplitude spectrum according to the energy accumulation ratio and removes the high-frequency part. It adopts a multi-layer fully connected network, and after standardization, ReLU activation, batch normalization and Dropout operation, it extracts the spectral structure information and outputs a 64-dimensional frequency domain feature vector. The multi-layer fully connected network includes three hidden units with layer sizes of 256, 128 and 64 respectively.
[0016] The absorbance feature extraction branch extracts component correlation features through principal component analysis (PCA) dimensionality reduction and a fully connected layer, and outputs a 64-dimensional absorbance feature vector.
[0017] 4. Feature Fusion: The feature vectors output from the three branches are adaptively weighted using an attention mechanism. The feature vectors from each branch learn attention scores through a fully connected layer, and are then normalized using the Softmax function to obtain the weight coefficients for each branch. These weighted coefficients are then summed to form a comprehensive feature vector. The feature fusion formula is as follows: F1, F2, and F3 represent the feature vectors extracted from the time-domain branch, frequency-domain branch, and absorbance branch, respectively. i The attention scores for each branch feature are learned through a fully connected layer. The attention weights, normalized by the Softmax function, reflect the importance of each branch feature in the final classification task.
[0018] 5. Classification Output: The comprehensive feature vector is input into a two-layer fully connected network for feature mapping, and the classification probability of Chinese medicinal materials is output through the Softmax layer to complete the identification of Chinese medicinal materials; the three-branch neural network model adopts an end-to-end training method and optimizes the network parameters through the backpropagation algorithm.
[0019] The medicinal materials mentioned include five categories, such as Angelica sinensis, Codonopsis pilosula, and American ginseng, which have similar chemical compositions or structures.
[0020] In summary, due to the adoption of the above-mentioned technologies, the beneficial effects of this invention are:
[0021] 1. This invention constructs a three-branch neural network model to process three types of features—time domain, frequency domain, and absorbance—in parallel, realizing the comprehensive utilization of the multimodal features of terahertz signals. The time domain branch captures time-series dynamic features, the frequency domain branch mines spectral structure information, and the absorbance branch supplements component correlation features. The three types of features complement each other, enabling a more comprehensive characterization of the physical and chemical features of the tested Chinese medicinal materials and effectively identifying subtle differences between Chinese medicinal materials with similar chemical compositions or structures.
[0022] 2. Data augmentation is performed using conditional generative adversarial networks. Independent generators and discriminators are trained for the three types of features respectively. Combined with the discriminator confidence screening strategy and FID index verification, the authenticity and consistency of the generated samples are ensured. This effectively solves the problems of limited number of terahertz samples and unbalanced class distribution, alleviates model overfitting, and improves the model's generalization ability.
[0023] 3. The feature fusion stage adopts an attention mechanism, which can automatically adjust the weight coefficients of different modal features according to the characteristics of the input samples, so that the model focuses on the feature information that is more critical to classification, and performs more robustly when there is noise or missing information in the multimodal signal, further improving the recognition accuracy.
[0024] 4. Each branch structure is specifically designed for the characteristics of terahertz signals. The time-domain branch adopts a CNN-LSTM combination structure to adapt to the temporal characteristics, while the frequency-domain branch and absorbance branch adopt network structures adapted to their own characteristics, which improves the effectiveness of feature extraction.
[0025] 5. The method of the present invention has good generalization ability. It is not only applicable to the identification of Chinese medicinal materials, but can also be extended to other terahertz application scenarios such as material identification and biological sample detection, and has good engineering application value. Attached Figure Description
[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention, making other features, objects, and advantages of the invention more apparent. The illustrative embodiments of the invention illustrated in the drawings and their descriptions are used to explain the invention and do not constitute an undue limitation of the invention. In the drawings:
[0027] Figure 1 This is a schematic diagram of the overall structure of the three-branch neural network model of the present invention;
[0028] Figure 2 This is a schematic diagram illustrating the principle of conditional generation of adversarial network data augmentation in this invention.
[0029] Figure 3 This is a waveform diagram of the terahertz absorbance of the present invention;
[0030] Figure 4 This is a schematic diagram of the attention mechanism feature fusion of the present invention. Detailed Implementation
[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention. Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the claimed invention, but merely to represent selected embodiments of the invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0032] In the description of this invention, it should be understood that the terms indicating orientation or positional relationship are based on the orientation or positional relationship shown in the drawings and are only for the convenience of describing the invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as a limitation of the invention.
[0033] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific context of the specification.
[0034] This invention provides a method for identifying Chinese medicinal materials based on terahertz spectral multimodal feature enhancement, the specific implementation steps of which are as follows:
[0035] Data preprocessing:
[0036] Terahertz spectroscopy detection equipment was used to collect the raw terahertz time-domain signals of five types of Chinese medicinal materials, and the raw time-domain signals were preprocessed by noise reduction, baseline correction and other operations.
[0037] The preprocessed original time-domain signal is divided into three paths: the first path is directly used as the time-domain input signal; the second path is converted into a frequency-domain amplitude spectrum through Fast Fourier Transform (FFT) and used as the frequency-domain input signal; the third path is calculated according to the absorbance formula. The absorbance characteristics were calculated and normalized by Z-score, which was then used as the absorbance input signal.
[0038] Data augmentation:
[0039] A conditional generative adversarial network is constructed to project three types of features—time domain, frequency domain, and absorbance—into a low-dimensional space, and an independent generator and discriminator are trained for each type of feature.
[0040] The generator takes random noise and Chinese medicinal herb category labels as input and generates corresponding synthetic features through a fully connected network; the discriminator receives real sample features and category labels and outputs the sample authenticity probability.
[0041] The discriminator confidence threshold is set to 0.8, and synthetic samples with confidence scores higher than 0.8 for all three modal branches are selected. The FID values of the selected synthetic samples and real samples are calculated, and synthetic samples with FID values less than the set threshold (set to 50 in this embodiment) are retained. These synthetic samples are combined with the original samples to form a balanced training dataset.
[0042] Three-branch feature extraction:
[0043] Temporal feature extraction branch: A CNN-LSTM combined structure is adopted. The first convolutional layer uses 16 kernels of size 16 with a stride of 1 and the activation function is ReLU, followed by batch normalization and max pooling (pooling kernel size 2, stride 2). The second convolutional layer uses 32 kernels of size 8 with a stride of 1 and the activation function is ReLU, followed by batch normalization and global average pooling. After passing through the Dropout layer (dropout rate=0.3), the input is fed into an LSTM layer with a hidden dimension of 64. The output of the LSTM layer is mapped to a 64-dimensional temporal feature vector through a fully connected layer.
[0044] Frequency domain feature extraction branch: Select the main components of the frequency domain amplitude spectrum according to the energy accumulation ratio (energy accumulation reaches 90% in this embodiment), and remove the high-frequency part; standardize the filtered frequency domain signal and input it into a multi-layer fully connected network. The first layer has 256 hidden units, the second layer has 128, and the third layer has 64. Each layer uses the ReLU activation function, and batch normalization and dropout layers (dropoutrate=0.2) are added between each layer. Finally, a 64-dimensional frequency domain feature vector is output.
[0045] Absorbance feature extraction branch: Principal component analysis (PCA) is performed on the standardized absorbance features to reduce dimensionality, retaining principal components with a cumulative contribution rate of 95%. The dimensionality-reduced features are input into a fully connected layer (64 hidden units and ReLU activation function) and output a 64-dimensional absorbance feature vector.
[0046] Feature fusion:
[0047] The 64-dimensional feature vectors output from the three branches are input into independent fully connected layers (32 hidden units and ReLU activation function) to learn the attention scores e1, e2, and e3 for each branch.
[0048] The attention score is normalized using the Softmax function to obtain the weight coefficients α1, α2, and α3.
[0049] The weighted sum is calculated according to the formula to obtain the comprehensive feature vector.
[0050] Classification output and model training:
[0051] The integrated feature vector is input into a two-layer fully connected network. The first layer has 128 hidden units and the activation function is ReLU, while the second layer has 64 hidden units and the activation function is ReLU.
[0052] The output of the fully connected network is processed by the Softmax layer to obtain the classification probabilities of five categories of Chinese medicinal materials;
[0053] The model uses the cross-entropy loss function, is trained end-to-end using the Adam optimizer, has a learning rate of 0.001, 100 training epochs, a batch size of 32, and optimizes network parameters using the backpropagation algorithm.
[0054] Chinese medicinal herb identification: The terahertz signal of the Chinese medicinal herb to be identified is processed according to steps 1-4 above, input into the trained model, output the classification probability, and select the category with the highest probability as the category of the Chinese medicinal herb to be identified to complete the identification.
[0055] In this embodiment, the above method is used to identify five types of Chinese medicinal materials with an accuracy rate of over 95%, and it has good anti-interference ability and generalization performance, effectively solving the shortcomings of the prior art.
Claims
1. A method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement, characterized in that: Includes the following steps: (1) Data preprocessing: Obtain the original terahertz time domain signal of Chinese medicinal materials and preprocess the original time domain signal; The preprocessed original time-domain signal is divided into three paths. The first path is directly used as the time-domain input signal. The second path is converted into the frequency domain amplitude spectrum through Fast Fourier Transform (FFT) and used as the frequency domain input signal. The third path calculates the absorbance characteristics and uses them as the absorbance input signal after standardization. (2) Data augmentation: A conditional generative adversarial network is used to generate synthetic samples. The generator takes random noise and class labels as input and generates synthetic features through a fully connected network. The discriminator receives real sample features and class labels and outputs the probability of sample authenticity. A discriminator confidence screening strategy is employed, retaining only synthetic samples whose confidence scores for all three modality branches are above a set threshold. Simultaneously, the Fréchet distance (FID) metric is used to verify the distributional difference between synthetic and real samples. The FID formula is as follows: Where μr and μg represent the feature mean of the real sample and the generated sample, respectively, and Σr and Σg represent their covariance matrix; when FID is small, the synthetic sample is retained, and the filtered synthetic sample and the original sample are combined to form the training dataset; (3) Three-branch feature extraction: A three-branch neural network model is constructed, which includes a time-domain feature extraction branch, a frequency-domain feature extraction branch, and an absorbance feature extraction branch. The three branches process the corresponding input signals in parallel: The temporal feature extraction branch adopts a CNN-LSTM combination structure, which extracts temporal dynamic features through multiple one-dimensional convolutions, batch normalization, pooling operations, Dropout layers, LSTM layers and fully connected layers, and outputs a 64-dimensional temporal feature vector. The frequency domain feature extraction branch selects the main components of the frequency domain amplitude spectrum according to the energy accumulation ratio and removes the high-frequency part. It adopts a multi-layer fully connected network, and after standardization, ReLU activation, batch normalization and Dropout operation, it extracts the spectral structure information and outputs a 64-dimensional frequency domain feature vector. The absorbance feature extraction branch extracts component correlation features through principal component analysis (PCA) dimensionality reduction and a fully connected layer, and outputs a 64-dimensional absorbance feature vector. (4) Feature Fusion: The feature vectors output by the three branches are adaptively weighted through an attention mechanism. The feature vectors of the three branches learn attention scores through a fully connected layer, and the weight coefficients of each branch are obtained by normalization using the Softmax function. The weighted sum is then used to form a comprehensive feature vector. The feature fusion formula is as follows: F1, F2, and F3 represent the feature vectors extracted from the time-domain branch, frequency-domain branch, and absorbance branch, respectively. i The attention scores for each branch feature are learned through a fully connected layer. The attention weights, normalized by the Softmax function, reflect the importance of each branch feature in the final classification task. (5) Classification output: The comprehensive feature vector is input into a two-layer fully connected network for feature mapping, and the classification probability of Chinese medicinal materials is output through the Softmax layer to complete the identification of Chinese medicinal materials.
2. The method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, The formula for calculating the absorbance characteristics in step 1 is as follows: Where A(ν) represents the absorbance at frequency ν, I(ν) is the intensity of the terahertz signal after the sample is transmitted, and I0(ν) is the reference signal.
3. The method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, In the temporal feature extraction branch, the first convolutional layer uses a larger convolutional kernel to capture the overall shape of the pulse and the position of the main peaks, while the second convolutional layer uses a smaller convolutional kernel to identify subtle local changes in the waveform. Global average pooling (GAP) is used after the convolutional layers instead of the traditional flattening operation.
4. The method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, The multilayer fully connected network of the frequency domain feature extraction branch includes three layers of hidden units, with layer sizes of 256, 128, and 64 respectively.
5. The method for identifying traditional Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, The three-branch neural network model is trained end-to-end, and the network parameters are optimized through the backpropagation algorithm.
6. The method for identifying Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, The medicinal materials mentioned include five categories, such as Angelica sinensis, Codonopsis pilosula, and American ginseng, which have similar chemical compositions or structures.
7. The method for identifying Chinese medicinal materials based on terahertz spectral multimodal feature enhancement according to claim 1, characterized in that, In step 2, corresponding conditional generative adversarial network training branches are constructed for time-domain features, frequency-domain features, and absorbance features, and each training branch generates enhanced samples with the same spatial dimension as the corresponding feature.