A dual-mode non-contact optical microscopic imaging device for traditional Chinese medicinal material structure and component detection
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2025-11-18
- Publication Date
- 2026-07-21
Smart Images
Figure CN121298615B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of optical microscopy, and in particular relates to a dual-modal non-contact optical microscopy device for detecting the structure and components of traditional Chinese medicine. Background Technology
[0002] Traditional Chinese medicine (TCM), as an important component of my country's traditional medicine, plays a crucial role in the prevention and treatment of common, prevalent, and intractable diseases. However, the quality of medicinal plants is complex: in the cultivation stage, some regions have not followed the standards for the construction of authentic medicinal plant production bases, neglecting factors such as soil and climate, leading to unstable medicinal properties and the loss of effective components; in the processing and transportation stages, improper storage and transportation may cause mold and expiration, and non-standard processing may result in excessive impurities, altering the medicinal properties. These problems not only weaken the efficacy of TCM but also endanger public health. The development of TCM urgently requires the improvement of multimodal information detection technology for medicinal materials to provide data support for subsequent quality identification.
[0003] Traditional and modern methods for detecting Chinese medicinal materials each have their limitations. Traditional methods, including source identification, morphological identification, and microscopic identification, each have their advantages: source identification can confirm the origin and part of the medicinal material; morphological identification relies on shape, color, and taste to quickly identify the quality of the material without the need for equipment; microscopic identification relies on optical microscopy to achieve standardized testing, but currently lacks intelligent analytical methods. Traditional methods mainly rely on the experience of the appraiser and lack objective evaluation standards. Modern methods for detecting Chinese medicinal materials mainly rely on physicochemical identification, consisting of chromatography, spectroscopy, and mass spectrometry. Characteristic spectral data are accurate, but the chemical composition analysis process is complex and time-consuming, suitable only for sampling inspections and not for large-scale quality control. Chromatography is a key tool for biochemical and pharmaceutical analysis; thin-layer chromatography is suitable for qualitative analysis but difficult for quantitative analysis; gas chromatography requires standard calibration and has requirements on sample molecular weight; liquid chromatography may be affected by extra-column effects. Spectroscopy is simple and rapid, requiring no complex calibration or special sample requirements, demonstrating unique technical advantages in the analysis of Chinese medicinal components. However, traditional ultraviolet-visible-infrared spectroscopy suffers from low resolution, significant temperature influence, limited ability to distinguish similar substances, and applicability limited by the structure of the material. Mass spectrometry for detecting organic matter also has limitations, typically requiring pretreatment of medicinal materials and often used in conjunction with other techniques. Current detection methods are mostly focused on specific analytical tasks. For example, mass spectrometry is commonly used to detect pesticide residues and heavy metal content, while spectroscopy and chromatography are used to analyze the active ingredients in medicinal materials. This fragmented approach not only increases the complexity and cost of the detection process but also reduces efficiency. Therefore, there is an urgent need to develop a highly efficient integrated detection technology that can combine structural feature detection and component analysis, enabling rapid, comprehensive, and non-contact detection of traditional Chinese medicine materials. This technology can simultaneously acquire multi-dimensional information related to the structure, chemical composition, and pesticide residues of medicinal materials, providing technical support for quality identification and origin classification in the traditional Chinese medicine industry.
[0004] Our team proposed a non-contact polarized photoacoustic detection device for the quality identification of Chinese medicinal materials in patent CN202510060944.6. In this scheme, the non-contact polarized photoacoustic detection device is used to obtain polarized photoacoustic images of Chinese medicinal materials and their anisotropic characteristics. However, this scheme only performs structural imaging on Chinese medicinal materials, such as obtaining the microstructure and internal anisotropic information of Chinese medicinal materials, but cannot obtain the component information of Chinese medicinal materials. Summary of the Invention
[0005] Based on the above background, in order to overcome the deficiencies in existing methods for detecting the structure and components of Chinese medicinal materials, this invention provides a (photoacoustic-fluorescence) dual-modal optical microscopy imaging device for detecting the structure and components of Chinese medicinal materials. Based on the characteristic that different types, origins, and processing techniques of Chinese medicinal materials result in different internal structural states and different contents of characteristic components, this invention combines all-light non-contact photoacoustic microscopy remote sensing imaging technology with fluorescence microscopy technology to achieve non-contact detection of the structure and components of Chinese medicinal materials. Simultaneously, intelligent algorithms are used for data processing and analysis to achieve rapid and non-destructive quality identification of Chinese medicinal materials.
[0006] This invention provides a dual-modal optical microscopy imaging device for detecting the structure and components of traditional Chinese medicinal materials, the device comprising: The excitation light module, including laser 1 and laser 2, is used to provide an excitation beam; The detection optical module is used to automatically adapt the optimal detection wavelength according to the selected excitation wavelength. The sample module is used to hold and position the sample to be tested; The fluorescence spectroscopy module is used to acquire the fluorescence spectral data of the sample to be tested according to the first optical path. The first optical path consists of a laser 1, a dichroic mirror 3, a dichroic mirror 2, a galvanometer, an objective lens, and a spectrometer. The photoacoustic imaging module is used to acquire the photoacoustic image of the sample under test according to the second optical path. The second optical path consists of a laser 2, a beam splitter, a photodetector 1, a collimator, a quarter-wave plate 1, a dichroic mirror 1, a dichroic mirror 2, a galvanometer, an objective lens, and a photodetector 2. The data processing module preprocesses the fluorescence spectral data obtained from each band, and then uses deep learning that integrates a lightweight residual network and a multi-head self-attention mechanism to process and fuse the acquired fluorescence spectral data and photoacoustic image data. Based on the position, shape and intensity of the spectral absorption peaks, the absorption peaks are assigned to the corresponding chemical components, establishing the correlation between spectral features and chemical components, as well as the microstructure and texture features of Chinese medicinal materials.
[0007] The Chinese medicinal materials information module summarizes the composition and structural characteristics of Chinese medicinal materials after processing fluorescence spectral data and photoacoustic imaging data from various bands. This includes multi-dimensional information on the spectral characteristics of chemical components, microstructure and texture attributes of Chinese medicinal materials, and constructs a unified multimodal feature database for Chinese medicinal materials, which is used for the identification of Chinese medicinal material categories, quality assessment and quantitative analysis of components.
[0008] This invention also proposes a dual-modal optical microscopy imaging method for detecting the structure and components of traditional Chinese medicine materials. This imaging method specifically includes: Step S1: Select a suitable excitation wavelength; Step S2: Select a suitable probe wavelength; Step S3: Obtain the fluorescence spectrum data of the sample in this band; Step S4: Obtain photoacoustic image data of the sample in this band; Step S5: Select another wavelength of excitation light and repeat steps S1-S4. Step S6: The obtained fluorescence spectrum data and photoacoustic image data are transmitted to the data processing module, and the fluorescence spectrum data and photoacoustic image data are processed separately. Step S7: Summarize the characteristic information of Chinese medicinal materials, such as components and structure, after processing the fluorescence spectral data and photoacoustic imaging data of each band. Step S8 involves integrating and applying multimodal data to ultimately achieve quality control of Chinese medicinal materials.
[0009] Compared with the prior art, the present invention has the following advantages: (1) The innovative aspect of the photoacoustic-fluorescence dual-modal optical microscopy imaging device proposed in this invention lies in the integration of fluorescence spectroscopy and photoacoustic microscopy remote sensing imaging technology. Its principle is simple and efficient, and theoretically applicable to the detection of multimodal information such as composition and structure of all Chinese medicinal materials. This invention provides an effective new strategy for realizing multimodal information detection of Chinese medicinal materials, which helps to break through the existing difficulties of complex detection steps, single detection information, and high time cost in the detection of Chinese medicinal materials, and promotes the application and development of the field of Chinese medicinal material detection.
[0010] (2) This invention proposes a photoacoustic-fluorescence dual-modal optical microscopy imaging device for the detection of structure and components of Chinese medicinal materials, which realizes rapid acquisition of fluorescence spectrum of Chinese medicinal materials and all-optical incoherent non-contact high-resolution photoacoustic imaging of Chinese medicinal materials, providing hardware support for subsequent large-scale detection and quality analysis of Chinese medicinal materials.
[0011] (3) This invention proposes a photoacoustic-fluorescence dual-modal optical microscopy imaging device for the detection of structure and components of Chinese medicinal materials, which realizes the acquisition of multimodal information such as components, texture and structure of a large number of Chinese medicinal materials, and provides data support for the large-scale application of subsequent detection of Chinese medicinal materials. Attached Figure Description
[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0013] Figure 1 This is a structural diagram of a dual-modal optical microscopic imaging device system for detecting the structure and components of traditional Chinese medicine materials proposed in this invention; Figure 2 This is a schematic diagram of the optical path of a dual-modal optical microscopic imaging device for detecting the structure and components of traditional Chinese medicine, as proposed in this invention. Figure 3 This is a schematic diagram of the data processing module in a traditional Chinese medicine detection device based on fluorescence spectroscopy and multi-wavelength photoacoustic microscopy. Figure 4 This is a flowchart of a method for detecting traditional Chinese medicinal materials based on fluorescence spectroscopy and multi-wavelength photoacoustic microscopy. Detailed Implementation
[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of the invention. However, those skilled in the art will understand that the invention can be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods are omitted so as not to obscure the description of the invention with unnecessary detail.
[0015] like Figure 1 As shown, the present invention provides a dual-modal optical microscopy imaging device for detecting the structure and components of traditional Chinese medicine materials, the device comprising: The excitation light module, including laser 1 and laser 2, is used to provide an excitation beam; The detection optical module is used to automatically adapt the optimal detection wavelength according to the selected excitation wavelength. The sample module is used to hold and position the sample to be tested; The fluorescence spectroscopy module is used to acquire the fluorescence spectral data of the sample to be tested according to the first optical path. The first optical path consists of a laser 1, a dichroic mirror 3, a dichroic mirror 2, a galvanometer, an objective lens, and a spectrometer. The photoacoustic imaging module is used to acquire the photoacoustic image of the sample under test according to the second optical path. The second optical path consists of a laser 2, a beam splitter, a photodetector 1, a collimator, a quarter-wave plate 1, a dichroic mirror 1, a dichroic mirror 2, a galvanometer, an objective lens, and a photodetector 2. The data processing module preprocesses the fluorescence spectral data obtained from each band, and then uses deep learning that integrates a lightweight residual network and a multi-head self-attention mechanism to process and fuse the acquired fluorescence spectral data and photoacoustic image data. Based on the position, shape and intensity of the spectral absorption peaks, the absorption peaks are assigned to the corresponding chemical components, establishing the correlation between spectral features and chemical components, as well as the microstructure and texture features of Chinese medicinal materials.
[0016] The Chinese medicinal materials information module summarizes the composition and structural characteristics of Chinese medicinal materials after processing fluorescence spectral data and photoacoustic imaging data from various bands. This includes multi-dimensional information on the spectral characteristics of chemical components, microstructure and texture attributes of Chinese medicinal materials, and constructs a unified multimodal feature database for Chinese medicinal materials, which is used for the identification of Chinese medicinal material categories, quality assessment and quantitative analysis of components.
[0017] Specifically, the excitation module consists of laser 1 and laser 2. Laser 1 is an ultraviolet tunable laser, whose emitted laser wavelength can be continuously adjusted within a certain range (200nm-450nm); laser 2 can output multiple discrete wavelengths, including 488nm, 532nm, and 1064nm lasers. Different Chinese medicinal materials exhibit specific fluorescence and photoacoustic responses under different wavelength excitations. For example, tanshinone, the main active ingredient in Salvia miltiorrhiza, emits a reddish-brown micro-fluorescence of about 550nm under 285nm excitation, while salvianolic acid emits a blue-green fluorescence of about 450nm under 320nm excitation; luteolin in Perilla frutescens emits a strong green fluorescence of 480nm under 370nm excitation. In addition, 532nm and 1064nm lasers can also be used as excitation sources for photoacoustic imaging.
[0018] Regarding the selection of excitation wavelengths, this device categorizes samples into two main types: known and unknown, based on the characteristic spectral peaks exhibited by medicinal herbs under specific wavelength excitation. For known medicinal herbs, the optimal excitation wavelength corresponding to their characteristic components can be directly selected for fluorescence excitation. For example, it is known that tanshinone, the main active ingredient in Salvia miltiorrhiza, emits a reddish-brown micro-fluorescence at approximately 550 nm under 285 nm excitation; therefore, 285 nm excitation light is used when detecting Salvia miltiorrhiza. For unknown samples, multiple wavelengths of excitation combined with spectral analysis methods are required to systematically acquire their fluorescence and photoacoustic characteristics, thereby analyzing the components of the medicinal herb. Finally, by fusing multispectral data with high-resolution photoacoustic images, accurate identification of medicinal herbs is achieved.
[0019] like Figure 2As shown, the probe light module provides the probe light source required for photoacoustic imaging. This module includes a laser capable of outputting wavelengths of 830nm and 1310nm. The module can automatically adapt the optimal probe wavelength according to the selected excitation wavelength: when the excitation wavelength is 532nm, the probe light is 1310nm; if the excitation wavelength is 1064nm, then the corresponding 830nm probe light is selected, thus ensuring high-quality detection and imaging of the photoacoustic signal.
[0020] Specifically, in the fluorescence spectroscopy module, the laser 1 of the excitation light module emits a laser beam, which enters the dichroic mirror 3. After passing through the dichroic mirror 3, the beam enters the dichroic mirror 2, then passes through the galvanometer and enters the objective lens, finally striking the sample stage to perform optical scanning on the sample. The fluorescence signal fed back by the sample, after passing through the original optical path, is separated at the dichroic mirror 3. The fed back fluorescence signal then enters the spectrometer after passing through the dichroic mirror 3. The spectrometer transmits the obtained sample fluorescence signal to the computer, ultimately obtaining the fluorescence spectral data of the sample in that wavelength band.
[0021] Specifically, in the photoacoustic imaging module, the laser 2 of the excitation light module emits a laser beam, which enters the beam splitter and is divided into two beams. One beam enters the photodetector 1 to provide a trigger signal for subsequent photoacoustic signal acquisition. The other beam enters the collimator and is then combined with the probe beam through the quarter-wave plate 1 and the dichroic mirror 1.
[0022] The laser of the detection optical module emits a corresponding laser beam. After passing through a half-wave plate, the beam enters a polarization beam splitter. Subsequently, the beam enters a quarter-wave plate 2 and is merged with the excitation beam by a dichroic mirror 1.
[0023] After the excitation beam and the probe beam are combined, the combined beam enters the dichroic mirror 2, then passes through the galvanometer and enters the objective lens, and finally hits the sample stage to perform optical scanning on the sample.
[0024] The photoacoustic signal fed back by the sample is split at the polarization beam splitter through the original optical path, and will not return to the probe laser, thus preventing damage to the laser. The photoacoustic signal enters photodetector 2, which transmits the photoacoustic signal to the computer, ultimately obtaining the photoacoustic image data of the sample in that wavelength band.
[0025] Specifically, in the data processing module, this invention employs a deep learning architecture that integrates a lightweight residual network and a multi-head self-attention mechanism, such as... Figure 3 As shown, the processing and feature fusion of the acquired fluorescence spectral data and photoacoustic image data can be mainly divided into three parts: fluorescence spectral feature extraction, photoacoustic image feature extraction, and multimodal feature fusion and multi-scale integration. The specific implementation method is as follows: In the fluorescence spectral feature extraction section, firstly, the acquired one-dimensional fluorescence spectral data sequence... It is mapped to a high-dimensional feature representation through an embedding layer. Where L is the sequence length of the fluorescence spectroscopy data, i.e., the number of data points, and D is the dimension after feature embedding. This embedding layer uses a learnable weight matrix. Implementation, in which This represents the original input dimension.
[0026] Subsequently, the feature sequence Input a multi-head self-attention module to mine global dependencies between positions in the spectral sequence. For the ... One attention point ( ), through a learnable weight matrix Calculate their query matrix respectively Key matrix Sum matrix :
[0027] The output of each head is calculated using a scaled dot product attention mechanism:
[0028] in, The feature dimension for each attention head is calculated using the following formula:
[0029] The outputs of all h heads are concatenated along the feature dimension and then projected through a learnable linear projection matrix. The data are then integrated to obtain the final spectral feature representation. :
[0030] in, It is a learnable linear projection weight matrix, not a fixed matrix, but a parameter that is continuously optimized and updated during neural network training through backpropagation and gradient descent. Its final value is learned by the model from the training data, aiming to optimally fuse multi-head information to complete the task of extracting spectral features of Chinese medicinal materials.
[0031] This mechanism enables the model to learn from different representation subspaces in parallel and dynamically identify and weight key spectral feature peaks associated with specific chemical components.
[0032] The photoacoustic image feature extraction section uses a two-dimensional photoacoustic microscopic image as input. , where H represents the height of the input image; W represents the width of the input image; 3 represents the number of channels of the input image, representing the three color channels: red, green and blue.
[0033] First, an initial feature encoding is performed on the two-dimensional photoacoustic microscope image using a standard two-dimensional convolutional layer (Conv2D). The Conv2D operation extracts local features and performs linear transformations by sliding a set of learnable filters (convolutional kernels) across the input image. The specific output of this operation is calculated by the following formula:
[0034] in, represents the learnable weight parameters (i.e., convolutional kernel) of the convolutional layer; ReLU is the linear rectified activation function, defined as follows: This function introduces nonlinearity into the network, enabling the model to learn and fit complex nonlinear mapping relationships; The feature map is obtained after initial convolution and nonlinear activation. .in, This is the height of the feature map after the initial convolution, and its specific value is determined by the kernel size, stride, and padding parameters. This represents the width of the feature map after the initial convolution, and its specific value is determined by the kernel size, stride, and padding parameters. The number of channels in the feature map after the initial convolution is a preset network hyperparameter.
[0035] Next, the initial feature map Deep feature extraction is performed using a network consisting of three cascaded lightweight residual modules. Each lightweight residual module contains two stages: depthwise separable convolution and channel attention mechanism, progressively mining cross-scale spatial features from microscopic cellular structure to macroscopic tissue arrangement. For generalization, we denote the input feature map of any module as [the following is a literal translation of the first part, which is not directly related to the previous sentence]. The structure and calculation process of each lightweight residual module are as follows.
[0036] The depthwise separable convolutional layer is responsible for spatial feature extraction. The first step is to process the input feature map. Perform channel-by-channel spatial filtering, without inter-channel mixing.
[0037]
[0038] in, The output is the learnable weights of the depthwise convolutional kernel. and The number of channels is the same.
[0039] The second step uses 1×1 convolution to fuse the components. All channel information.
[0040]
[0041] in, These are the learnable weights of point convolutions. Output These are the final convolutional features that are prepared to be input into the subsequent attention module.
[0042] The channel attention mechanism employs squeezing and incentive steps to calculate channel weights.
[0043] The squeezing step generates channel statistics through global average pooling. :
[0044] The incentive step involves learning the channel importance weights through a two-layer fully connected network with a bottleneck structure. :
[0045] in, , For learnable weights, It is the ReLU activation function. This is the Sigmoid function.
[0046] The weights obtained from the activation steps With feature map Multiplying channel by channel yields the calibrated features. .
[0047] Finally, the final outputs of the three lightweight residual modules are generated through residual connections to ensure effective gradient propagation:
[0048] The three modules are connected in a feedforward manner, and their output channels can be respectively The output of the third module This is the final output of the entire photoacoustic image feature extraction path, and is denoted as deep spatial features. This notation accurately describes As a structure of a three-dimensional real tensor, where, This means that all elements in the tensor are real numbers; This represents the final height (in pixels) of the feature map. Due to the stride of operations such as convolution and pooling in the network, this value may be smaller than the height of the original input image. ; This represents the final width (in pixels) of the feature map; again, this value may be smaller than the width of the original input image. ; This represents the final number of channels in the feature map.
[0049] In the multimodal feature fusion and multi-scale integration section, during the feature fusion stage, the system deeply fuses the spatial features extracted by the lightweight residual module with the spectral features mined by the multi-head self-attention mechanism at the channel dimension to form a unified multimodal representation. This composite feature is then further refined through a multi-layer convolutional backbone network to uncover the intrinsic correlations between different modal features. A specially introduced spatial pyramid pooling module, through multi-scale pooling operations, achieves adaptive integration of features at different resolutions, effectively balancing the contribution of each modal feature and improving the system's robustness to scale changes.
[0050] Specifically, this mainly involves the extracted spectral features. Image features To achieve deep integration.
[0051] First, feature alignment and fusion are performed: spectral features are... Mapped to channel dimensions that match image features through fully connected layers. ,get Along its sequence dimension The average is calculated to obtain the global spectral description vector. and spatially broadcast it to image features Same size ,get Subsequently, and The concatenation is performed along the channel dimension and then processed by a 1×1 convolution. Dimensionality reduction and integration were performed to obtain preliminary fusion features. .
[0052] Secondly, spatial pyramid pooling is performed: to enhance the model's robustness to changes in feature scale, a spatial pyramid pooling module is introduced. This module... Max pooling operations with kernel sizes of 1×1, 2×2, and 4×4 are performed in parallel. The resulting feature maps at each scale are then upsampled to a uniform size. Then, the data is stitched together along the channel dimension to form a multi-scale fused feature. .
[0053] Finally, through a 1×1 convolutional layer This feature is further refined to output a unified and robust multimodal feature representation. .
[0054]
[0055] A unified multimodal feature database of Chinese medicinal materials was constructed using multimodal features for the identification, quality assessment and quantitative analysis of components of Chinese medicinal materials.
[0056] like Figure 4 As shown, this invention also proposes a dual-modal optical microscopy imaging method for detecting the structure and components of traditional Chinese medicine materials. This imaging method specifically includes: Step S1: Select a suitable excitation wavelength; Step S2: Select a suitable probe wavelength; Step S3: Obtain the fluorescence spectrum data of the sample in this band; Step S4: Obtain photoacoustic image data of the sample in this band; Step S5: Select another wavelength of excitation light and repeat steps S1-S4. Step S6: The obtained fluorescence spectrum data and photoacoustic image data are transmitted to the data processing module, and the fluorescence spectrum data and photoacoustic image data are processed separately. Step S7: Summarize the characteristic information of Chinese medicinal materials, such as components and structure, after processing the fluorescence spectral data and photoacoustic imaging data of each band. Step S8 involves integrating and applying multimodal data to ultimately achieve quality control of Chinese medicinal materials.
[0057] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.
[0058] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.
[0059] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.
[0060] In the embodiments provided by this invention, it should be understood that the disclosed apparatus / terminal devices and methods can be implemented in other ways. For example, the apparatus / terminal device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0061] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.
[0062] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.
[0063] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.
Claims
1. A dual-modal optical microscopy imaging device for detecting the structure and components of traditional Chinese medicinal materials, characterized in that, The device includes: The excitation light module includes laser one and laser two, which are used to provide the excitation beam; The detection optical module is used to automatically adapt the optimal detection wavelength according to the selected excitation wavelength. The sample module is used to hold and position the sample to be tested; The fluorescence spectroscopy module is used to acquire the fluorescence spectral data of the sample to be tested according to the first optical path. The first optical path consists of a laser I, a dichroic mirror III, a dichroic mirror II, a galvanometer, an objective lens, and a spectrometer. The photoacoustic imaging module is used to acquire the photoacoustic image of the sample under test according to the second optical path. The second optical path consists of a second laser, a beam splitter, a first photodetector, a collimator, a first quarter-wave plate, a first dichroic mirror, a second dichroic mirror, a galvanometer, an objective lens, and a second photodetector. In the fluorescence spectroscopy module, the laser beam emitted from the excitation light module enters the dichroic mirror three, then enters the dichroic mirror two, and finally enters the objective lens through the galvanometer, ultimately striking the sample stage to perform optical scanning on the sample. The fluorescence signal fed back by the sample is separated at the dichroic mirror three via the original optical path. The fed back fluorescence signal enters the spectrometer after passing through the dichroic mirror three, and the spectrometer transmits the obtained sample fluorescence signal to the computer, ultimately obtaining the fluorescence spectrum data of the sample in the laser band emitted by the laser. In the photoacoustic imaging module, the laser beam emitted by the laser of the excitation light module enters the beam splitter and is split into two beams. One beam enters the photodetector, and the other beam enters the collimator and is then combined with the detection beam through the quarter-wave plate and the dichroic mirror. The laser of the detection optical module emits a corresponding laser beam. After passing through a half-wave plate, the beam enters a polarization beam splitter. Subsequently, the beam enters a quarter-wave plate and is merged with the excitation beam by a dichroic mirror. After the excitation beam and the probe beam are combined, the combined beam enters the dichroic mirror, then passes through the galvanometer and enters the objective lens, and finally hits the sample stage to perform optical scanning on the sample. The data processing module uses deep learning that integrates a lightweight residual network and a multi-head self-attention mechanism to process and fuse the collected fluorescence spectral data and photoacoustic image data. Based on the position, shape and intensity of the spectral absorption peaks, the absorption peaks are assigned to the corresponding chemical components, establishing the correlation between spectral features and chemical components, as well as the microstructure and texture features of Chinese medicinal materials. The data processing module includes a fluorescence spectral feature extraction section, a photoacoustic image feature extraction section, and a multimodal feature fusion and multi-scale integration section; In the fluorescence spectral feature extraction section, the feature sequence Input a multi-head self-attention module to uncover global dependencies between positions in a spectral sequence; In the fluorescence spectral feature extraction section, firstly, the acquired one-dimensional fluorescence spectral data sequence... It is mapped to a high-dimensional feature representation through an embedding layer. Where L is the sequence length of the fluorescence spectroscopy data, i.e., the number of data points, and D is the dimension after feature embedding; this embedding layer uses a learnable weight matrix. Implementation, in which The original input dimension; Subsequently, the feature sequence Input a multi-head self-attention module to mine global dependencies between positions in the spectral sequence; for the ... One attention point, among which Through a learnable weight matrix Calculate their query matrix respectively Key matrix Sum matrix : ， The output of each head is calculated using a scaled dot product attention mechanism: ， in, The feature dimension for each attention head is calculated using the following formula: ， The outputs of all h heads are concatenated along the feature dimension and then projected through a learnable linear projection matrix. The data are then integrated to obtain the final spectral feature representation. : ， in, It is a learnable linear projection weight matrix, which is not a fixed matrix, but a parameter that is continuously optimized and updated during the training of the neural network through backpropagation and gradient descent. Its final value is learned by the model from the training data, aiming to optimally fuse multi-head information to complete the task of extracting spectral features of Chinese medicinal materials. In the photoacoustic image feature extraction section, the input is a two-dimensional photoacoustic microscopic image. ,in Indicates the height of the input image; 3 represents the width of the input image; 3 represents the number of channels in the input image. First, a standard two-dimensional convolutional layer is used to encode the initial features of the two-dimensional photoacoustic microscope image to obtain the initial feature map. ; Initial feature map Deep feature extraction is performed by inputting a network consisting of three levels of lightweight residual modules connected in series. Each lightweight residual module contains two components: depthwise separable convolution and channel attention mechanism; The depthwise separable convolutional layer is responsible for spatial feature extraction. The first step is to process the input feature map. Perform channel-by-channel spatial filtering, without inter-channel mixing; （1） in, Learnable weights for depthwise convolutional kernels; output and Same number of channels; Then, fusion is performed using 1×1 convolutions. All channel information; （2） in These are the learnable weights of point convolutions. , Number of channels; Output These are the final convolutional features prepared as input to the subsequent attention module; The channel attention mechanism employs squeezing and excitation steps to calculate channel weights. ; The weights obtained from the activation steps With feature map Multiplying channel by channel yields the calibrated features. ; Finally, the final outputs of the three lightweight residual modules are generated by residual concatenation. （3） The three modules are connected via a feedforward method, and their output channels can be respectively... The output of the third module This is the final output of the entire photoacoustic image feature extraction path, and is denoted as deep spatial features. This notation describes As a structure of a three-dimensional real tensor, where, This means that all elements in the tensor are real numbers; This represents the final height of the feature map; This represents the final width of the feature map; This represents the final number of channels in the feature map; In the multimodal feature fusion and multi-scale integration section, during the feature fusion stage, the spatial features extracted by the lightweight residual module and the spectral features mined by the multi-head self-attention mechanism are deeply fused in the channel dimension to form a unified multimodal representation. The composite features are then further refined through a multi-layer convolutional backbone network to explore the intrinsic correlation between different modal features. A spatial pyramid pooling module is introduced to achieve adaptive integration of features at different resolutions through multi-scale pooling operations. The Chinese medicinal materials information module summarizes the component and structural characteristics of Chinese medicinal materials after processing fluorescence spectral data and photoacoustic imaging data from various bands, and constructs a unified multimodal characteristic database of Chinese medicinal materials for the identification of Chinese medicinal materials, quality assessment and quantitative analysis of components.
2. The dual-modal optical microscopic imaging device for detecting the structure and components of Chinese medicinal materials according to claim 1, wherein the information on the components and structural features of Chinese medicinal materials after processing the fluorescence spectral data and photoacoustic imaging data of each band includes the spectral features corresponding to the chemical components, and multi-dimensional information on the microstructure and texture attributes of the Chinese medicinal materials.