A multimodal imaging and detection system for oral and tongue

By designing a multimodal imaging and detection system for oral tongue, the simultaneous acquisition of OCT, Raman and multispectral images is achieved using public optical paths and multispectral technologies, and combined with deep feature extraction and fusion judgment modules, the existing tongue cancer examination is solved The inefficiency and inaccuracy of the method improve the diagnostic efficiency and accuracy of tongue diseases.

CN118697294BActive Publication Date: 2024-11-26HANGZHOU DIANZI UNIV
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411205946.1
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-08-30
Publication Date
2024-11-26
Estimated Expiration
2044-08-30

AI Technical Summary

Technical Problem

The existing tongue cancer examination methods cannot achieve one-time imaging, high cost, low resolution, and cannot be fully registered for examination locations. The lack of a detection system that integrates multimodal data, resulting in insufficient diagnostic efficiency and accuracy.

Method used

A multimodal imaging and detection system for oral tongue body is designed, using the combination of a common optical path and OCT, Raman and multispectral transmitting and receiving light paths, and the disjoint light branches are formed through long-wave dichroic filters to achieve OCT Simultaneous acquisition of images, Raman spectral images and multispectral images, and the depth feature extraction and fusion judgment modules are used to improve detection accuracy.

Benefits of technology

It realizes efficient integration and accurate identification of multimodal data, improves the accuracy of recognition of tongue target tissue, reduces the error introduced by multiple repeated acquisitions, and eliminates image matching operations, improving diagnostic efficiency and accuracy.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118697294B_ABST
    Figure CN118697294B_ABST
Patent Text Reader

Abstract

The present invention discloses a multimodal imaging and detection system for oral tongue. The imaging system includes a common optical path and three light receiving and light receiving paths; the three light receiving and light receiving paths are respectively an OCT light receiving and light receiving path, a Raman light receiving and light receiving path and a multispectral light receiving and light receiving path; the common optical path includes a first long-wave pass dichroic filter, a second long-wave pass dichroic filter, an X-Y scanning galvanometer and a focusing lens; the present invention uses two long-wave pass dichroic filters to form three light branches with mutually non-intersecting wavelength ranges; thereby integrating the optical signals of OCT acquisition, Raman spectrum acquisition and multispectral spectrum acquisition in the same optical path, thereby realizing the simultaneous acquisition of OCT images, Raman spectrum images and multispectral images, improving the efficiency of multimodal data acquisition; and eliminating the errors introduced in multiple repeated acquisitions, improving the accuracy of multimodal fusion recognition. At the same time, the image matching operation when multimodal data fusion is used is also omitted.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of oral cavity and tongue imaging, and in particular relates to an oral cavity and tongue multimodal imaging and detection system. Background Art

[0002] Oral cancer is a common malignant tumor in clinical practice, among which tongue cancer has the highest incidence. The characteristics of tongue cancer include high malignancy, high rate of neck metastasis and high mortality rate. With the change of people's eating habits and lifestyles, the incidence of tongue cancer is increasing year by year and gradually tends to be younger. At present, the main treatment for tongue cancer is surgery, and its effect depends to a large extent on whether lymph node metastasis occurs before surgery. Studies have shown that the metastasis rate of cervical lymph nodes is closely related to the thickness of tongue cancer.

[0003] However, existing tongue cancer examination methods mainly include oral examination, imaging examination and pathological examination, etc. These methods cannot achieve one-time imaging. Patients need to go through different departments to complete comprehensive examinations. In addition, the cost is high, the resolution is relatively low, and the examination position cannot be fully aligned. In addition, these examination results need to be directly judged by professional doctors. There is a lack of a multimodal detection system that integrates multimodal data to provide guidance to doctors. Summary of the invention

[0004] The object of the present invention is to provide a multimodal imaging and detection system for the oral cavity and tongue.

[0005] In the first aspect, the present invention provides an oral tongue multimodal imaging system, which includes a common optical path and three light receiving and light receiving paths; the three light receiving and light receiving paths are respectively an OCT light receiving and light receiving path, a Raman light receiving and light receiving path and a multi-spectral light receiving and light receiving path; the common optical path includes a first long-wave pass dichroic filter, a second long-wave pass dichroic filter, an XY scanning galvanometer and a first focusing lens; the first long-wave pass dichroic filter and the second long-wave pass dichroic filter form three light branches; the three light branches are respectively connected to the OCT light receiving and light receiving path, the Raman light receiving and light receiving path, and the multi-spectral light receiving and light receiving path; the wavelength ranges corresponding to the three light branches do not intersect with each other, and correspond to the spectral ranges of parallel light beams output by the OCT light receiving and light receiving path, the Raman light receiving and light receiving path, and the multi-spectral light receiving and light receiving path, respectively. During operation, parallel light beams output by two of the light receiving and transmitting paths are synthesized into a first synthesized light beam in the first long-wave pass dichroic filter; the first synthesized light beam and the parallel light beam output by the third light receiving and transmitting path are synthesized into a second synthesized light beam through the second long-wave pass dichroic filter; the second synthesized light beam is transmitted to the first focusing lens through the XY scanning galvanometer, and is emitted to the sample after being focused; the focused light is scanned on the sample by adjusting the XY scanning galvanometer, and the reflected light of the sample is propagated back along the original path, and the OCT light receiving and transmitting path, the Raman light receiving and transmitting path and the multi-spectral light receiving and transmitting path respectively generate OCT images, Raman spectrum images and multi-spectral images according to the received reflected light signals.

[0006] Preferably, the OCT light receiving and transmitting circuit is connected to the reflection branch of the first long-wave pass dichroic filter; the Raman light receiving and transmitting circuit is connected to the transmission branch of the first long-wave pass dichroic filter; the multi-spectrum light receiving and transmitting circuit is connected to the reflection branch of the second long-wave pass dichroic filter; the transmission band of the first long-wave pass dichroic filter in the common optical path is 720-1100nm, and the reflection band is 565-700nm; the transmission band of the second long-wave pass dichroic filter is 610nm-1100nm, and the reflection band is 495-590nm; the spectral range corresponding to the OCT light receiving and transmitting circuit is 600-700nm; the spectral range corresponding to the Raman light receiving and transmitting circuit is 810nm-1027nm; the spectral range corresponding to the multi-spectrum light receiving and transmitting circuit is 525-575nm.

[0007] Preferably, the OCT light receiving and transmitting path comprises an OCT light source, a first collimating lens, a first fiber coupler, a reference arm collimating lens, a reference mirror and an OCT spectrometer; the common optical path and the OCT light receiving and transmitting path together constitute an OCT imaging system; in the OCT imaging system, the OCT light source transmits light to the first fiber coupler through a single-mode optical fiber; the first fiber coupler divides the light into two parts; a part of the light output by the first fiber coupler enters the reference arm collimating lens through the optical fiber, and the light beam collimated by the reference arm collimating lens is directed to the reference mirror, and the reference light obtained by reflection from the reference mirror returns along the original optical path; another part of the light output by the first fiber coupler becomes a parallel light beam A after passing through the first collimating lens; the parallel light beam A passes through the common optical path and is reflected as OCT light carrying sample information; the OCT light returns to the first fiber coupler along the original optical path; the OCT light and the reference light interfere in the first fiber coupler and are transmitted to the OCT spectrometer to obtain the spectral information and image information of the OCT image.

[0008] Preferably, the Raman light transmitting and receiving circuit includes a Raman light source, a second collimating lens, a second optical fiber coupler and a Raman spectrometer; the common optical path and the Raman light transmitting and receiving circuit together constitute a Raman imaging system; in the Raman imaging system, the Raman light source transmits light to the second optical fiber coupler; the second optical fiber coupler converts the light into a parallel light beam B after passing through the second collimating lens; the parallel light beam B passes through the common optical path and is reflected as Raman light carrying sample information; the Raman light returns to the second optical fiber coupler along the original optical path; the second optical fiber coupler transmits the Raman light to the Raman spectrometer to obtain a Raman spectrum image.

[0009] Preferably, the multi-spectral light receiving and transmitting circuit includes a multi-spectral light source, a third collimating lens, a third fiber coupler and a multi-spectral spectrometer; the common optical path and the multi-spectral light receiving and transmitting circuit together constitute a multi-spectral imaging system; in the multi-spectral imaging system, the multi-spectral light source transmits light to the third fiber coupler; the third fiber coupler converts the light into a parallel light beam C after passing through the third collimating lens; the parallel light beam C passes through the common optical path and is reflected as multi-spectral light carrying sample information; the multi-spectral light returns to the third fiber coupler along the original optical path; the third fiber coupler transmits the multi-spectral light to the multi-spectral spectrometer to obtain a multi-spectral image.

[0010] In the second aspect, the present invention provides an oral tongue detection system, which includes a data acquisition module and a recognition module; the data acquisition module adopts the above-mentioned oral tongue multimodal imaging system; the recognition module includes three single-modal recognition sub-modules and a fusion judgment module; the three single-modal recognition sub-modules are respectively an OCT recognition sub-module, a Raman recognition sub-module and a multi-spectral recognition sub-module; the OCT recognition sub-module, the Raman recognition sub-module and the multi-spectral recognition sub-module respectively recognize the OCT image, the Raman spectrum image and the multi-spectral image, and judge the probability of the presence of the target tissue in the oral tongue. The Raman recognition submodule and the multispectral recognition submodule both extract deep features of input data through the feature extraction module, and output classification results through the fully connected layer; the OCT recognition submodule includes a spatial feature extraction module, a Conformer module and a fully connected layer; the spatial feature extraction module is composed of a three-dimensional convolution block and a two-dimensional convolution block; the three-dimensional convolution block operation is used to capture the spatial features of the OCT image at different depths; the two-dimensional convolution block operation is used to further extract the shallow spatial features of the OCT image; the Conformer module is divided into a local feature extraction block, a global feature extraction block and a feature interaction block; the feature interaction block is used to add the local features extracted by the local feature extraction block to the global feature extraction block after downsampling processing, and to add the global features extracted by the global feature extraction block to the local feature extraction block after upsampling processing; the fusion judgment module is used to fuse the output results of the three single-modal recognition submodules to obtain the final detection result.

[0011] Preferably, the Raman recognition submodule and the multispectral recognition submodule both include a feature extraction module and a fully connected layer; the feature extraction module is composed of multiple residual blocks; each residual block includes two 1×3 one-dimensional convolution operations, a batch normalization layer, a ReLU activation function and a fully connected layer.

[0012] Preferably, the local feature extraction block is composed of two residual blocks, each residual block contains two 1×1 two-dimensional convolution blocks and a batch normalization structure, and one 3×3 two-dimensional convolution block and a batch normalization structure; the global feature extraction block includes an attention residual block and a multi-layer perceptron residual block; the attention residual block includes a multi-head self-attention mechanism and layer normalization; the multi-layer perceptron residual block includes a multi-layer perceptron MLP and layer normalization.

[0013] Preferably, the feature interaction block includes a local feature-global feature branch and a global feature-local feature branch; the local feature-global feature branch maps the local features to the dimensions corresponding to the global features through a 1×1 two-dimensional convolution block operation, a downsampling operation, and a layer normalization operation; the global feature-local feature branch maps the global features to the dimensions corresponding to the local features through an upsampling operation, a 1×1 two-dimensional convolution block operation, and a batch normalization operation.

[0014] Preferably, the fusion judgment module obtains the final detection result by weighted averaging.

[0015] Beneficial effects of the present invention:

[0016] 1. The present invention uses two long-wave dichroic filters to form three light branches with mutually non-intersecting wavelength ranges; thereby integrating the optical signals of OCT acquisition, Raman spectrum acquisition, and multi-spectral acquisition in the same optical path, thereby realizing the simultaneous acquisition of OCT images, Raman spectrum images, and multi-spectral images, improving the efficiency of multi-modal data acquisition; and eliminating the errors introduced in multiple repeated acquisitions, improving the accuracy of multi-modal fusion recognition. At the same time, it also saves the image matching operation when using multi-modal data fusion.

[0017] 2. The present invention uses three submodules to detect OCT images, Raman spectrum images, and multispectral images respectively, and fuses the detection results, thereby improving the accuracy of tongue target tissue recognition.

[0018] 3. The OCT recognition submodule in the present invention first extracts spatial features of the OCT image, and then extracts and interacts local features and global features, which can significantly improve the accuracy of OCT image recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 It is a schematic diagram of the structure of embodiment 1 of the present invention.

[0020] Figure 2 It is a schematic diagram of the structure of the OCT spectrometer in Example 1 of the present invention.

[0021] Figure 3 It is a schematic diagram of the structure of the Raman spectrometer in Example 1 of the present invention.

[0022] Figure 4 It is a schematic diagram of the structure of the multi-spectrum spectrometer in Example 1 of the present invention.

[0023] Figure 5 It is a system block diagram of the identification module in Example 2 of the present invention.

[0024] Figure 6 It is a schematic diagram of the Raman recognition submodule and the multi-spectral recognition submodule in Example 2 of the present invention.

[0025] Figure 7 It is a schematic diagram of the OCT identification submodule in Example 2 of the present invention.

[0026] Figure 8 It is a schematic diagram of the Conformer module in Example 2 of the present invention.

[0027] Fig. 9 It is a schematic diagram of the feature interaction block in Example 2 of the present invention. DETAILED DESCRIPTION

[0028] The present invention is further described below in conjunction with the accompanying drawings. Example

[0029] like Figure 1 As shown, an oral tongue multimodal imaging system includes a common optical path and three modal light receiving and emitting paths. The three modal light receiving and emitting paths are an OCT light receiving and emitting path, a Raman light receiving and emitting path, and a multispectral light receiving and emitting path.

[0030] The common optical path includes a first long-wave pass dichroic filter 1, a second long-wave pass dichroic filter 2, an XY scanning galvanometer 3 and a first focusing lens 4; the OCT light receiving and transmitting path includes an OCT light source 5, a first collimating lens 6, a first fiber coupler 7, a reference arm collimating lens 8, a reference mirror 9 and an OCT spectrometer 10; the Raman light receiving and transmitting path includes a Raman light source 11, a second collimating lens 12, a second fiber coupler 13 and a Raman spectrometer 14; the multi-spectral light receiving and transmitting path includes a multi-spectral light source 15, a third collimating lens 16, a third fiber coupler 17 and a multi-spectral spectrometer 18. In the multi-modal imaging system, single-mode optical fiber is used to transmit light.

[0031] The first long-wave dichroic filter 1 and the second long-wave dichroic filter 2 form two-stage three-light branches. The two light branches corresponding to the first long-wave dichroic filter 1 are respectively connected to the OCT light receiving and transmitting circuit and the Raman light receiving and transmitting circuit. The one light branch corresponding to the second long-wave dichroic filter 2 is connected to the multi-spectrum light receiving and transmitting circuit.

[0032] Specifically, in the imaging system, the parallel light beam A obtained by the OCT light receiving and transmitting path and the parallel light beam B obtained by the Raman light receiving and transmitting path are synthesized into a first synthesized light beam through the first long-wave pass dichroic filter 1, and the first synthesized light beam and the parallel light beam C obtained by the multi-spectral light receiving and transmitting path are synthesized into a second synthesized light beam through the second long-wave pass dichroic filter 2. The second synthesized light beam is transmitted to the first focusing lens 4 through the XY scanning galvanometer 3, and is emitted to the sample after being focused. The focused light is scanned on the sample by adjusting the XY scanning galvanometer 3, and is transmitted back along the original path after being reflected on the sample.

[0033] The common optical path and the OCT light-receiving and light-emitting path together constitute the OCT imaging system. In the OCT imaging system, the OCT light source transmits light to the first optical fiber coupler 7 through a single-mode optical fiber; the first optical fiber coupler 7 divides the light into two parts, one part enters the reference arm through the optical fiber, wherein the reference arm is composed of a reference arm collimating lens 8 and a reference mirror 9, the light is collimated by the reference arm collimating lens 8 and then directed to the reference mirror, and is reflected by the reference mirror and returned along the original optical path; the other part enters the sample arm through the optical fiber, wherein the sample arm is composed of a first collimating lens 6 and a common optical path, and the light becomes a parallel beam A after passing through the first collimating lens 6. After passing through the above-mentioned common optical path, the OCT light carrying the sample information is returned along the original optical path, the OCT light returned in the sample arm interferes with the light reflected back in the reference arm in the first optical fiber coupler, and is transmitted to the OCT spectrometer 10 through the optical fiber to obtain spectral information and image information. The spectral information can reflect the difference between the normal part and the diseased part of the oral tongue, and the image information can not only reflect the two-dimensional plane layer, but also reflect the depth information, both of which are convenient for the subsequent identification and diagnosis of oral tongue diseases.

[0034] like Figure 2 As shown, the OCT spectrometer 10 is composed of a collimating lens A, a transmission diffraction grating 19, a second focusing lens 20 and a linear array CCD 21. Light is collimated by the collimating lens A and then emitted to the transmission diffraction grating 19. After being split by the grating, the light is focused on the linear array CCD 21 by the focusing lens 20.

[0035] The common optical path and the Raman light transmission and receiving path together constitute the Raman imaging system. In the Raman imaging system, the Raman light source 11 transmits light to the second optical fiber coupler 13 through a single-mode optical fiber; the second optical fiber coupler 13 transmits light through an optical fiber to the focus of the second collimating lens 12, and the light passes through the second collimating lens to obtain a parallel light beam B. After passing through the above-mentioned common optical path, the Raman light carrying the sample information is returned along the original optical path, and enters the Raman spectrometer 14 through the optical fiber coupler to obtain spectral information. This spectral information can obtain the difference between the spectral peaks of the normal position and the diseased position on the oral tongue, which is convenient for the subsequent differential diagnosis of oral tongue diseases.

[0036] like Figure 3As shown, the Raman spectrometer 14 is composed of a reflective collimating lens 22, a reflective diffraction grating 23, a reflective focusing lens 24 and a linear array CCD 25. It adopts a CT structure (reflective structure). The light passes through the reflective collimating lens 22 to collimate the light and reflect it to the reflective diffraction grating 23. After the grating splits the light, it is reflected to the reflective focusing lens 24, and then it is focused by the focusing lens and reflected to the linear array CCD 25.

[0037] The common optical path and the multi-spectral light-receiving and light-emitting path together constitute a multi-spectral imaging system. In the multi-spectral imaging system, the multi-spectral light source 15 transmits light to the third optical fiber coupler 17 through a single-mode optical fiber, and the coupler transmits light to the focus of the third collimating lens 16 through the optical fiber. After the light passes through the third collimating lens, a parallel light beam C is obtained. After passing through the above-mentioned common optical path, the multi-spectral light carrying the sample information is returned according to the original optical path, and enters the multi-spectral spectrometer 18 through the optical fiber coupler to obtain spectral information and image information. This spectral information can reflect the differences in tongue quality and tongue coating on the tongue body, which is convenient for the subsequent separation of tongue coating and tongue quality, so as to distinguish changes in tongue coating and tongue quality caused by diseases.

[0038] like Figure 4 As shown, the multi-spectral spectrometer 18 is composed of a collimating lens B, a transmissive diffraction grating 26, a third focusing lens 27 and a linear array CCD 28. Light is collimated by the collimating lens and then directed to the transmissive diffraction grating 26. After being split by the grating, the light is focused on the linear array CCD 28 by the focusing lens 27.

[0039] In this embodiment, the first long-wave pass dichroic filter 1 in the common optical path has a cutoff wavelength of 710 nm, a transmission band of 720 to 1100 nm, and a reflection band of 565 to 700 nm.

[0040] In this embodiment, the second long-wave pass dichroic filter 2 in the common optical path has a cutoff wavelength of 600 nm, a transmission band of 610 nm to 1100 nm, and a reflection band of 495 nm to 590 nm.

[0041] In this embodiment, the effective focal length of the first focusing lens 4 in the common optical path is 30 mm, and the lens diameter is 15 mm.

[0042] In this embodiment, the OCT light source 5 in the OCT imaging system is a superluminescent diode, which has a central wavelength of 650 nm, a power of 5 mW, and a full width at half maximum of 45 nm.

[0043] In this embodiment, the first collimating lens 6, the collimating lens A and the reference arm collimating lens 8 in the OCT imaging system all have an effective focal length of 20 mm and a diameter of 12.85 mm.

[0044] In this embodiment, the transmission diffraction grating 19 in the OCT imaging system has a line rate of 1500 lines / mm, an incident angle of 29.176°, and a size of 8 mm×8 mm.

[0045] In this embodiment, the effective focal length of the second focusing lens 20 of the spectrometer in the OCT imaging system is 150 mm, and the diameter is 25 mm.

[0046] In this embodiment, the linear array CCD 21 of the OCT imaging system has 2048 pixels, and the pixel size is 14um×14um.

[0047] In this embodiment, the first optical fiber coupler 7 in the OCT imaging system is a single-mode 2×2 broadband optical fiber coupler, using a central wavelength of 650 nm±50 nm and a coupling ratio of 50:50.

[0048] In this embodiment, the single-mode optical fiber used by the OCT imaging system has an operating wavelength of 600-700 nm and a numerical aperture NA of 0.12.

[0049] In this embodiment, the spot size of the light after the first collimating lens 6, the collimating lens A and the reference arm collimating lens 8 of the OCT imaging system collimate the light is about 4.8 mm.

[0050] In this embodiment, the OCT imaging system can be used to obtain high-resolution tomographic depth images and stereoscopic images of the oral tongue. Its lateral resolution ∆x is about 5.172um, and its axial resolution ∆z is about 4.131um. OCT images can intuitively show the cross-sectional structure, thickness, lesion depth, and vascular distribution of the tongue, which is helpful for diagnosing and evaluating the scope and depth of lesions.

[0051] In this embodiment, the Raman light source 11 in the Raman imaging system is a semiconductor laser with a central wavelength of 785±0.5nm, an effective power of 20mW, and a collection area of ​​400-3000cm -1 spectral range.

[0052] In this embodiment, the effective focal length of the second collimating lens 12 in the Raman imaging system is 20 mm, and the diameter is 12.85 mm.

[0053] In this embodiment, the curvature radius of the reflective collimating objective lens 22 of the Raman imaging system is 194 mm, the reflective focal length is 82 mm, and the incident angle is 5°.

[0054] In this embodiment, the reflective diffraction grating 23 of the Raman imaging system has a line rate of 1200 lines / mm and an incident angle of 47.73°.

[0055] In this embodiment, the curvature radius of the reflective focusing lens 24 of the Raman imaging system is 194 mm, the reflective focal length is 100 mm, and the incident angle is 12°.

[0056] In this embodiment, the linear array CCD 25 of the Raman imaging system has 2048 pixels, and the pixel size is 14um×200um.

[0057] In this embodiment, the second optical fiber coupler in the Raman imaging system is a single-mode 1×2 broadband optical fiber coupler, using a central wavelength of 900±150 nm and a coupling ratio of 50:50.

[0058] In this embodiment, the single-mode optical fiber used by the Raman imaging system has an operating wavelength of 780-1100 nm and a numerical aperture NA of 0.12.

[0059] In this embodiment, the spot size of the light after the second collimating lens 12 of the Raman imaging system collimates the light is about 4.8 mm.

[0060] In this embodiment, the Raman imaging system can be used to obtain molecular functional information on tongue tissue. The Raman scattering wavelength is between 810nm and 1027nm, that is, the spectrum range is 400 to 3000cm -1 , spectral resolution is 7~10.5cm -1 , which helps doctors diagnose and monitor the health of the tongue and study related pathogenesis.

[0061] In this embodiment, the multi-spectral light source 15 in the multi-spectral imaging system is a diode laser with a central wavelength of 550 nm and a power of 5 mW.

[0062] In this embodiment, the third collimating lens 16 and the collimating lens B in the multi-spectral imaging system both have an effective focal length of 20 mm and a diameter of 12.85 mm.

[0063] In this embodiment, the transmission diffraction grating 26 in the multi-spectral imaging system has a wavelength of 1200 lines / mm, an incident angle of 19.269°, and a size of 8 mm×8 mm.

[0064] In this embodiment, the third focusing lens 27 in the multi-spectral imaging system has an effective focal length of 150 mm and a diameter of 30 mm.

[0065] In this embodiment, the linear array CCD 28 of the multi-spectral imaging system has 2048 pixels, and the pixel size is 14um×14um.

[0066] In this embodiment, the third optical fiber coupler in the multi-spectral imaging system is a single-mode 1×2 broadband optical fiber coupler, using a central wavelength of 550±50 nm and a coupling ratio of 50:50.

[0067] In this embodiment, the single-mode optical fiber used in the multi-spectral imaging system has an operating wavelength of 488-633 nm and a numerical aperture NA of 0.12.

[0068] In this embodiment, the spot size of the light after the third collimating lens 16 and the collimating lens B of the multi-spectral imaging system collimate the light is about 4.8 mm.

[0069] In this embodiment, the multi-spectral imaging system can be used to obtain tongue surface and multi-band spectral information of the tongue. The spectral range is 525nm to 575nm, and this spectral information can be used to analyze the biochemical composition, tissue structure and metabolic state of the tongue, thereby helping to diagnose and evaluate the health of the tongue and monitor changes in diseases. Example

[0070] like Figure 5 As shown, an oral tongue detection system includes a data acquisition module and a recognition module. The data acquisition module adopts the oral tongue multimodal imaging system described in Example 1 to quickly obtain OCT, Raman, and multispectral detection data of the same position of the tongue. In some other embodiments, the data acquisition module can also use independent OCT imaging systems, Raman imaging systems, and multispectral imaging systems to perform data acquisition.

[0071] The recognition module includes three single-modality recognition submodules and a fusion judgment module; the three single-modality recognition submodules are respectively an OCT recognition submodule, a Raman recognition submodule and a multi-spectral recognition submodule.

[0072] The Raman recognition submodule and the multispectral recognition submodule are both based on the one-dimensional residual network (1DResNet), which are used to analyze Raman spectral data and multispectral data respectively; the Raman recognition submodule and the multispectral recognition submodule both use the feature extraction module to extract the deep features of the data and output the classification results through the fully connected layer. The OCT recognition submodule is based on the improved Conformer model (i.e., the 3D-Conformer model) and is used to analyze OCT data; in the OCT recognition submodule, the input data first passes through the spatial feature extraction layer to capture the spatial features and shallow information of the OCT image at different depths; then, the local and global features are further extracted and integrated through the Conformer module. Finally, the classification results are obtained after processing in the fully connected layer.

[0073] like Figure 6As shown in the figure, the Raman recognition submodule and the multispectral recognition submodule both include a feature extraction module and a fully connected layer. The feature extraction module consists of multiple residual blocks; each residual block includes two 1×3 one-dimensional convolutional layers (1×3, 1DConv), a batch normalization layer (BatchNorm, BN), a ReLU activation function, and a fully connected layer. These components are used together to deeply extract the feature information of multispectral and Raman spectral data. After the feature extraction is completed, it is input into the fully connected layer for further processing to finally obtain the classification result.

[0074] like Figure 7 As shown in the figure, the OCT recognition submodule includes a spatial feature extraction module, a Conformer module, and a fully connected layer. The spatial feature extraction module (SFE Module) consists of a three-dimensional convolution block (3DConv) and a two-dimensional convolution block (2DConv); the three-dimensional convolution block is used to capture the spatial features of the OCT image at different depths; the two-dimensional convolution block is used to further extract the shallow spatial features of the OCT image.

[0075] The Conformer module is divided into a local feature extraction block (LFEBlock), a global feature extraction block (GFEBlock), and a feature interaction block (FIBlock). LFEBlock focuses on extracting local features of OCT images, while GFEBlock focuses on capturing global features of images. FIBlock adds the local features extracted by LFEBlock to GFEBlock after downsampling to enhance local details in global features; similarly, FIBlock adds the global features extracted by GFEBlock to LFEBlock after upsampling, which enhances the global perception of the local feature extraction block. After feature extraction by the Conformer module, it is input into the fully connected layer for further processing to obtain the classification result.

[0076] like Figure 8 As shown in the figure, LEFBlock is designed based on the traditional ResNet architecture. Each LEFBlock consists of two residual blocks, each of which contains two 1×1 two-dimensional convolution blocks and batch normalization (1×1Conv+BN) and one 3×3 two-dimensional convolution block and batch normalization (3×3Conv+BN). It is designed to effectively extract local features.

[0077] GFEBlock is designed based on the Transformer model. It consists of a residual block based on the attention mechanism and a multi-layer perceptron residual block. The attention residual block contains a multi-head self-attention mechanism (MHSA) and layer normalization (LayerNorm, LN), while the multi-layer perceptron residual block consists of a multi-layer perceptron (MLP) and layer normalization. It is designed to capture and extract global features.

[0078] FIBlock consists of two key parts: local feature-global feature branch (LF-GF Route) and global feature-local feature branch (GF-LF Route). This structure is designed to promote the interaction and fusion between local features and global features, thereby optimizing the feature integration process and improving the overall performance of the model.

[0079] LF-GF Route maps local features to the dimensions corresponding to global features through a 1×1 2D convolution block operation (1×1Conv), downsampling operation (Down), and layer normalization (LayerNorm, LN), and performs corresponding integration operations to fuse the two features. In contrast, GF-LF Route uses upsampling (Upsampling), a 1×1 2D convolution block operation, and batch normalization to map global features back to the dimensions corresponding to local features and effectively integrate them with local features.

[0080] In this embodiment, the local features and global features extracted by LFEBlock and GFEBlock are inconsistent in dimension, so it is necessary to unify the dimensions of these features through FIBlock.

[0081] like Fig. 9 As shown in the figure, the feature dimension extracted by LFEBlock is C×H×W. In the local feature-global feature branch, these features are first adjusted to C1×H×W through a 1×1 convolution block operation. Then, through the downsampling operation, the dimension of the feature is further transformed to C1×H1×W1. Subsequently, these features are reshaped into a vector table with H1×W1 rows and C1 columns, so that it is aligned with the word vector table extracted by GFEBlock in terms of dimension, that is, H1×W1 equals a and C1 equals b. Through a specific integration operation, the two features are effectively fused.

[0082] In the global feature-local feature branch, feature processing begins by adjusting the word vector table of a×b dimensions to C2×H2×W2 dimensions. After upsampling, the feature dimension becomes C2×H3×W3. After that, through a 1×1 convolution layer, the feature dimension is finally adjusted to C3×H3×W3, which matches the feature dimension extracted by LFEBlock, that is, H3 equals H, W3 equals W, and C3 equals C. Finally, through a specific fusion operation, these global features are effectively combined with local features.

[0083] The fusion judgment module is used to weight or average the results (specifically, the predicted probabilities) output by the three single-modal recognition sub-modules through a soft voting mechanism, thereby selecting the category with the highest probability as the final result and making the final judgment to identify whether there is a lesion on the tongue.

[0084] In this embodiment, the target tissue detected by the oral tongue detection system is the tissue corresponding to tongue squamous cell carcinoma, and the detection method is as follows:

[0085] Step 1: Use the data acquisition module to collect multimodal image data of different tongues to construct a labeled tongue dataset; samples of the tongue dataset include OCT images, Raman spectral images, multispectral images of the tongue, and a label indicating whether there is tongue squamous cell carcinoma.

[0086] Step 2: Use the tongue dataset to train the OCT recognition submodule, Raman recognition submodule and multispectral recognition submodule respectively.

[0087] Step 3: Input the OCT image, Raman spectrum image, and multi-spectral image of the subject into the OCT recognition submodule, Raman recognition submodule, and multi-spectral recognition submodule respectively. The detection results output by the OCT recognition submodule, Raman recognition submodule, and multi-spectral recognition submodule are input into the fusion judgment module. The fusion judgment module outputs the final tongue lesion detection result. The detection result obtained in this embodiment is only an intermediate result to assist the doctor, and the diagnosis result of the disease cannot be directly obtained.

Claims

1. An oral tongue multimodal detection system, comprising a data acquisition module and a recognition module; characterized in that: The recognition module includes three single-modality recognition submodules and a fusion judgment module; the three single-modality recognition submodules are respectively an OCT recognition submodule, a Raman recognition submodule and a multi-spectral recognition submodule; the OCT recognition submodule, the Raman recognition submodule and the multi-spectral recognition submodule respectively recognize the OCT image, the Raman spectrum image and the multi-spectral image, and judge the probability of the target tissue existing in the oral tongue; The Raman recognition submodule and the multispectral recognition submodule both extract deep features of input data through the feature extraction module, and output classification results through the fully connected layer; the OCT recognition submodule includes a spatial feature extraction module, a Conformer module and a fully connected layer; The spatial feature extraction module is composed of a three-dimensional convolution block and a two-dimensional convolution block; the three-dimensional convolution block is used to capture the spatial features of the OCT image at different depths; the two-dimensional convolution block is used to further extract the shallow spatial features of the OCT image; the Conformer module is divided into a local feature extraction block, a global feature extraction block and a feature interaction block; the feature interaction block is used to add the local features extracted by the local feature extraction block to the global feature extraction block after downsampling, and to add the global features extracted by the global feature extraction block to the local feature extraction block after upsampling; The fusion judgment module is used to fuse the output results of the three single-modal recognition sub-modules to obtain the final detection result; The data acquisition module adopts an oral tongue multimodal imaging system; the oral tongue multimodal imaging system comprises a common optical path and three light receiving and light receiving paths; the three light receiving and light receiving paths are respectively an OCT light receiving and light receiving path, a Raman light receiving and light receiving path and a multispectral light receiving and light receiving path; the common optical path comprises a first long-wave pass dichroic filter (1), a second long-wave pass dichroic filter (2), an XY scanning galvanometer (3) and a first focusing lens (4); the first long-wave pass dichroic filter (1) and the second long-wave pass dichroic filter (2) form three light branches; the three light branches are respectively connected to the OCT light receiving and light receiving path, the Raman light receiving and light receiving path and the multispectral light receiving and light receiving path; the wavelength ranges corresponding to the three light branches do not intersect with each other, and respectively correspond to the spectral ranges of parallel light beams output by the OCT light receiving and light receiving path, the Raman light receiving and light receiving path and the multispectral light receiving and light receiving path; During operation, parallel light beams output by two of the light receiving and transmitting paths are synthesized into a first synthesized light beam in a first long-wavelength pass dichroic filter (1); the first synthesized light beam and the parallel light beam output by the third light receiving and transmitting path are synthesized into a second synthesized light beam through a second long-wavelength pass dichroic filter (2); the second synthesized light beam is transmitted to a first focusing lens (4) through an XY scanning galvanometer (3) and is emitted to a sample after being focused; the XY scanning galvanometer (3) is adjusted so that the focused light is scanned on the sample, and the reflected light of the sample is propagated back along the original path, and the OCT light receiving and transmitting path, the Raman light receiving and transmitting path and the multi-spectral light receiving and transmitting path respectively generate an OCT image, a Raman spectrum image and a multi-spectral image according to the received reflected light signal.

2. The oral tongue multimodal detection system according to claim 1, characterized in that: The OCT light receiving and transmitting circuit is connected to the reflection branch of the first long-wave pass dichroic filter (1); the Raman light receiving and transmitting circuit is connected to the transmission branch of the first long-wave pass dichroic filter (1); the multi-spectral light receiving and transmitting circuit is connected to the reflection branch of the second long-wave pass dichroic filter (2); the transmission band of the first long-wave pass dichroic filter (1) in the common optical path is 720-1100nm, and the reflection band is 565-700nm; the transmission band of the second long-wave pass dichroic filter (2) is 610nm-1100nm, and the reflection band is 495-590nm; the spectral range corresponding to the OCT light receiving and transmitting circuit is 600-700nm; the spectral range corresponding to the Raman light receiving and transmitting circuit is 810nm-1027nm; and the spectral range corresponding to the multi-spectral light receiving and transmitting circuit is 525-575nm.

3. The oral tongue multimodal detection system according to claim 1, characterized in that: The OCT light receiving and transmitting path comprises an OCT light source (5), a first collimating lens (6), a first optical fiber coupler (7), a reference arm collimating lens (8), a reference mirror (9) and an OCT spectrometer (10); the common optical path and the OCT light receiving and transmitting path together constitute an OCT imaging system; in the OCT imaging system, the OCT light source transmits light to the first optical fiber coupler (7) via a single-mode optical fiber; the first optical fiber coupler (7) divides the light into two parts; a part of the light output by the first optical fiber coupler (7) enters the reference arm collimating lens (8) via the optical fiber, the light beam collimated by the reference arm collimating lens (8) is directed toward the reference mirror (9), and the reference light obtained by reflection from the reference mirror is returned along the original optical path; another part of the light output by the first optical fiber coupler (7) is converted into a parallel light beam A after passing through the first collimating lens (6); The parallel light beam A passes through the common optical path and is reflected to obtain OCT light carrying sample information; the OCT light returns to the first optical fiber coupler (7) along the original optical path; the OCT light and the reference light interfere with each other in the first optical fiber coupler and are transmitted to the OCT spectrometer (10) to obtain spectral information and image information of the OCT image.

4. The oral tongue multimodal detection system according to claim 1, characterized in that: The Raman light-receiving and light-emitting path comprises a Raman light source (11), a second collimating lens (12), a second optical fiber coupler (13) and a Raman spectrometer (14); the common optical path and the Raman light-receiving and light-emitting path together constitute a Raman imaging system; in the Raman imaging system, the Raman light source (11) transmits light to the second optical fiber coupler (13); the second optical fiber coupler (13) converts the light into a parallel light beam B after passing through the second collimating lens (12); the parallel light beam B passes through the common optical path and is reflected to obtain Raman light carrying sample information; the Raman light returns to the second optical fiber coupler (13) along the original optical path; the second optical fiber coupler (13) transmits the Raman light to the Raman spectrometer (14) to obtain a Raman spectrum image.

5. The oral tongue multimodal detection system according to claim 1, characterized in that: The multi-spectral light receiving and transmitting path comprises a multi-spectral light source (15), a third collimating lens (16), a third optical fiber coupler (17) and a multi-spectral spectrometer (18); the common optical path and the multi-spectral light receiving and transmitting path together constitute a multi-spectral imaging system; in the multi-spectral imaging system, the multi-spectral light source (15) transmits light to the third optical fiber coupler (17); the third optical fiber coupler (17) converts the light into a parallel light beam C after passing through the third collimating lens (16); the parallel light beam C passes through the common optical path and is reflected to obtain multi-spectral light carrying sample information; the multi-spectral light returns to the third optical fiber coupler (17) along the original optical path; the third optical fiber coupler (17) transmits the multi-spectral light to the multi-spectral spectrometer (18) to obtain a multi-spectral image.

6. The oral tongue multimodal detection system according to claim 1, characterized in that: The Raman recognition submodule and the multispectral recognition submodule both include a feature extraction module and a fully connected layer; the feature extraction module is composed of multiple residual blocks; each residual block includes two 1×3 one-dimensional convolutional layers, a batch normalization layer, a ReLU activation function and a fully connected layer.

7. The oral tongue multimodal detection system according to claim 1, characterized in that: The local feature extraction block is composed of two residual blocks, each residual block contains two 1×1 two-dimensional convolution blocks and a batch normalization structure, and one 3×3 two-dimensional convolution block and a batch normalization structure; the global feature extraction block includes an attention residual block and a multi-layer perceptron residual block; the attention residual block includes a multi-head self-attention mechanism and layer normalization; the multi-layer perceptron residual block includes a multi-layer perceptron MLP and layer normalization.

8. The oral and tongue multimodal detection system according to claim 1, characterized in that: The feature interaction block includes a local feature-global feature branch and a global feature-local feature branch; the local feature-global feature branch maps the local features to the dimensions corresponding to the global features through two-dimensional convolution operations, downsampling operations, and layer normalization operations; the global feature-local feature branch maps the global features to the dimensions corresponding to the local features through upsampling operations, two-dimensional convolution operations, and batch normalization operations.

9. The oral and tongue multimodal detection system according to claim 1, characterized in that: The fusion judgment module obtains the final detection result by weighted averaging.