Dual-mode imaging apparatus and detection system for optical coherence tomography and endoscopy of the digestive tract
The digestive tract optical coherence tomography and endoscopic dual-modal imaging device, which combines a dual-modal capsule probe with an OCT extracorporeal optical path module, realizes dual-modal image acquisition and real-time intelligent detection at the same location in the digestive tract. It solves the shortcomings of traditional endoscopes and improves the accuracy and portability of early lesion detection.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HANGZHOU DIANZI UNIV
- Filing Date
- 2022-10-25
- Publication Date
- 2026-05-22
AI Technical Summary
Existing gastrointestinal examination techniques are difficult to acquire dual-modal images of the same location simultaneously. Traditional endoscopy requires sedation and places high demands on the physician's expertise. Computer-aided diagnostic systems have failed to effectively combine optical endoscopy with optical coherence tomography (OCT) images for multimodal lesion detection.
A dual-modal imaging device for the digestive tract using optical coherence tomography and endoscopy was designed. It combines a dual-modal capsule probe with an OCT extracorporeal optical path module, separates near-infrared light and visible light using a dichroic mirror, achieves 360° image acquisition using a rotation drive component, and combines a lightweight convolutional neural network for lesion detection, realizing real-time imaging and intelligent detection of the digestive tract surface and depth.
It enables the simultaneous acquisition of OCT and endoscopic images of the same location in the digestive tract, reducing the risk of damage to the digestive tract, reducing the size of the device, providing real-time detection capabilities, improving the screening and detection capabilities for early lesions of the digestive tract, and reducing the professional requirements for doctors.
Smart Images
Figure CN115553686B_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of medical device technology, specifically relating to a dual-modal imaging device and detection system for the digestive tract optical coherence tomography and endoscopy. Background Technology
[0002] With the continuous development of society, the incidence of digestive tract diseases is rising year by year. Digestive tract cancers caused by these diseases have become a significant threat to people's health. Related studies have shown that early detection and treatment of digestive tract diseases are effective means to prevent their further deterioration. Therefore, achieving early screening for digestive tract lesions is of great significance for reducing morbidity and mortality.
[0003] Currently, most clinically used endoscopes employ cold light sources as illumination media, using optical sensors to image the surface tissues of the digestive tract for the detection of digestive tract diseases. While common digestive tract lesions can be diagnosed by observing the tissue surface, early symptoms of some lesions often occur 1-3 mm below the mucosa, making them difficult to detect with traditional optical endoscopes. Optical coherence tomography (OCT), based on the principle of weakly coherent light interference, obtains structural information in the tissue depth direction by measuring the interference signal between the sample's reflected light and the reference arm's reflected light. Its imaging depth is greater than 3 mm, and it can achieve two-dimensional or three-dimensional imaging of the sample using a scanning device on the sample arm, offering advantages such as high resolution and non-invasive imaging. Therefore, combining optical coherence tomography with traditional optical endoscopes to achieve both color imaging of the digestive tract surface and imaging in the tissue depth direction can overcome the shortcomings of traditional optical endoscopes. A review of existing technologies reveals that there is still much room for improvement in current multimodal endoscopic examination techniques that combine OCT technology with traditional endoscopy. Patents CN110881942A, CN114587245A, and CN102824154 all combine rigid endoscopes with OCT technology. When applied to gastrointestinal examinations, these technologies require patient sedation, causing significant discomfort. The dual-modal capsule endoscopy system proposed in patent CN102697438A cannot acquire dual-modal images of the same location in the digestive tract and cannot adjust the detection direction. Furthermore, none of these disclosed technologies address the processing of the acquired multimodal data to achieve intelligent lesion detection.
[0004] Multimodal examination methods generate a large amount of data from different modalities simultaneously, requiring clinicians to be proficient in reviewing data from both modalities. This places higher demands on clinicians' professional skills and limits the application and development of multimodal examination methods in remote areas. With the development of deep learning technology, many researchers have studied computer-aided diagnostic techniques for gastrointestinal endoscopy. However, most current computer-aided diagnostic systems are designed for traditional optical endoscopic images and cannot combine information from lesion surface images and coherent optical tomography images to achieve multimodal lesion detection. Therefore, developing a dual-modal imaging device and detection system for the gastrointestinal tract using optical coherence tomography and endoscopy, which collects dual-modal information from the same location in the gastrointestinal tract and further combines deep learning technology to achieve real-time intelligent detection of lesions in both color images and coherent optical tomography images, is of great significance for the early detection and diagnosis of gastrointestinal diseases. Summary of the Invention
[0005] The purpose of this invention is to address the above-mentioned problems by providing a dual-modal imaging device and detection system for the digestive tract using optical coherence tomography and endoscopy.
[0006] In a first aspect, the present invention provides a dual-modal imaging device for the digestive tract using optical coherence tomography and endoscopy, comprising an OCT extracorporeal optical path module and a dual-modal capsule probe. The dual-modal capsule probe is connected to the OCT extracorporeal optical path module via a signal transmission line containing an optical fiber core.
[0007] The dual-modal capsule probe includes a capsule shell, a dichroic mirror, an OCT acquisition module, an endoscope imaging module, and a rotation drive assembly. A transparent window surrounds the capsule shell in its center. The dichroic mirror is installed inside the capsule shell and is driven to rotate by the rotation drive assembly. The rotation axis of the dichroic mirror is parallel to the central axis of the capsule shell; the side of the dichroic mirror forms an angle of 30° to 60° with the central axis of the capsule shell. The dichroic mirror reflects near-infrared light and transmits visible light.
[0008] The OCT acquisition module includes a lens holder, a beam collimator, an achromatic lens, a liquid lens, and an aspherical lens. The lens holder is fixed to the end of the capsule shell's inner cavity; the beam collimator, achromatic lens, liquid lens, and aspherical lens are housed in the central mounting hole of the lens holder; the fiber optic core of the signal transmission line is connected to the central mounting hole of the lens holder. Near-infrared light from the dual-mode capsule probe input from the fiber optic core passes sequentially through the beam collimator, achromatic lens, liquid lens, and aspherical lens, and is reflected by a dichroic mirror towards the transparent window of the capsule shell, thus entering the tissue.
[0009] The liquid lens described herein is based on the principle of electrowetting. By applying an external voltage to change the shape of the liquid, the curvature of the liquid is altered, thereby achieving automatic focusing and zooming.
[0010] The endoscopic imaging module is fixed to the dichroic mirror; and the signal receiving surface of the endoscopic imaging module faces the dichroic mirror. The orientation of the endoscopic imaging module is consistent with the direction of the near-infrared light emitted by the OCT acquisition module reflected on the dichroic mirror.
[0011] During operation, near-infrared and visible light reflection signals from the same location and at the same time in the digestive tract are reflected towards the dichroic mirror. The near-infrared light reflection signal, after reflection by the dichroic mirror, is transmitted from the OCT acquisition module to the OCT extracorporeal optical path module. The visible light reflection signal, after transmission through the dichroic mirror, illuminates the endoscope imaging module.
[0012] Preferably, the signal transmission line is a hybrid optical-electrical cable. The hybrid optical-electrical cable contains optical fiber cores and electrical signal transmission cores. The optical fiber cores in the hybrid optical-electrical cable are connected to the optical fiber coupler in the OCT external optical path module.
[0013] Preferably, the OCT in vitro optical path module includes a near-infrared light source, a photodetector, a fiber optic coupler, and a reference arm optical path. The near-infrared light source is used for OCT tomography; the photodetector converts the OCT interference light signal into an electrical signal and transmits it to a computer for processing; the fiber optic coupler splits the near-infrared light source proportionally into sample light and reference light, and causes interference between the reflected reference light and sample light; the reference arm optical path reflects the reference light back to the fiber optic coupler. The sample light output from the fiber optic coupler is transmitted to the OCT acquisition module of the dual-modal capsule probe.
[0014] Preferably, an illumination lamp is provided inside the capsule shell. The illumination lamp, liquid lens, rotation drive assembly, and endoscope imaging module are all supplied with power and control signals through the electrical signal transmission core wire in the signal transmission line; the image data detected by the endoscope imaging module is output through the electrical signal transmission core wire.
[0015] Preferably, during operation, the internal illumination lamp of the dual-modal capsule probe provides the necessary lighting conditions for image acquisition by the endoscopic imaging module.
[0016] Preferably, the rotary drive assembly includes a micromotor, a motor bracket, and a mounting bracket. The micromotor is fixed to the end of the inner cavity of the capsule shell via the motor bracket; the output axis of the micromotor coincides with the axis of the capsule shell; the mounting bracket is fixed to the output shaft of the micromotor. The dichroic mirror is fixed to the mounting bracket. The mounting bracket has a cutout in the area between the endoscope imaging module and the dichroic mirror.
[0017] Preferably, the side of the dichroic mirror forms a 45° angle with the central axis of the capsule shell. The orientation of the endoscopic imaging module is perpendicular to the central axis of the capsule shell.
[0018] Preferably, the endoscopic imaging module includes an endoscopic imaging lens and an imaging sensor; both the endoscopic imaging lens and the imaging sensor are fixed to a dichroic mirror. The image detection area of the endoscopic imaging lens is aligned with that of the imaging sensor. The endoscopic imaging lens faces the dichroic mirror and is able to receive the signal light transmitted from the dichroic mirror.
[0019] Secondly, the present invention provides a digestive tract detection system, including an image preprocessing module, a lesion detection module, and the aforementioned digestive tract optical coherence tomography and endoscopic dual-modal imaging device. The dual-modal capsule probe outputs OCT spectral data and endoscopic images, which are transmitted to the image preprocessing module. The image preprocessing module then transmits the preprocessed OCT images and endoscopic images to the lesion detection module.
[0020] Preferably, the dichroic mirror rotates 360° every time the dual-modal capsule probe reaches a collection position; the dichroic mirror pauses once every time it rotates a preset angle, and the OCT external optical path module and the endoscope imaging module collect data when the dichroic mirror pauses, obtaining OCT spectral data and endoscope images of different orientations within a 360° range of the collection position.
[0021] Preferably, the image preprocessing module preprocesses the OCT spectral data by converting the OCT spectral data into an OCT image, denoising the OCT image, and enhancing the OCT image. The image preprocessing module preprocesses the endoscopic image by extracting image frames, removing image edges, denoising the image, enhancing the image, and matching the image.
[0022] Preferably, the lesion detection module includes an OCT image lesion detection network and an endoscopic image lesion detection network. Both the OCT image lesion detection network and the endoscopic image lesion detection network employ lightweight target detection networks based on convolutional neural networks, including a feature extraction network, a context enhancement module, convolutional layers, a spatial attention module, and fully connected layers. To address the issue of varying lesion sizes in OCT images, the feature extraction network of the OCT image lesion detection network uses a feature pyramid structure to fuse multi-scale features, enhancing the network's ability to detect lesions of different scales. To address the issue of lesion areas appearing similar to normal areas of digestive tract tissue in endoscopic images, and the lesion sizes varying, a convolutional block attention mechanism is added to the feature extraction network of the endoscopic image lesion detection network to strengthen key features of the lesion area.
[0023] Preferably, the digestive tract detection system also includes a multimodal information fusion module; the lesion detection module transmits the feature signals output by the OCT image lesion detection network and the endoscopic image lesion detection network to the multimodal information fusion module; the multimodal information fusion module is used to combine the feature signals output by the OCT image lesion detection network and the endoscopic image lesion detection network to assist doctors in judging the condition.
[0024] Preferably, the digestive tract detection system also includes a display and early warning module; the display and early warning module includes a real-time early warning module and a real-time display output module. The real-time early warning module outputs real-time early warnings based on the output signal of the multimodal information fusion module; the real-time display module is used to display the OCT images and endoscopic images output by the image preprocessing module in real time, and displays the target locations detected by the OCT image lesion detection network and the endoscopic image lesion detection network, respectively.
[0025] The beneficial effects of this invention are as follows:
[0026] 1. The present invention has a dichroic mirror tilted in the middle of the inner cavity of the capsule probe to separate the near-infrared light and visible light reflected from the same location and at the same time in the digestive tract and direct them to the OCT acquisition module and the endoscope imaging module respectively; thereby simultaneously acquiring OCT images and endoscope images of the same location in the digestive tract to facilitate comprehensive analysis of digestive tract lesions.
[0027] 2. This invention can simultaneously change the image acquisition direction of OCT and endoscopy simply by driving the dichroic mirror to rotate. Combined with the light-transmitting window surrounding the capsule probe, it enables OCT and endoscopy image acquisition within a 360° range around the capsule probe. The optical fiber, beam collimator, achromatic lens, liquid lens, and aspherical lens used to import near-infrared sample light signals are all fixed inside the capsule probe, allowing for adjustable acquisition direction while maintaining the system's compactness.
[0028] 3. The OCT acquisition module in this invention is located at the end of the capsule probe and does not occupy the space of the light-transmitting window area. This allows for full utilization of the internal space of the capsule probe, reduces the size of the capsule probe for dual-modal data acquisition, and lowers the risk of damage to the human digestive tract from the capsule probe.
[0029] 4. This invention employs a lightweight target detection network, enabling real-time detection while maintaining accuracy. Due to the small number of parameters, the system operates on portable devices, facilitating transportation and carrying. Furthermore, it integrates OCT endoscopy with traditional optical endoscopy to achieve real-time imaging of the digestive tract surface and tissue depth, overcoming the shortcomings of traditional optical endoscopy in missing lesions due to a lack of depth information.
[0030] 5. This invention combines optical coherence tomography (OCT) and liquid lensing techniques to achieve real-time, high-resolution zoom imaging of the internal structures of the digestive tract, providing clear and effective morphological structural information for monitoring digestive tract diseases. Simultaneously, this invention combines color image information of the digestive tract tissue surface with depth information from OCT images, utilizing deep learning technology to achieve intelligent detection and classification of multimodal digestive tract diseases, enhancing the screening and detection capabilities for lesions beneath the digestive tract epithelium and early-stage digestive tract diseases. Attached Figure Description
[0031] Figure 1 This is a structural block diagram of the dual-modal capsule endoscopy lesion automatic detection device of the present invention.
[0032] Figure 2 This is a schematic diagram of the internal structure of the dual-modal capsule probe in this invention.
[0033] Figure 3 This is a network structure diagram of the lesion detection module in this invention. Detailed Implementation
[0034] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, but the present invention is not limited to the following specific embodiments.
[0035] like Figure 1 and 2 As shown, a dual-modal imaging device for gastrointestinal optical coherence tomography and endoscopy includes an OCT external optical path module 1, a dual-modal capsule probe 2, a hybrid optical-electrical cable 14, an image preprocessing module 3, a lesion detection module 4, a multimodal information fusion module 5, and a display and early warning module 6. The hybrid optical-electrical cable 14 contains optical fiber cores and electrical signal transmission cores. The optical fiber cores in the hybrid optical-electrical cable 14 are connected to the optical fiber coupler in the OCT external optical path module 1.
[0036] The OCT extracorporeal optical path module 1 and the dual-modal capsule probe 2 are connected via optical fiber. The photoelectric signal collected by the photodetector in the OCT extracorporeal optical path module 1 and the endoscopic image collected by the dual-modal capsule probe 2 are transmitted to the image preprocessing module 3. The processed image output by the image preprocessing module 3 is transmitted to the lesion detection module 4; the feature signal output by the lesion detection module 4 is sent to the multimodal information fusion module 5; the multimodal information fusion module 5 is connected to the display and early warning module 6.
[0037] The OCT extracorporeal optical path module 1 is used to realize the transmission and conversion of OCT signals; the dual-modal capsule probe 2 is used for endoscopic imaging of the digestive tract and acquisition of OCT sample signals; the image preprocessing module 3 is used to preprocess the endoscopic and OCT images of the digestive tract to improve image quality and enhance the contrast of lesion areas; the lesion detection module 4 is used to detect, identify and classify lesions in the input OCT and endoscopic images; the multimodal information fusion module 5 is used to comprehensively analyze the lesion detection results based on OCT and endoscopic images; and the display and early warning module 6 is used to display the endoscopic examination process in real time and provide real-time early warning for detected lesions.
[0038] The OCT external optical path module 1 includes a near-infrared light source, a photodetector, a fiber optic coupler, and a reference arm optical path. The near-infrared light source is used for OCT imaging; the photodetector is used to convert the OCT interference light signal into an electrical signal and transmit it to the computer for processing; the fiber optic coupler is used to proportionally split the near-infrared light source into sample light and reference light and to cause interference between the reflected reference light and the sample light; the reference arm optical path is used to provide reference light and reflect the reference light back to the fiber optic coupler to interfere with the signal light.
[0039] The dual-modal capsule probe 2 includes a capsule shell 7, an illumination lamp 8, a dichroic mirror 12, an OCT acquisition module, an endoscopic imaging module, and a rotation drive assembly. A transparent window 21 surrounds the capsule shell 7 in the center to facilitate image acquisition from outside the capsule shell 7. The OCT acquisition module is used for depth-oriented signal acquisition of the digestive tract mucosa; the endoscopic imaging module is used for color imaging of the digestive tract surface; and the rotation drive assembly drives the OCT acquisition module and the endoscopic imaging module to rotate circumferentially for imaging.
[0040] The rotary drive assembly includes a micromotor 9, a motor bracket 10, and a mounting bracket 11. The micromotor 9 is fixed to one end of the inner cavity of the capsule shell 7 via the motor bracket 10; the output axis of the micromotor 9 coincides with the axis of the capsule shell 7; the mounting bracket 11 is fixed to the output shaft of the micromotor 9. A dichroic mirror 12 is fixed to the mounting bracket 11. The dichroic mirror 12 reflects near-infrared light and transmits visible light. The side of the dichroic mirror 12 forms a 45° angle with the axis of the output shaft of the micromotor 9.
[0041] The OCT acquisition module includes a lens holder 13, a beam collimator 15, an achromatic lens 16, a liquid lens 17, and an aspherical lens 18. The lens holder 13 is fixed to the other end of the inner cavity of the capsule shell 7; a central mounting hole is provided inside the lens holder 13. An optical-electric hybrid cable 14 contains an electrical signal transmission core and an optical fiber core. An external controller and power supply provide power and control signals to the micro-motor 9, the imaging sensor 20 in the endoscope imaging module, and the illumination lamp 8 via the electrical signal transmission core. Image data acquired by the imaging sensor 20 is transmitted to the image preprocessing module 3 within the controller via the electrical signal transmission core.
[0042] One end of the hybrid optical-electric cable 14 is fixed to the end of the capsule shell 7 where the OCT acquisition module is installed, and the end of the fiber core in the hybrid optical-electric cable 14 is aligned with the outer end of the central mounting hole. The central mounting hole houses a beam collimator 15, an achromatic lens 16, a liquid lens 17, and an aspherical lens 18, arranged sequentially from the outside to the inside, all with their axes coinciding with the axis of the capsule shell 7. The aspherical lens 18 is aligned with the dichroic mirror 12. The reflecting surface of the dichroic mirror 12 forms a 45° angle with the axis of the aspherical lens 18.
[0043] The OCT sampling light signal (near-infrared light) output from the OCT external optical path module 1 passes sequentially through the optoelectronic hybrid cable 14, beam collimator 15, achromatic lens 16, liquid lens 17, and aspherical lens 18 before being directed to the dichroic mirror 12. The OCT sampling light signal reflected from the dichroic mirror 12 passes through the transparent window on the side of the capsule shell 7 and illuminates the area to be measured. The light signal reflected from the area to be measured returns to the OCT external optical path module 1 along the original path and interferes with the reference light to achieve optical coherence tomography.
[0044] The endoscope imaging module includes an endoscope imaging lens 19 and an imaging sensor 20; the endoscope imaging lens 19 and the imaging sensor 20 are fixed to the side of the mounting bracket 11.
[0045] The mounting bracket 11 has a cutout in the area between the endoscope imaging lens 19 and the dichroic mirror 12, thus preventing the mounting bracket 11 from obstructing the endoscope imaging module's imaging of the digestive tract. The endoscope imaging lens 19 is aligned with the image detection area of the imaging sensor 20. The axis of the endoscope imaging lens 19 is perpendicular to the axis of the capsule shell. The endoscope imaging lens 19 faces the dichroic mirror 12; since the dichroic mirror 12 can transmit visible light, the visible light reflected from the measured area passes through the dichroic mirror 12 and enters the endoscope imaging module, allowing the imaging sensor 20 to acquire endoscopic images through the transparent window of the capsule shell. Therefore, the OCT acquisition module and the endoscope imaging module can simultaneously acquire OCT images and endoscopic images of the same location, respectively.
[0046] During use, the dual-modal capsule probe 2 enters the digestive tract from the mouth to the location to be measured. The near-infrared light source enters the fiber optic coupler along the optical fiber and is split into sample light and reference light. The sample light enters the beam collimator 15 located in the dual-modal capsule probe 2 along the optical fiber in the optoelectronic hybrid cable 14, is collimated, and then enters the achromatic lens 16. After the achromatic beam enters the liquid lens 17 for autofocusing, it enters the aspherical lens 18 for focusing. It is reflected by the dichroic mirror into the digestive tract area to be detected. At the same time, the backscattered light generated by the digestive tract tissue returns to the fiber optic coupler through the optical path and interferes with the reference light. The interference light signal is converted into an electrical signal by the photodetector, completing the OCT signal acquisition. Visible light emitted by the illumination lamp 8 illuminates the surface of the digestive tract tissue. The reflected light passes through the endoscope imaging lens 19 and enters the imaging sensor 20. The imaging sensor 20 converts the light signal into an electrical signal and outputs it to the image preprocessing module through the electrical signal transmission core wire in the optoelectronic hybrid cable 14 to complete the endoscope image acquisition. The micro motor 9 drives the output shaft to rotate, and at the same time drives the dichroic mirror 12, the endoscope imaging lens 19, and the imaging sensor 20 to rotate, realizing the circumferential acquisition of OCT signals and endoscope signals.
[0047] Image preprocessing module 3 performs preprocessing on OCT images, including converting OCT spectral data into OCT images, OCT image denoising, and OCT image enhancement. Image preprocessing module 3 also performs preprocessing on endoscopic images, including endoscopic image frame extraction, endoscopic image edge removal, endoscopic image denoising, endoscopic image enhancement, and image matching.
[0048] The lesion detection module 4 includes an OCT image lesion detection network and an endoscopy image lesion detection network. The OCT image lesion detection network uses transfer learning to identify lesion regions in preprocessed OCT images, obtaining lesion region detection results based on OCT images. The endoscopy image lesion detection network uses transfer learning to identify lesion regions in preprocessed endoscopy images, obtaining lesion region detection results based on endoscopy images. Both lightweight object detection networks are trained using their respective training sets. The training sets are obtained by image augmentation of initial lesion sample images. Image augmentation methods include traditional image augmentation methods (translation, rotation, flipping, scaling) and data augmentation based on generative adversarial networks (GANs). Specifically, traditional data augmentation methods are first used to augment the OCT image dataset and the endoscopy image dataset for the lesion images in the original dataset, and then a GAN is used to further augment the lesion data. During the image augmentation process, through continuous iteration of the generator and discriminator, generated images similar to real lesion images are obtained. Using GANs to augment data can effectively improve the impact of insufficient data samples on medical image data training. The dataset obtained from image augmentation is divided into a training set and a validation set for model training.
[0049] Both lightweight object detection networks are constructed using a two-stage lightweight object detector based on convolutional neural networks, enabling real-time lesion detection on ARM devices. The transfer learning strategy involves transferring the pre-trained backbone network weights from the COCO Natural dataset to the backbone network of the lesion detection network. Fine-tuning is then performed using a training set (amplified OCT or endoscopic images), employing stochastic gradient descent to reduce the discrepancy between predicted and actual values, and calculating gradient coefficients using backpropagation. Finally, a deep learning model suitable for the target network is obtained. Model testing involves using the trained deep learning network to detect lesions on input OCT and endoscopic images.
[0050] The structural diagram of lesion detection module 4 is as follows: Figure 3 As shown, both the OCT image lesion detection network and the endoscopic image lesion detection network in lesion detection module 4 employ lightweight target detection networks based on convolutional neural networks, including a feature extraction network, a context enhancement module, convolutional layers, a spatial attention module, and a fully connected layer connected sequentially. To address the imaging differences between OCT and endoscopic lesions, the feature extraction network is improved. To address the issue of varying lesion sizes in OCT images, a feature pyramid structure is used in the feature extraction network to fuse multi-scale features, enhancing the network's ability to detect lesions at different scales. To address the issue of similar imaging and varying lesion sizes between lesion areas and normal digestive tract tissue areas in endoscopic images, a convolutional block attention mechanism is added to the feature extraction network to strengthen key features of the lesion area. The context enhancement module utilizes semantic and contextual information from multiple scales, while the spatial attention module incorporates information from the region generation network to refine the feature distribution. During network training, the pre-trained weight parameters of the feature extraction network are first obtained by pre-training on the COCO dataset. Then, the pre-trained weights are transferred to the OCT image lesion detection branch and the endoscopic image lesion detection branch, respectively. Finally, the network is fine-tuned according to the target dataset. During training, stochastic gradient descent (SGD) is used for end-to-end training with a weight decay of 0.0001, a momentum of 0.9, and a batch size of 16.
[0051] The model's lesion detection performance was analyzed using Average Precision (AP), Mean Average Precision (mAP), Average Recall (AR), Intersection Over Union (IOU), and Frames Per Second (FPS) as evaluation metrics. Average Precision is the percentage of correctly identified lesions out of the total number of identified lesions. Mean Average Precision represents the average accuracy across all categories. Average Recall represents the percentage of correctly identified lesions out of the total number of lesions identified in the test set. IOU represents the area intersection ratio (IoU) between the predicted bounding box and the ground truth bounding box. Frames Per Second (FPS) represents the detector's processing speed.
[0052] The multimodal information fusion module 5 includes an anomaly diagnosis module and a multimodal lesion information quantification module. The anomaly diagnosis module analyzes lesion types by combining the output of the multimodal lesion detection network; the multimodal lesion information quantification module extracts information on lesion surface area, depth, and category from the analysis results, thereby helping doctors make a comprehensive judgment on the condition.
[0053] The display and early warning module 6 includes a real-time early warning module and a real-time display output module. The real-time early warning module is used to output real-time early warnings based on the comprehensive diagnostic results, while the real-time display module is used to display the OCT images and endoscopic images during the endoscopic examination process in real time, and to display the labeled lesions in real time.
[0054] The process of detecting gastrointestinal lesions using the dual-modal capsule endoscopy lesion detection device of the above system is as follows:
[0055] Step 1: The dual-modal capsule probe is inserted into the digestive tract to be tested. The near-infrared light source in the OCT extracorporeal optical path module emits near-infrared light to the fiber optic coupler. The sample light enters the OCT acquisition module. After being reflected by the sample, the signal light returns to the fiber optic coupler and interferes with the reference light. The interference light is converted into an electrical signal by the photodetector.
[0056] Step 2: The illumination lamp in the dual-modal capsule probe emits visible light to the surface of the digestive tract and then reflects it into the endoscope imaging module. After the endoscope image is acquired, it is transmitted to the computer via a hybrid optical-electric cable.
[0057] Step 3: The endoscopic image in the dual-modal capsule probe 2 and the OCT signal in the OCT extracorporeal optical path module 1 are transmitted to the computer through the electrical signal transmission core wire and the data acquisition card, respectively. The computer's image preprocessing module 3 performs preprocessing to achieve signal conversion, image denoising, image enhancement, and image matching.
[0058] During preprocessing, OCT data is converted into OCT images via Fourier transform. OCT image denoising employs Gaussian blurring, median filtering, and mean filtering, and anisotropic diffusion filtering is introduced to achieve image denoising and enhancement. Gamma curves are used to adjust image contrast and brightness. Endoscopic images are denoised using wavelet thresholding, and gamma curves are used to adjust image contrast and brightness. Image matching is performed one-to-one between the two modal imaging information based on the imaging time axis, compensating for image misalignment caused by inconsistent acquisition rates between the two modalities, and ensuring that the multimodal images input to the lesion detection module represent the same digestive tract region.
[0059] Step 4: The image preprocessing module 3 inputs the preprocessed digestive tract OCT images and endoscopic images into the lightweight target detection network of the lesion detection module 4, thereby performing lesion location detection and classification.
[0060] Step 5: The lesion detection module 4 inputs the detection results into the multimodal information quantification module 5. After abnormal diagnosis of the lesion information, the module outputs auxiliary diagnostic information such as lesion area, lesion depth, and lesion type based on the comprehensive detection results.
[0061] The multimodal information fusion module 5 first determines whether the OCT image detects a lesion. If no lesion is detected, it outputs information to the normal or abnormal module 1. If a lesion is detected, it outputs lesion information to the abnormal module 2 or abnormal module 3. Next, it determines whether the endoscopic image detects a lesion. If no lesion is detected, it outputs information to the normal or abnormal module 2. If a lesion is detected, it outputs lesion information to the abnormal module 1 or abnormal module 3. For the normal module, it directly outputs the detected image to the display and warning module 6 for real-time display. For the output results of abnormal module 1, abnormal module 2, and abnormal module 3, they enter the lesion information quantification module, which makes a comprehensive judgment based on the lesion type, lesion surface area, and lesion depth information.
[0062] Step 6: The output results of the multimodal information fusion module 5 are input into the display and early warning module 6 to provide real-time early warnings and display the lesions.
[0063] The real-time early warning output converts the quantification results of lesion information in the multimodal information fusion module 5 into text information (lesion type, lesion surface area, lesion depth) and outputs and issues an early warning; the real-time display output displays the OCT images and endoscopic images during the examination process on the display screen and annotates the detected lesions in real time.
[0064] Matters not covered in this invention are common knowledge.
[0065] The above embodiments are only for illustrating the technical concept and features of the present invention, and are intended to enable those skilled in the art to understand the content of the present invention and implement it accordingly. They should not be construed as limiting the scope of protection of the present invention. All equivalent changes or modifications made in accordance with the spirit and essence of the present invention should be covered within the scope of protection of the present invention.
Claims
1. A digestive tract detection system, characterized in that: It includes an image preprocessing module (3), a lesion detection module (4), and a dual-modal imaging device for optical coherence tomography and endoscopy of the digestive tract; the dual-modal imaging device for optical coherence tomography and endoscopy of the digestive tract includes an OCT extracorporeal optical path module (1) and a dual-modal capsule probe (2); the dual-modal capsule probe (2) is connected to the OCT extracorporeal optical path module (1) through a signal transmission line with an internal fiber optic core. The dual-modal capsule probe (2) includes a capsule shell (7), a dichroic mirror (12), an OCT acquisition module, an endoscope imaging module, and a rotation drive assembly; a transparent window surrounding the capsule shell (7) is provided in the middle of the capsule shell (7); the dichroic mirror (12) is installed inside the capsule shell (7) and is driven to rotate by the rotation drive assembly; the rotation axis of the dichroic mirror (12) is parallel to the central axis of the capsule shell (7); the side of the dichroic mirror (12) forms an angle of 30 to 60° with the central axis of the capsule shell (7); the dichroic mirror (12) reflects near-infrared light and transmits visible light; The OCT acquisition module includes a lens bracket (13), a beam collimator (15), an achromatic lens (16), a liquid lens (17), and an aspherical lens (18); the lens bracket (13) is fixed to the end of the inner cavity of the capsule shell (7); the beam collimator (15), the achromatic lens (16), the liquid lens (17), and the aspherical lens (18) are arranged in the central mounting hole of the lens bracket (13); the fiber core of the signal transmission line is connected to the central mounting hole of the lens bracket (13); the near-infrared light input from the fiber core to the dual-mode capsule probe (2) passes through the beam collimator (15), the achromatic lens (16), the liquid lens (17), and the aspherical lens (18) in sequence, and is reflected by the dichroic mirror (12) to the transparent window of the capsule shell (7); The rotary drive assembly includes a micro motor (9), a motor bracket (10), and a mounting bracket (11); the micro motor (9) is fixed to the end of the inner cavity of the capsule shell (7) via the motor bracket (10); the output axis of the micro motor (9) coincides with the axis of the capsule shell (7); the mounting bracket (11) is fixed on the output shaft of the micro motor (9); and the dichroic mirror (12) is fixed on the mounting bracket (11). The endoscope imaging module is fixed to the side of the mounting bracket (11); and the endoscope imaging module faces the dichroic mirror (12); the orientation of the endoscope imaging module is consistent with the direction of the near-infrared light emitted by the OCT acquisition module reflected on the dichroic mirror (12); the mounting bracket (11) has a hollowed-out area between the endoscope imaging module and the dichroic mirror; During the operation, near-infrared light reflection signals and visible light reflection signals at the same location and time in the digestive tract are reflected towards the dichroic mirror (12); after being reflected by the dichroic mirror (12), the near-infrared light reflection signal is transmitted to the OCT extracorporeal optical path module (1) via the OCT acquisition module; after being transmitted by the dichroic mirror (12), the visible light reflection signal is irradiated to the endoscope imaging module. The dual-modal capsule probe (2) outputs OCT spectral data and endoscopic images and transmits them to the image preprocessing module (3); the image preprocessing module (3) transmits the preprocessed OCT images and endoscopic images to the lesion detection module (4). The digestive tract detection system also includes a multimodal information fusion module (5) and a display and early warning module (6); the image preprocessing module (3) preprocesses the OCT spectral data by converting the OCT spectral data into an OCT image, denoising the OCT image, and enhancing the OCT image; the image preprocessing module (3) preprocesses the endoscope image by extracting the image frame, removing the image edge, denoising the image, enhancing the image, and matching the image. The lesion detection module (4) includes an OCT image lesion detection network and an endoscopy image lesion detection network. Both the OCT image lesion detection network and the endoscopy image lesion detection network adopt a lightweight target detection network based on convolutional neural networks, including a feature extraction network, a context enhancement module, convolutional layers, a spatial attention module, and a fully connected layer. The feature extraction network of the OCT image lesion detection network adopts a feature pyramid structure to fuse multi-scale features, thereby enhancing the network's ability to detect lesions at different scales. The feature extraction network of the endoscopy image lesion detection network adopts a convolutional block attention mechanism to strengthen the key features of the lesion area. The lesion detection module (4) transmits the feature signals output by the OCT image lesion detection network and the endoscope image lesion detection network to the multimodal information fusion module (5); the multimodal information fusion module (5) is used to combine the feature signals output by the OCT image lesion detection network and the endoscope image lesion detection network to provide assistance to doctors in judging the condition; The aforementioned display and warning module (6) includes a real-time warning module and a real-time display output module; The real-time early warning module outputs real-time early warning based on the output signal of the multimodal information fusion module (5); The real-time display module is used to display the OCT images and endoscope images output by the image preprocessing module (3) in real time, and to display the target locations detected by the OCT image lesion detection network and the endoscope image lesion detection network, respectively.
2. The digestive tract detection system according to claim 1, characterized in that: Each time the dual-modal capsule probe (2) reaches a collection position, the dichroic mirror (12) rotates 360°; the dichroic mirror (12) pauses once every time it rotates a preset angle, and the OCT external optical path module (1) and the endoscope imaging module collect data when the dichroic mirror (12) pauses, so as to obtain OCT spectral data and endoscopic images of different orientations within a 360° range of the collection position.
3. The digestive tract detection system according to claim 1, characterized in that: The signal transmission line is a hybrid optical-electric cable (14); the hybrid optical-electric cable (14) is provided with optical fiber core wire and electrical signal transmission core wire; the optical fiber core wire in the hybrid optical-electric cable (14) is connected to the optical fiber coupler in the OCT external optical path module (1).
4. The digestive tract detection system according to claim 1, characterized in that: The OCT in vitro optical path module (1) includes a near-infrared light source, a photodetector, a fiber coupler, and a reference arm optical path; the near-infrared light source is used for OCT tomography; the photodetector is used to convert the OCT interference light signal into an electrical signal and transmit it to a computer for processing; the fiber coupler is used to divide the near-infrared light source into sample light and reference light proportionally and to make the reflected reference light and sample light interfere; The reference arm optical path is used to reflect the reference light back to the fiber coupler; the sample light output from the fiber coupler is transmitted to the OCT acquisition module of the dual-mode capsule probe (2).
5. The digestive tract detection system according to claim 1, characterized in that: The capsule shell (7) is equipped with an illumination lamp (8); the illumination lamp (8), the rotation drive assembly and the endoscope imaging module are all powered and controlled by the electrical signal transmission core wire in the signal transmission line; the image data detected by the endoscope imaging module is output through the electrical signal transmission core wire.
6. The digestive tract detection system according to claim 1, characterized in that: The side of the dichroic mirror (12) forms a 45° angle with the central axis of the capsule shell (7); the orientation of the endoscope imaging module is perpendicular to the central axis of the capsule shell.
7. The digestive tract detection system according to claim 1, characterized in that: The endoscopic imaging module includes an endoscopic imaging lens (19) and an imaging sensor (20); both the endoscopic imaging lens (19) and the imaging sensor (20) are fixed to the dichroic mirror (12); the endoscopic imaging lens (19) is aligned with the image detection area of the imaging sensor (20); the endoscopic imaging lens (19) faces the dichroic mirror (12).