Endoscope imaging fusion enhancement method and device

Abstract

According to the endoscope imaging fusion enhanced imaging method and device, imaging characteristics of white light (an overall structure background), red light (a deep blood vessel), green light (a high-contrast capillary vessel) and blue light (an enhanced surface microstructure) are comprehensively utilized, digestive tract mucosa can be analyzed in a layered mode, and comprehensive diagnosis and early focus recognition from the surface layer to the deep layer are achieved; during image fusion, adaptive weight configuration based on focus types is supported by adopting a weight dynamic adjustment mechanism, so that the quality of the finally fused image is influenced, and therefore, a neural network architecture is adopted to train each channel weight, and a more reasonable weight value can be obtained.

Description

Technical Field

[0001] The present invention relates to the technical field of endoscopic imaging, and in particular to an endoscopic imaging fusion enhancement method and device. Background Art

[0002] In the field of modern medical imaging, endoscopic technology plays an important role in disease diagnosis and treatment. Although endoscopic technology is indispensable in medical imaging diagnosis and treatment, traditional imaging methods are limited to the surface or a single information dimension, making it difficult to meet the needs of deep analysis of tissue structure and lesions. Multispectral imaging technology provides a new way to obtain information on tissue optical properties, hemodynamics, and biomolecule distribution by collecting data from different spectral channels. However, to truly interpret the complexity of tissues, it is necessary to go beyond a single multispectral image and achieve the coordinated fusion of multi-channel spectral information with other modal data. The current multispectral fusion solution in endoscopic applications still has technical bottlenecks, and its insufficient imaging quality (such as resolution and contrast) remains a problem that needs to be solved urgently.

[0003] Despite its significant advantages, endoscopic multispectral imaging still has the following key drawbacks in terms of technical implementation and clinical application, which restrict its further development and popularization:

[0004] 1. Hardware and system design flaws

[0005] 1. Limited spatial resolution

[0006] Multispectral imaging requires time-sharing and channel-by-channel acquisition of images in different wavelengths, shortening the exposure time of a single frame and reducing the signal-to-noise ratio. Narrowband filtering further reduces light throughput, significantly reducing resolution, especially when imaging deep tissue (e.g., in the red wavelength band).

[0007] 2. Lack of real-time performance

[0008] Traditional time-sharing scanning (switching filters or light sources) results in low frame rates (often <5 fps), making it difficult to capture dynamic changes such as digestive tract motility and prone to motion artifacts. Parallel imaging systems (such as beam splitters) require complex optical designs, increase the diameter of the endoscope, and reduce clinical operability.

[0009] 3. High system complexity and cost

[0010] Multispectral light sources (e.g., tunable lasers, LED arrays) and high-speed, high-sensitivity detectors are expensive. Miniaturization challenges: Integrating a multispectral module into the slender endoscope tip requires addressing heat dissipation, power consumption, and mechanical stability.

[0011] 2. Image Processing and Analysis Bottlenecks

[0012] 1. Multispectral data fusion is difficult

[0013] Images in different wavelengths are spatially misaligned (due to time-sharing acquisition or optical distortion), requiring high-precision registration algorithms. However, tissue deformation and motion make registration errors difficult to avoid. There is a lack of universal fusion models: White light, narrowband spectrum, and fluorescence images have significantly different physical meanings, and simple superposition can lead to redundant or contradictory information.

[0014] 2. Insufficient ability to decouple organizational characteristics

[0015] Spectral aliasing: The absorption spectra of biomolecules such as hemoglobin, collagen, and lipids overlap, resulting in impure feature extraction. Affected by tissue scattering characteristics: Deep vascular information is easily masked by surface structural noise, making it difficult for traditional algorithms to effectively separate tomographic information.

[0016] 3. Lack of standardization in quantitative analysis

[0017] Variations in light source power, tissue distance, and angle lead to unstable spectral intensity, making it difficult to establish quantitative diagnostic standards across devices. Lack of large-scale annotated datasets: The multispectral feature library for lesions is not yet complete, limiting the generalization capabilities of AI models. Summary of the Invention

[0018] In view of this, the present invention provides an endoscopic imaging fusion enhancement method and device.

[0019] The specific technical solutions adopted in the present invention are as follows:

[0020] An endoscopic imaging fusion enhancement method, comprising:

[0021] Step 1: Acquire an endoscopic multispectral image;

[0022] Step 2: Preprocess the multispectral image;

[0023] Step 3: Multispectral image fusion, specifically:

[0024] Step 1: 3D construction of multi-channel data;

[0025] Step 2: Channel weight allocation;

[0026] Step 3: Pixel-level weighted fusion.

[0027] Preferably, step 1 specifically includes: successively connecting illumination light sources of multiple bands, including white light, green light, red light, and blue light, to the light cone port of the endoscope, and obtaining multispectral images of the endoscope under illumination of the above light sources of each band.

[0028] Preferably, the step 2 includes:

[0029] Low-pass filtering is used to filter the endoscope multispectral image to eliminate the background noise of the endoscope optical system and eliminate endoscope motion artifacts; the reflection component and illumination component of the spectral image of each band are separated to eliminate non-uniform illumination.

[0030] Preferably, the step 2 includes: aligning the multi-band spectral images to eliminate the slight displacement caused by the shooting time difference; wherein the alignment is achieved by using the SIFT algorithm.

[0031] Preferably, in step 3, Step 1 specifically includes:

[0032] After image alignment and brightness calibration of the spectral images of each band, the standard spectral image of the channel is obtained. The images are stacked according to the preset channel order (such as increasing wavelength), thereby obtaining a data cube with dimensions of H×W×C, where H represents the image height, W represents the image width, and C represents the number of spectral channels.

[0033] Preferably, in step 3, Step 2 specifically includes: using a neural network architecture to train the weights of each channel to obtain more reasonable weight values.

[0034] Preferably, the neural network architecture includes:

[0035] Input layer, the input is the H×W×C data cube of each channel;

[0036] Feature extraction layer, which extracts features from the H×W×C data cube of each channel;

[0037] The channel attention module uses a SE-Net variant to perform attention processing on the feature data of each channel and output the weight of each channel;

[0038] The weighted fusion layer fuses the images of each channel according to the predicted weights;

[0039] Output layer, outputs a fused image with a dimension of H×W;

[0040] The neural network architecture is back-propagated to update weights based on a loss function. The loss function is a weighted sum of three items: image structure loss, edge loss, and spectral constraint loss. Based on the trained neural network architecture, more accurate channel weight values ​​can be obtained.

[0041] A device for implementing the endoscopic imaging fusion enhancement method according to claim 1, comprising:

[0042] An image acquisition module, used for acquiring endoscopic multispectral images;

[0043] Image processing module, used for preprocessing multispectral images, including low-pass filtering, eliminating non-uniform illumination, and image alignment;

[0044] The image fusion module is used to implement the multispectral image fusion process, which mainly includes multi-channel data stereo construction, channel weight allocation and pixel-level weighted fusion.

[0045] The present invention has the following beneficial effects:

[0046] The present invention provides an endoscopic imaging fusion enhanced imaging method and device, which comprehensively utilizes the imaging characteristics of white light (overall structural background), red light (deep blood vessels), green light (high-contrast capillaries) and blue light (enhanced surface microstructure), and can analyze the digestive tract mucosa in layers, realizing comprehensive diagnosis and early lesion identification from the surface to the deep layer.

[0047] During image fusion, a dynamic weight adjustment mechanism is used to support adaptive weight configuration based on lesion type. The choice of weights will affect the quality of the final fused image. Therefore, the present invention uses a neural network architecture to train the weights of each channel in order to obtain more reasonable weight values.

[0048] Alignment is achieved using the SIFT algorithm, which includes scale-space extremum detection, precise keypoint location, orientation determination, and keypoint descriptor generation. The algorithm can find keypoints (feature points) at different scales and calculate their orientation. SIFT finds keypoints that are highly prominent and remain unchanged by factors such as lighting, affine transformations, and noise. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 This is a structural diagram of an endoscope in the prior art.

[0050] Figure 2 Flow chart of the method of the present invention. DETAILED DESCRIPTION

[0051] The present invention provides an endoscope imaging fusion enhancement method and device, such as Figure 2 As shown, the following steps are included:

[0052] Step 1: Obtain endoscopic multispectral images:

[0053] Illumination light sources of multiple wavelength bands are successively connected to the light cone of the endoscope. The wavelength bands of the present invention include white light, green light, red light, and blue light, and multispectral images of the endoscope under illumination of the above wavelength bands are obtained respectively.

[0054] Among them, white light, green light, red light and blue light use LED cold light source;

[0055] The white light used in the present invention is a composite band of visible light, which can illuminate the surface of the mucosa and structures at a certain depth, presenting the natural color of the tissue. The resulting image has good overall morphological visibility and is suitable for basic observation and navigation; red light, due to its long wavelength and strong tissue penetration ability, can penetrate deeper tissues of 1mm-2mm. Green light uses a band of 500nm-600nm. Since this band covers the hemoglobin absorption peak of 540nm-580nm, the strong absorption makes the blood vessels clearly visible, so the shallow capillary network is presented with high contrast, enhancing the visibility of blood vessels in the tissue; 400nm-500nm blue light, due to its short wavelength and strong scattering, is suitable for the surface tissue of the target, and the resulting image is rich in details.

[0056] By comprehensively utilizing the imaging characteristics of white light (overall structural background), red light (deep blood vessels), green light (high-contrast capillaries) and blue light (enhanced surface microstructure), the digestive tract mucosa can be analyzed in layers, achieving comprehensive diagnosis and early lesion identification from the surface to the deep layer.

[0057] Step 2: Preprocess the multispectral image, including:

[0058] Low-pass filtering is used to filter the endoscope multispectral image to eliminate the background noise of the endoscope optical system and eliminate endoscope motion artifacts; the reflection component and illumination component of the spectral image of each band are separated to eliminate non-uniform illumination.

[0059] Multi-band spectral images are aligned to eliminate minor shifts caused by time differences in capture. SIFT can be used for alignment. SIFT involves detecting extreme values ​​in scale space, accurately locating key points, determining their orientation, and generating key point descriptors. The algorithm searches for key points (feature points) across different scales and calculates their orientation. SIFT identifies key points that are prominent and invariant to factors such as lighting, affine transformations, and noise. These include corner points, edge points, two points in a dark area, and dark points in a bright area.

[0060] In addition, for spectral images in the red band, since they detect deep tissue images, non-local means denoising is used to preserve vascular texture. For spectral images in the blue band, since they detect surface mucosal images of the target, wavelet threshold denoising is used to preserve high-frequency details.

[0061] During endoscope illumination, the captured image may have uneven distribution due to the influence of non-uniform illumination. Therefore, the method adopted in the present invention is to separate the reflection component of each channel and eliminate the illumination component based on the Retinex theory.

[0062] Step 3: Multispectral image fusion process

[0063] Step 1: Multi-channel data 3D construction

[0064] After image alignment and brightness calibration of the spectral images of each band (channel), the standard spectral image of the channel is obtained. The images are stacked in a preset channel order (such as increasing wavelength) to obtain a data cube with dimensions of H×W×C, where H represents the image height, W represents the image width, and C represents the number of spectral channels.

[0065] Step 2: Channel weight allocation

[0066] Weights are assigned to each channel image based on the importance of each band's impact on the imaging effect. In this embodiment, white light can be weighted with a W1 of 0.2-0.5 based on surface structure clarity requirements. Red light images can be weighted with a W2 of 0.4-0.6 based on deep vascular sensitivity and response intensity requirements for specific biomarkers. Green light provides high capillary contrast, and blue light enhances surface microstructures, with weights W3 and W4 both of 0.1-0.4; W1+W2+W3+W4=1.

[0067] In the implementation of this invention, the weights are dynamically adjusted to support adaptive weight configuration based on lesion type. The choice of weights will affect the quality of the final fused image. Therefore, the present invention uses a neural network architecture to train the weights of each channel in order to obtain more reasonable weight values. The specific scheme is as follows:

[0068] The neural network architecture includes:

[0069] Input layer, the input is the H×W×C data cube of each channel;

[0070] Feature extraction layer, which extracts features from the H×W×C data cube of each channel;

[0071] The channel attention module uses a SE-Net variant to perform attention processing on the feature data of each channel and output the weight of each channel;

[0072] The weighted fusion layer fuses the images of each channel according to the predicted weights;

[0073] Output layer, outputs a fused image with a dimension of H×W.

[0074] The neural network architecture is back-propagated to update weights based on a loss function. This loss function is a weighted sum of three image loss terms: structural loss, edge loss, and spectral constraint loss. Based on the trained neural network architecture, more accurate channel weights can be obtained.

[0075] This solution transforms channel weights from static parameters to dynamic sensors, enabling the fusion process to understand clinical scenarios and providing the core driving force for the next generation of intelligent endoscopy systems.

[0076] Step 3: Pixel-level weighted fusion

[0077] Using weights, perform weighted fusion on the images of each channel and output a fused multi-dimensional pixel matrix:

[0078]

[0079] I k (x,y) represents the k-th channel image; w k Represents the normalized weight of the k-th channel image.

[0080] Clinical Example: Gastrointestinal Disease Diagnosis

[0081] System Configuration:

[0082] |Channel|Wavelength Range|Target Biometrics|

[0083] | Channel 1 | White light | Mucosal surface morphology |

[0084] |Channel 2| Red light | Deep vascular network |

[0085] |Channel 3| Green light | Capillary network|

[0086] |Channel 4| Blue light| Enhanced surface microstructure|

[0087] Execution process:

[0088] 1. Preprocessing

[0089] Register three-channel images (SIFT feature matching);

[0090] Histogram equalization eliminates lighting differences;

[0091] 2. Weight distribution

[0092] 'visible light': 0.3;

[0093] 'Red light': 0.4;

[0094] 'greenlight': 0.2;

[0095] 'bluelight': 0.1;

[0096] 3. Diagnostic gain:

[0097] The contrast of deep blood vessels increased by 2.8 times; the early tumor detection rate increased from 68% to 91%.

[0098] 4. Summary of technical advantages:

[0099] 1. Information integrity:

[0100] Preserve both surface morphology (visible light) and functional information (infrared light)

[0101] 2. Scalability: Supports plug-and-play integration of new UV / fluorescence channels

[0102] 3. Clinical adaptability:

[0103] Quickly adapt to different diagnostic scenarios (such as gastric cancer vs. colon polyps) through weight configuration

[0104] The process of this embodiment has been integrated into a multimodal endoscopy intelligent platform, and the fusion time is <50ms / frame (NVIDIA Jetson AGX Orin).

[0105] Based on the above-mentioned endoscopic multispectral fusion enhanced imaging method, the present invention also provides an endoscopic multispectral fusion enhanced imaging device, comprising:

[0106] An image acquisition module, used for acquiring endoscopic multispectral images;

[0107] The image processing module is used to pre-process multispectral images, including low-pass filtering, elimination of non-uniform illumination, image alignment, etc.

[0108] The image fusion module is used to implement the multispectral image fusion process, which mainly includes multi-channel data stereo construction, channel weight allocation and pixel-level weighted fusion.

[0109] The above specific embodiments merely illustrate the design principles of the present invention. The shapes and names of the components described herein may vary and are not limiting. Therefore, those skilled in the art may modify or substitute equivalents for the technical solutions described in the above embodiments. Such modifications and substitutions, without departing from the inventive spirit and technical solutions of the present invention, shall fall within the scope of protection of the present invention.

Claims

1. A method for endoscopic imaging fusion enhancement, characterized in that: include: Step 1: Acquire an endoscopic multispectral image; Step 2: Preprocess the multispectral image; Step 3: Multispectral image fusion, specifically: Step 1: 3D construction of multi-channel data; Step 2: Channel weight allocation; Step 3: Pixel-level weighted fusion.

2. The endoscopic imaging fusion enhancement method according to claim 1, characterized in that: The step 1 specifically includes: successively connecting illumination light sources of multiple wavelength bands, including white light, green light, red light, and blue light, to the light cone port of the endoscope, and obtaining multispectral images of the endoscope under illumination of the above wavelength band light sources respectively.

3. The endoscopic imaging fusion enhancement method according to claim 1, characterized in that: The step 2 includes: Low-pass filtering is used to filter the endoscope multispectral image to eliminate the background noise of the endoscope optical system and eliminate endoscope motion artifacts; the reflection component and illumination component of the spectral image of each band are separated to eliminate non-uniform illumination.

4. The endoscopic imaging fusion enhancement method according to claim 1, characterized in that: The step 2 includes: aligning the multi-band spectral images to eliminate the slight displacement caused by the shooting time difference; wherein the alignment is achieved by using the SIFT algorithm.

5. The endoscopic imaging fusion enhancement method according to claim 1, characterized in that: In the step 3, Step 1 specifically includes: After image alignment and brightness calibration of the spectral images of each band, the standard spectral image of the channel is obtained. The images are stacked according to the preset channel order (such as increasing wavelength), thereby obtaining a data cube with dimensions of H×W×C, where H represents the image height, W represents the image width, and C represents the number of spectral channels.

6. The endoscopic imaging fusion enhancement method according to claim 1, characterized in that: In the step 3, Step 2 specifically includes: using a neural network architecture to train the weights of each channel to obtain more reasonable weight values.

7. The endoscopic imaging fusion enhancement method according to claim 6, characterized in that: The neural network architecture includes: Input layer, the input is the H×W×C data cube of each channel; Feature extraction layer, which extracts features from the H×W×C data cube of each channel; The channel attention module uses a SE-Net variant to perform attention processing on the feature data of each channel and output the weight of each channel; The weighted fusion layer fuses the images of each channel according to the predicted weights; Output layer, outputs a fused image with a dimension of H×W; The neural network architecture is back-propagated to update weights based on a loss function. The loss function is a weighted sum of three items: image structure loss, edge loss, and spectral constraint loss. Based on the trained neural network architecture, more accurate channel weight values ​​can be obtained.

8. A device for implementing the endoscopic imaging fusion enhancement method according to claim 1, characterized in that: include: An image acquisition module, used for acquiring endoscopic multispectral images; Image processing module, used for preprocessing multispectral images, including low-pass filtering, eliminating non-uniform illumination, and image alignment; The image fusion module is used to implement the multispectral image fusion process, which mainly includes multi-channel data stereo construction, channel weight allocation and pixel-level weighted fusion.

Images