Medical endoscope 2D video real-time 3D conversion method based on deep learning

Abstract

The invention provides a medical endoscope 2D video real-time 3D conversion method based on deep learning, and belongs to the technical field of medical image processing. The medical endoscope 2D video real-time 3D conversion method based on deep learning comprises the following steps: preprocessing an original 2D endoscope video, including adaptive histogram equalization processing and specular reflection suppression; and carrying out depth estimation on the preprocessed video frame through a lightweight depth estimation network EndoDepthNet. And processing the continuous multi-frame video by using a motion recovery structure algorithm. And generating a binocular parallax image according to the optimized depth map, carrying out sub-pixel-level time sequence smoothing processing, and outputting a real-time 3D stereoscopic video. According to a deep learning model for endoscope scene optimization, multi-modal geometric constraint and hardware parallel optimization, the defects of a traditional method in precision, real-time performance and anti-interference capability are overcome.

Description

Technical Field

[0001] The present invention relates to the field of grid node technology, and more specifically, to a method for real-time 3D conversion of medical endoscope 2D videos based on deep learning, which is suitable for the real-time visualization of tissue three-dimensional structures in minimally invasive surgeries such as laparoscopy and arthroscopy. Background Art

[0002] Traditional 3D endoscopes require dual optical modules or Time-of-Flight sensors, which increase the device diameter, increase cost, and are susceptible to interference from intraoperative smoke and liquid. Monocular 2D-to-3D conversion solutions rely on traditional stereo matching algorithms, which can lead to significant depth estimation errors in areas lacking tissue texture (such as smooth organ surfaces) and cannot meet real-time requirements (>30fps).

[0003] Current technical pain points include interference from motion blur, specular highlights, and tissue deformation in endoscopic images. Common depth estimation models (such as MiDaS) suffer from reduced accuracy in these scenarios. Real-time 3D reconstruction requires processing a single frame in less than 33ms, but existing algorithms struggle to achieve a balance between speed and accuracy.

[0004] How to invent a real-time 3D conversion method for medical endoscope 2D video based on deep learning to improve these problems has become an urgent problem to be solved by technicians in this field. Summary of the Invention

[0005] To overcome the above shortcomings, the present invention provides a real-time 3D conversion method for medical endoscope 2D videos based on deep learning, aiming to improve the problems of traditional 3D endoscope equipment, such as large diameter, high cost, significant depth estimation error, and inability to meet real-time requirements.

[0006] The present invention is achieved in that: The present invention provides a method for real-time 3D conversion of medical endoscope 2D video based on deep learning, comprising the following steps: S1: Preprocessing of the original 2D endoscope video, including adaptive histogram equalization and specular reflection suppression; S2: The lightweight depth estimation network EndoDepthNet is used to estimate the depth of the preprocessed video frame and generate a single-frame depth map; S3: Use the structure-from-motion algorithm to process multiple frames of video and optimize the depth map through multi-frame fusion and non-rigid deformation compensation; S4: Generate binocular disparity images based on the optimized depth map, perform sub-pixel temporal smoothing, and output real-time 3D stereo video.

[0007] Preferably, in the preprocessing step: adaptive histogram equalization is used to enhance the contrast of blood vessel texture in the video image; and specular reflection areas are identified and suppressed by an instrument semantic segmentation algorithm based on a U-Net model.

[0008] Preferably, the lightweight depth estimation network EndoDepthNet adopts a multi-task encoding and decoding architecture, including: a main branch is used to predict the depth map, and the loss function adopts SILog loss; an auxiliary branch jointly predicts the surface normal map and the optical flow field, and improves the depth estimation accuracy of weak texture areas through geometric consistency constraints; the total loss function is L_total=0.7×L_depth+0.2×L_normal+0.1×L_flow.

[0009] Preferably, the training data of EndoDepthNet includes: 300 synthetic surgical videos (UnityEndo dataset) generated based on the Unity engine, simulating intraoperative smoke, blood adhesion and instrument occlusion scenes; and annotated data of 50 real cholecystectomy videos, which improve the model's generalization ability for real scenes through domain adaptation training.

[0010] Preferably, the EndoDepthNet supports INT8 quantization deployment and achieves a single-frame inference time of ≤25ms on the NVIDIA Jetson AGX Orin platform.

[0011] Preferably, the motion recovery structure algorithm includes: using the ORB-SLAM3 algorithm to extract the camera pose and instrument motion information of 5 adjacent frames of video, and constructing a single-frame depth map optimized by extreme geometric constraints; predicting the tissue deformation field through the LSTM module, and dynamically compensating for the non-rigid deformation error of the organ surface caused by breathing or instrument touch.

[0012] Preferably, the binocular disparity generation step includes: generating left and right eye views based on the depth map, and using the CR-Stereo algorithm to fill the disparity information of the occluded area; the disparity range of the left and right eye views is constrained by the prior parameters of the surgical scene to avoid stereoscopic distortion.

[0013] Preferably, the sub-pixel temporal smoothing processing includes: using the Motion-Flow optical flow field on the GPU side to guide interpolation between video frames to eliminate jitter artifacts in the 3D reconstruction process; using a double buffering mechanism and CUDA stream parallel technology to achieve pipeline operations of depth estimation and 3D rendering, ensuring end-to-end delay ≤28ms.

[0014] Preferably, the hardware architecture of the system includes: a 1080p@60fps 2D endoscopic camera connected to an embedded processing unit via an HDMI / USB 3.0 interface; the embedded processing unit adopts the NVIDIA Jetson AGX Orin platform, equipped with 16GB of video memory, and supports real-time parallel computing.

[0015] Preferably, the method is suitable for minimally invasive surgical scenarios such as laparoscopy and arthroscopy. It has been verified by in vitro pig liver puncture experiments that the root mean square error (RMSE) of depth estimation is ≤1.2mm, and the 3D perception accuracy of surgeons' blind tests is ≥93%.

[0016] EndoDepthNet, a lightweight depth estimation network, features a multi-task encoder-decoder architecture. The main branch extracts features based on a ResNet-18 encoder, while the decoder uses deconvolutional layers to output a single-channel depth map. The SILog loss function is used, focusing on relative depth accuracy. The auxiliary branch concurrently predicts the surface normal map (L_normal loss) and the optical flow field (L_flow loss). Geometric consistency constraints (such as depth-normal angle constraints and optical flow-depth motion consistency) are employed to improve depth estimation accuracy in weakly textured regions (such as smooth organs). Data augmentation and domain adaptation: Synthetic data: A virtual surgical scene was constructed using the Unity engine. 300 synthetic videos containing smoke (0-50% concentration), blood deposition (10-40% coverage), and instrument occlusion (15-30% occlusion rate) were generated, and the ground-truth depth values ​​were annotated. Real-world data: Fifty real-world cholecystectomy videos were collected and domain-adapted using CycleGAN to reduce the distributional differences between the synthetic and real-world data. Hardware adaptation: Model quantization: TensorRT is used for INT8 quantization, and the model parameters are compressed to 12MB. An inference speed of 25ms / frame is achieved on Jetson AGX Orin, meeting real-time requirements.

[0017] Real-time 3D reconstruction engine: Multi-frame motion recovery optimization: Pose estimation based on ORB-SLAM3: Extracts ORB feature points from five consecutive frames, calculates the camera pose transformation matrix using the PNP algorithm, constructs epipolar constraint equations, and performs global optimization of the single-frame depth map (reprojection error ≤ 1.5 pixels). Non-rigid deformation compensation: Designs an LSTM module that inputs the gradient features of the depth map from three consecutive frames to predict the tissue deformation field (displacement vector field). Dynamically corrects local deformation errors caused by breathing (frequency 0.2-0.5Hz) or instrument contact (RMSE reduced by 60% after compensation).

[0018] Stereoscopic video generation and rendering optimization: Binocular disparity generation: Left and right eye views are generated based on the depth map D. The left eye view is the original image, and the right eye view is generated using a translation transformation T(x)=[xd(x),y], where the disparity d(x)=baseline×f / D(x) (baseline is the virtual binocular baseline, and f is the focal length). Occlusion filling: The CR-Stereo algorithm is used to detect occluded areas through cross-validation. Holes are filled using a weighted median filter, with the SSIM between the filled and ground truth areas ≥ 0.92. Temporal smoothing: Motion flow between adjacent frames is calculated on the GPU using optical flow estimation (RAFT algorithm), guiding the generation of intermediate interpolated frames. The interpolated frame rate is twice the input frame rate (60 fps → 120 fps), eliminating jitter artifacts in 3D reconstruction (jitter amplitude ≤ 0.5 pixel).

[0019] The present invention achieves the following benefits: It achieves the conversion of 2D endoscopic video into real-time 3D stereoscopic video through a four-stage pipeline: preprocessing, depth estimation, motion optimization, and parallax generation. The key breakthrough lies in a deep learning model optimized for endoscopic scenes, multimodal geometric constraints, and hardware parallel optimization, overcoming the shortcomings of traditional methods in terms of accuracy, real-time performance, and anti-interference capabilities. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the implementation methods of the present application, the following is a brief introduction to the drawings required for use in the implementation methods. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0021] Figure 1 This is a schematic diagram of the system hardware connection provided in the implementation mode of this application. DETAILED DESCRIPTION

[0022] To make the purpose, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0023] See also Figure 1 This application provides a method for real-time 3D conversion of 2D video from a medical endoscope based on deep learning. Specific implementation methods include: System Architecture: Hardware: Sensor: 1 / 2.3-inch CMOS camera, 1920×1080 resolution, 60fps, supporting automatic white balance and exposure control. Processing unit: Jetson AGX Orin development board (6-core ARM CPU + 200TOPS GPU), equipped with 16GB of LPDDR4X video memory, receiving video stream via USB 3.2 Gen2 interface. Software pipeline: Raw video input → Preprocessing module → EndoDepthNet depth estimation → Motion recovery optimization → Binocular disparity generation → 3D display output.

[0024] Detailed description of key steps: Preprocessing Module: CLAHE processing: Divides the image into 8×8 pixel sub-blocks, performs histogram equalization on each sub-block, limits the contrast gain to 40, and enhances the visibility of low-contrast textures such as blood vessels (contrast improvement of 1.8x). Specular Rejection: Uses a pre-trained U-Net model to perform semantic segmentation on instruments (such as forceps and electrocautery), detects specular reflection areas (IoU ≥ 0.85), and uses guided filtering to remove the intensity values ​​of highlight pixels (retaining 70% of the low-frequency information in the original image).

[0025] EndoDepthNet training details: Network Architecture: Encoder: ResNet-18 first 10 layers, outputting a 512-dimensional feature vector; Decoder: 4 layers of deconvolution, with the number of channels per layer increasing from 256, 128, 64, and 1, ultimately outputting a single-channel depth map. Training Parameters: AdamW optimizer, initial learning rate 1e-4, cosine annealing decay; batch size 8, training 50 epochs. Training took approximately 24 hours on an NVIDIA RTX4090.

[0026] Real-time performance measures include: Pipeline parallelism: Depth estimation and 3D rendering are assigned to different CUDA streams (Stream0 and Stream1), leveraging the GPU's asynchronous computing capabilities to achieve overlapping processing and reduce idle time (increasing utilization to 92%). Double buffering: Two video memory buffers are used to alternately store input frames. While the current frame is being processed in Stream0, the next frame is already being transferred to the Stream1 buffer via DMA, eliminating data transfer latency (reducing latency by 12ms).

[0027] Experimental verification: In vitro pig liver puncture experiment: Equipment: laparoscope (diameter 3mm), puncture needle positioning error ≤ 2mm; Comparison algorithm: traditional SGBM algorithm, commercial 3D endoscope (Stryker3DHD); Results: Depth error: RMSE of the present invention = 1.2mm, traditional algorithm = 4.5mm, commercial equipment = 1.8mm; Latency: 28ms for the present invention, 85ms for the traditional algorithm, 40ms for the commercial equipment; Doctor rating: A blind test was conducted by 10 surgeons, and the 3D perception accuracy of the present invention was 93%, while that of the commercial equipment was 89%.

[0028] This deep learning-based method for real-time 3D conversion of 2D medical endoscope videos uses multi-task learning and multi-frame optimization to reduce depth error in weakly textured areas by 73%, achieving submillimeter accuracy, meeting surgical navigation requirements and improving precision. With an end-to-end latency of 28ms, it supports 60fps input and 30fps 3D output with zero visual lag, achieving a breakthrough in real-time performance. In scenarios with 30% smoke density and 20% blood coverage, the RMSE of depth estimation increases by only 0.5mm, significantly outperforming traditional algorithms (which increase error by 3.2mm) and improving interference resistance.

[0029] The above is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Those skilled in the art will readily appreciate that the present invention is susceptible to various modifications and variations. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention shall be included within the scope of protection of the present invention.

Claims

1. A real-time 3D conversion method for medical endoscope 2D video based on deep learning, characterized in that: The steps include: S1: Preprocessing of the original 2D endoscope video, including adaptive histogram equalization and specular reflection suppression; S2: The lightweight depth estimation network EndoDepthNet is used to estimate the depth of the preprocessed video frame and generate a single-frame depth map; S3: Use the structure-from-motion algorithm to process multiple frames of video and optimize the depth map through multi-frame fusion and non-rigid deformation compensation; S4: Generate binocular disparity images based on the optimized depth map, perform sub-pixel temporal smoothing, and output real-time 3D stereo video.

2. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1 is characterized in that: In the preprocessing step: adaptive histogram equalization is used to enhance the contrast of blood vessel texture in the video image; and the instrument semantic segmentation algorithm based on the U-Net model is used to identify and suppress the specular reflection area.

3. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1 is characterized in that: The lightweight depth estimation network EndoDepthNet adopts a multi-task encoding and decoding architecture, including: a main branch for predicting the depth map, and the loss function adopts SILog loss; an auxiliary branch jointly predicts the surface normal map and optical flow field, and improves the depth estimation accuracy of weak texture areas through geometric consistency constraints; the total loss function is L_total = 0.7×L_depth+0.2×L_normal+0.1×L_flow.

4. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 3 is characterized in that: The training data for EndoDepthNet includes: 300 synthetic surgical videos (UnityEndo dataset) generated based on the Unity engine, simulating intraoperative smoke, blood adhesion, and instrument occlusion scenarios; and annotated data of 50 real cholecystectomy videos, which are used to improve the model's generalization ability to real-world scenarios through domain adaptation training.

5. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1, characterized in that: The EndoDepthNet supports INT8 quantization deployment and achieves a single-frame inference time of ≤25ms on the NVIDIA Jetson AGX Orin platform.

6. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1, characterized in that: The motion recovery structure algorithm includes: using the ORB-SLAM3 algorithm to extract the camera pose and instrument motion information of 5 adjacent frames of video, constructing a single-frame depth map optimized by extreme geometric constraints; predicting the tissue deformation field through the LSTM module, and dynamically compensating for non-rigid deformation errors on the organ surface caused by breathing or instrument contact.

7. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1, characterized in that: The binocular disparity generation step includes: generating left and right eye views based on the depth map, and using the CR-Stereo algorithm to fill the disparity information of the occluded area; the disparity range of the left and right eye views is constrained by the prior parameters of the surgical scene to avoid stereoscopic distortion.

8. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1, characterized in that: The sub-pixel temporal smoothing process includes: using the Motion-Flow optical flow field on the GPU to guide interpolation between video frames to eliminate jitter artifacts during the 3D reconstruction process; using a double buffering mechanism and CUDA stream parallel technology to implement pipeline operations for depth estimation and 3D rendering, ensuring end-to-end latency ≤ 28ms.

9. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to any one of claims 1 to 8, characterized in that: The hardware architecture of the system includes a 1080p@60fps 2D endoscope camera connected to an embedded processing unit via an HDMI / USB 3.0 interface; the embedded processing unit uses the NVIDIA Jetson AGX Orin platform, equipped with 16GB of video memory, and supports real-time parallel computing.

10. The method for real-time 3D conversion of medical endoscope 2D video based on deep learning according to claim 1, characterized in that: The method is suitable for minimally invasive surgical scenarios such as laparoscopy and arthroscopy. It has been verified in an in vitro pig liver puncture experiment that the root mean square error (RMSE) of depth estimation is ≤1.2mm, and the 3D perception accuracy of surgeons in blind tests is ≥93%.

Images