Scene Reconstruction-Based Nasal Endoscopic Surgical Navigation System

Abstract

This invention discloses a nasal endoscopic surgical navigation system based on scene reconstruction, comprising a device acquisition unit, a deblurring module, a feature extraction module, a depth estimator, and a model reconstruction module. The deblurring module is signal-connected to the device acquisition unit and is used to acquire video image sequences inside the nasal cavity and perform frame-by-frame deblurring of the video image sequences. The feature extraction module is connected to both the deblurring module and the device acquisition unit, and is used to acquire the deblurred image sequences and corresponding camera poses, and extract 3D features from the deblurred image sequences and corresponding camera poses. The depth estimator is connected to the feature extraction module and is used to acquire 3D features and extract depth information from the 3D features. The model reconstruction module is connected to the depth estimator and is used to construct a 3D reconstruction model. This invention can reconstruct the nasal endoscopic scene in real time and with high quality, providing doctors with rich 3D information and helping to reduce surgical risks.

Description

Technical Field

[0001] This invention relates to the field of medical device technology, and in particular to a nasal endoscopic surgical navigation system based on scene reconstruction. Background Technology

[0002] Nasal endoscopy, a minimally invasive surgical technique, typically involves accessing the patient's body through the nasal cavity to remove tumors. Surgeons must navigate precisely within this confined space, avoiding critical areas as much as possible. Traditional surgical navigation relies primarily on 2D endoscopic images, but these images have limited fields of view and lack depth perception, potentially leading to inaccurate surgical positioning. To improve the overall perception of the surgical environment, 3D reconstruction technology can track the endoscope's position, providing crucial information for minimally invasive surgery.

[0003] In the field of computer vision, depth estimation and reconstruction using image sequences acquired by cameras has always been an active research area.

[0004] Wang attempted to reconstruct a bronchial feature point model using a monocular bronchoscope and ORB-SLAM. ORB-SLAM was used to estimate the endoscope's orientation and provide a corresponding intraoperative map. However, the reconstruction results exhibited considerable sparsity, only reproducing the general outline of the bronchi, and the reconstruction quality needs improvement.

[0005] Batlle proposed a deep learning architecture called LightNeus, which reconstructs colon slices by utilizing differentiable rendering and neural networks combined with SDF principles. However, when faced with the complex scenarios of nasal endoscopy, this method often struggles to capture nasal cavity features and lacks generalization performance.

[0006] Because the nasal cavity is narrow and complex, rich in blood vessels, and lacks textural features, coupled with the influence of endoscopic lighting, its structural features are often difficult to capture. Furthermore, the quality of endoscopic images is hard to guarantee, frequently exhibiting blurriness and contamination. Therefore, to assist surgeons in precise navigation during nasal endoscopic surgery, it is necessary to design an effective 3D reconstruction model for nasal endoscopic scenarios to aid surgical navigation. Summary of the Invention

[0007] To address the aforementioned issues, this invention provides a scene reconstruction-based nasal endoscopic surgical navigation system that can provide accurate and rich three-dimensional information in real time during surgery and synthesize new perspectives, further reducing interference from blind spots in the surgical field and minimizing surgical risks.

[0008] To achieve the above objectives, the present invention provides a nasal endoscopic surgical navigation system based on scene reconstruction, the technical solution of which is as follows:

[0009] A scene reconstruction-based nasal endoscopic surgical navigation system, comprising:

[0010] The device acquisition unit is used to acquire video image sequences inside the nasal cavity and record the corresponding camera pose;

[0011] A deblurring module, which is connected to the device's acquisition unit, is used to acquire a video image sequence inside the nasal cavity. The deblurring module is configured to perform frame-by-frame deblurring processing on the video image sequence.

[0012] A feature extraction module is connected to the deblurring module and the device collector, respectively, and is used to acquire the deblurred image sequence and the corresponding camera pose. The feature extraction module is configured to extract three-dimensional features from the deblurred image sequence and the corresponding camera pose. The three-dimensional features include the structural and shape information of the inside of the nasal cavity.

[0013] A depth estimator, connected to the feature extraction module, is used to acquire the three-dimensional features, and the depth estimator is configured to extract depth information from the three-dimensional features;

[0014] A model reconstruction module, connected to the depth estimator, is used to acquire the three-dimensional features and depth information. The model reconstruction module is configured to perform mathematical modeling using differentiable rendering technology based on the acquired three-dimensional features and depth information, converting the image into a geometric representation in three-dimensional space to obtain a three-dimensional reconstruction model.

[0015] Furthermore, the video image sequence deblurring module is further configured to perform frame-by-frame deblurring of the video image sequence based on semantic segmentation algorithm and optical flow estimation. The semantic segmentation algorithm uses blurred frames in the video image sequence to understand the scene content, and the optical flow estimation uses motion models to guide estimation for different regions of the video image sequence.

[0016] Furthermore, the semantic segmentation algorithm assigns each pixel in the video image sequence to a set category to understand the content of the nasal endoscopy scene, provides boundary information of the endoscopy scene, and guides the optical flow estimation.

[0017] Furthermore, the optical flow estimation obtains the motion information of pixels during the blurring process by calculating the motion trajectory of each pixel in the image between consecutive frames, thereby reducing visual artifacts caused by blurring.

[0018] Furthermore, the feature extraction module is further configured to obtain contour points in three-dimensional space by back-projection using video image sequences and corresponding camera poses.

[0019] Furthermore, the feature extraction module is further configured to merge contour points to obtain structural and shape information inside the nasal cavity.

[0020] Furthermore, the depth estimator includes an MLP module and a Transformer module. The MLP module is used to initially extract features and depth estimation results from the nasal endoscopy image, and the Transformer module is used to capture the global dependencies between the features.

[0021] Furthermore, the MLP module uses a convolutional neural network to perform convolution operations on the input video image sequence using multiple convolutional layers and kernels to obtain the local features and one-dimensional feature vectors of the video image sequence. Through multiple fully connected layers, using the weight matrix and bias vector contained in each layer, the feature vector and depth information after linear transformation are obtained.

[0022] Furthermore, the Transformer module obtains the attention weight matrix of the 3D features through positional encoding, calculates the attention weight matrix of the input 3D features through a multi-head self-attention mechanism and captures the global dependencies between features to obtain the feature representation after self-attention processing, and processes the feature representation through a feedforward neural network and a decoder to obtain the decoded depth information.

[0023] Furthermore, the model reconstruction module is further configured as follows:

[0024] The internal structure and shape information of the nasal cavity are processed to obtain a triangular network model;

[0025] Based on the camera pose, the 3D model is projected onto a 2D plane;

[0026] The image projected onto a two-dimensional plane is rasterized to generate pixel information;

[0027] The rasterized pixels are then colored to obtain the rendered 3D reconstruction model.

[0028] The present invention has at least the following beneficial effects:

[0029] This invention can reconstruct the nasal endoscope scene in real time and with high quality, providing doctors with rich three-dimensional information and helping to reduce surgical risks. Attached Figure Description

[0030] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the accompanying drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. In all the drawings, similar elements or parts are generally identified by similar reference numerals. In the drawings, the elements or parts are not necessarily drawn to scale.

[0031] Figure 1 A schematic diagram of a scene reconstruction-based nasal endoscopic surgical navigation system according to an embodiment of the present invention is shown.

[0032] Figure 2 A flowchart illustrating the workflow of a scene reconstruction-based nasal endoscopic surgical navigation system according to an embodiment of the present invention is shown.

[0033] Figure 3 A schematic diagram of the depth estimator of a scene reconstruction-based nasal endoscopic surgical navigation system according to an embodiment of the present invention is shown.

[0034] Figure 4 A flowchart illustrating the differentiable rendering technique according to an embodiment of the present invention is shown. Detailed Implementation

[0035] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0036] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0037] Please see Figure 1 This is a structural diagram of a nasal endoscopic surgical navigation system based on scene reconstruction. This embodiment of the invention provides a nasal endoscopic surgical navigation system based on scene reconstruction, which includes a device acquisition unit 100, a deblurring module 200, a feature extraction module 300, a depth estimator 400, and a model reconstruction module 500. Figure 1 The arrows in the diagram not only show the connection relationships between the various components of the system, but also the signal feed directions between the components.

[0038] Please see Figure 2 This is a flowchart of a scene reconstruction-based nasal endoscopic surgical navigation system. The following will combine... Figure 2 This paper details the connections between the device acquisition unit 100, the deblurring module 200, the feature extraction module 300, the depth estimator 400, and the model reconstruction module 500 included in the system, as well as the working principles of each component.

[0039] The device acquisition unit 100 is used to acquire video image sequences inside the nasal cavity and record the corresponding camera pose.

[0040] It should be noted that the device acquisition unit 100 mentioned in this article is an existing instrument or device, such as a professional endoscope. The professional endoscope is used to acquire video image sequences of the inside of the nasal cavity and record the corresponding camera pose.

[0041] Considering that the video image sequence acquired by the device acquisition unit 100 may be jittery and blurry due to device movement or surgical operations, the deblurring module 200 is connected to the device acquisition unit 100 to obtain the video image sequence inside the nasal cavity, and the deblurring module 200 is configured to perform frame-by-frame deblurring processing on the video image sequence.

[0042] In some embodiments, the deblurring module 200 applies semantic segmentation algorithms and optical flow estimation to perform image deblurring. The semantic segmentation algorithm uses the blurred frames of the nasal endoscope to understand the scene content, assigning each pixel in the nasal endoscope image to a specific category, understanding the endoscopic scene content, providing boundary information of the endoscopic scene, and guiding the optical flow estimation; the optical flow estimation uses motion models to guide estimation for different regions of the nasal endoscope image, calculating the motion trajectory of each pixel in the image between consecutive frames to obtain the motion information of the pixels during the blurring process, reducing visual artifacts caused by blurring.

[0043] The feature extraction module 300 is connected to the deblurring module 200 and the device acquisition unit 100 respectively to obtain the deblurred image sequence and the corresponding camera pose. The feature extraction module 300 is configured to extract three-dimensional features from the deblurred image sequence and the corresponding camera pose. The three-dimensional features include the structure and shape information inside the nasal cavity.

[0044] In some embodiments, the feature extraction module 300 is configured with a visual shell algorithm to extract three-dimensional features. Specifically, the visual shell algorithm extracts three-dimensional features from the deblurred image sequence and the corresponding camera pose, the three-dimensional features including the structural and shape information of the nasal cavity.

[0045] The visual shell algorithm obtains contour points in three-dimensional space by back-projecting nasal endoscope images and corresponding camera poses. These contour points are then merged, and three-dimensional feature extraction and three-dimensional modeling techniques are used to obtain the structural and shape information of the nasal cavity.

[0046] The depth estimator 400 is connected to the feature extraction module 300 to obtain the three-dimensional features, and the depth estimator 400 is configured to extract depth information from the three-dimensional features.

[0047] In some embodiments, please refer to Figure 3 This is a structural diagram of the depth estimator 400, which includes an MLP module and a Transformer module.

[0048] Specifically, please combine Figure 3 The principle behind depth information extraction by the depth estimator 400 is as follows:

[0049] The input consists of a deblurred image and 3D features, which are processed simultaneously by the MLP module and the Transformer module.

[0050] The MLP module uses a convolutional neural network to perform convolution operations on the input nasal endoscope image using multiple convolutional layers and kernels. After vector normalization, it obtains the local features and one-dimensional feature vector of the nasal endoscope image. Through multiple fully connected layers, it uses the weight matrix and bias vector contained in each layer to obtain depth estimation information.

[0051] like Figure 3 As shown, the convolutional neural network encompasses three identical basic structures, which include convolutional layers with a 3x3 kernel size, a ReLU activation function, and a max pooling layer.

[0052] The Transformer module obtains the attention weight matrix of the 3D features through position encoding, calculates the attention weight matrix of the input 3D features through a multi-head self-attention mechanism and captures the global dependencies between features to obtain the feature representation after self-attention processing, and processes the feature representation through a feedforward neural network and a decoder to obtain more accurate depth estimation information after decoding.

[0053] like Figure 3 As shown, the depth estimator 400 also includes a fusion module, which further fuses depth estimation information 1 and depth estimation information 2 to obtain and output the final depth information.

[0054] The model reconstruction module 500 is connected to the depth estimator 400 to obtain the three-dimensional features and depth information. The model reconstruction module 500 is configured to perform mathematical modeling using differentiable rendering technology based on the obtained three-dimensional features and depth information, convert the image into a geometric representation in three-dimensional space, and obtain a three-dimensional reconstruction model.

[0055] In some embodiments, please refer to Figure 4 The following is a flowchart illustrating the workflow of differentiable rendering technology. The specific process by which the model reconstruction module 500 constructs a 3D reconstruction model based on differentiable rendering technology is as follows:

[0056] Inputting 3D features such as depth information, a geometry processing module processes the internal structure and shape information of the nasal cavity to obtain a triangular network model; a viewpoint transformation module projects the 3D model onto a 2D plane according to the camera pose; a rasterization module processes the image projected onto the 2D plane, performing rasterization to generate pixel information; and a shading module colors the rasterized pixels according to the physical principles of diffuse and specular reflection to obtain the rendered 3D reconstructed model.

[0057] The above embodiments are only used to illustrate the present invention and are not intended to limit the present invention. Those skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, all equivalent technical solutions also fall within the scope of the present invention, and the patent protection scope of the present invention should be defined by the claims.

Claims

1. A nasal endoscopic surgical navigation system based on scene reconstruction, characterized in that, include: The device acquisition unit is used to acquire video image sequences inside the nasal cavity and record the corresponding camera pose; A deblurring module, which is connected to the device's acquisition unit, is used to acquire a video image sequence inside the nasal cavity. The deblurring module is configured to perform frame-by-frame deblurring processing on the video image sequence. A feature extraction module is connected to the deblurring module and the device collector, respectively, and is used to acquire the deblurred image sequence and the corresponding camera pose. The feature extraction module is configured to extract three-dimensional features from the deblurred image sequence and the corresponding camera pose. The three-dimensional features include the structural and shape information of the inside of the nasal cavity. A depth estimator, connected to the feature extraction module, is used to acquire the three-dimensional features, and the depth estimator is configured to extract depth information from the three-dimensional features; A model reconstruction module, connected to the depth estimator, is used to acquire the three-dimensional features and depth information. The model reconstruction module is configured to perform mathematical modeling using differentiable rendering technology based on the acquired three-dimensional features and depth information, converting the image into a geometric representation in three-dimensional space to obtain a three-dimensional reconstruction model.

2. The system according to claim 1, characterized in that, The video image sequence deblurring module is further configured to perform frame-by-frame deblurring of the video image sequence based on semantic segmentation algorithm and optical flow estimation. The semantic segmentation algorithm uses blurred frames in the video image sequence to understand the scene content, and the optical flow estimation uses motion models to guide estimation for different regions of the video image sequence.

3. The system according to claim 2, characterized in that, The semantic segmentation algorithm assigns each pixel in the video image sequence to a set category to understand the content of the nasal endoscopy scene, provides boundary information of the endoscopy scene, and guides the optical flow estimation.

4. The system according to claim 2, characterized in that, The optical flow estimation obtains the motion information of pixels during the blurring process by calculating the motion trajectory of each pixel in the image between consecutive frames, thereby reducing visual artifacts caused by blurring.

5. The system according to claim 1, characterized in that, The feature extraction module is further configured to obtain contour points in three-dimensional space by back-projecting video image sequences and corresponding camera poses.

6. The system according to claim 5, characterized in that, The feature extraction module is further configured to merge contour points to obtain structural and shape information inside the nasal cavity.

7. The system according to claim 1, characterized in that, The depth estimator includes an MLP module and a Transformer module. The MLP module is used to initially extract features and depth estimation results from the nasal endoscopy image, and the Transformer module is used to capture the global dependencies between the features.

8. The system according to claim 7, characterized in that, The MLP module uses a convolutional neural network to perform convolution operations on the input video image sequence using multiple convolutional layers and kernels to obtain the local features and one-dimensional feature vectors of the video image sequence. Through multiple fully connected layers, using the weight matrix and bias vector contained in each layer, the feature vector and depth information after linear transformation are obtained.

9. The system according to claim 8, characterized in that, The Transformer module obtains the attention weight matrix of the 3D features through position encoding, calculates the attention weight matrix of the input 3D features through a multi-head self-attention mechanism and captures the global dependencies between features to obtain the feature representation after self-attention processing, and processes the feature representation through a feedforward neural network and a decoder to obtain the decoded depth information.

10. The system according to claim 1, characterized in that, The model reconstruction module is further configured as follows: The internal structure and shape information of the nasal cavity are processed to obtain a triangular network model; Based on the camera pose, the 3D model is projected onto a 2D plane; The image projected onto a two-dimensional plane is rasterized to generate pixel information; The rasterized pixels are then colored to obtain the rendered 3D reconstruction model.

Images