Endoscope mixed reality navigation method based on geometric structure perception
By using a geometry-based monocular endoscope deep learning network, the real-time problem of endoscopic 3D reconstruction and navigation was solved, achieving efficient 3D reconstruction and mixed reality navigation without additional equipment, thus improving the accuracy and safety of the operation.
Patent Information
- Application Number
- CN202310176848.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-28
- Publication Date
- 2025-12-12
- Estimated Expiration
- 2043-02-28
AI Technical Summary
Existing endoscopic 3D reconstruction methods are difficult to meet real-time requirements under conditions of soft tissue changes, and external optical navigation devices cannot accurately locate the endoscope, resulting in poor mixed reality navigation performance.
A mixed reality navigation model is generated by using a geometrically perceptive monocular endoscope deep learning network, which constructs a dataset, trains a depth estimation network, reprojects point clouds, and performs adaptive scale registration.
It enables 3D reconstruction and real-time navigation in endoscopic scenarios without additional equipment, improving operational accuracy and safety, and enhancing the ability to plan 3D scenarios that combine virtual and real elements.
Smart Images
Figure CN116229027B_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The application belongs to the field of mixed display navigation, in particular to an endoscope mixed reality navigation method based on geometric structure perception. BACKGROUND
[0002] Endoscopy can see lesions that X-rays cannot display. Virtual reality technology can make it difficult to experience physical information within a certain time and space range in the real world, and the information is superimposed on the real world after simulation and is perceived by human senses. At present, although the mixed reality technology provides great convenience for surgical operation, it still cannot synchronize the preoperative three-dimensional information with the visual field information in actual operation. Therefore, the mixed reality technology combining virtual and real can effectively solve the above problems. Through three-dimensional reconstruction and space mapping model, the virtual three-dimensional model is combined with the synthesized image to provide more realistic three-dimensional information for the operator, facilitate better three-dimensional planning, and improve the accuracy and safety of operation.
[0003] The existing endoscope image three-dimensional reconstruction method estimates the depth of the current frame of the endoscope image through a neural network model and obtains a local point cloud. The camera pose corresponding to each frame of image is solved by forward kinematics modeling of the endoscope robot, and the point cloud splicing is realized to obtain the global point cloud. The process of obtaining the camera pose of the three-dimensional reconstruction method needs a large number of mathematical operations, which is difficult to adapt to the characteristics of soft tissue changes, and the point cloud updating process is difficult to meet the real-time requirement. The existing endoscope navigation registration method acquires the position of the endoscope by image acquisition, and performs coordinate registration by using an optical navigation device. The external optical navigation device used in the registration method cannot obtain the position of the endoscope, so it is difficult to apply to the mixed reality navigation of interventional endoscopes. SUMMARY
[0004] The application proposes a monocular endoscope depth learning network based on geometric structure perception for endoscope mixed reality navigation. First, a set of synthetic data sets for training neural network is constructed for depth estimation training. Second, the estimated depth and the re-projection generate the three-dimensional shape of the tissue surface. Finally, the three-dimensional model reconstructed by preoperative CT is registered with the tissue surface shape of the synthesized endoscopic image by an adaptive scale registration method to generate a mixed reality navigation model. The problem of inaccurate three-dimensional reconstruction of the endoscope under the condition of sparse tissue texture under the endoscope and unstable endoscope illumination in the existing method is overcome.
[0005] The technical scheme adopted by the application to achieve the above-mentioned purposes is:
[0006] The endoscope mixed reality navigation method based on geometric structure perception comprises the following steps:
[0007] Construct an endoscope data set;
[0008] constructing and training a depth estimation network, and performing depth estimation on data in the endoscope dataset by using the trained depth estimation network;
[0009] reprojecting the estimated endoscope depth data to complete point clouds of consecutive frames of the endoscope;
[0010] performing adaptive scale registration between the cavity point cloud model in the endoscope dataset and the point cloud reconstructed from the synthetic data by using an adaptive scale point cloud registration method to obtain a registered point cloud model;
[0011] visually displaying the registered point cloud model by using a mixed reality technology.
[0012] The endoscope dataset is constructed by the following steps:
[0013] obtaining human cavity CT images from a database, segmenting and reconstructing the images to obtain a reconstructed cavity point cloud model;
[0014] simulating an endoscope imaging process, iteratively rendering the channel, generating a subset of multiple data sets, using different colors, textures and light intensities for each subset to obtain synthetic RGB images of the endoscope and corresponding depth ground truths.
[0015] The depth estimation network is constructed by using an encoder-decoder structure, wherein: the backbone of the encoder network is a ResNet18 network, and the encoder network is modified by using a spatial attention module, the decoder network is formed by upsampling convolution, and a skip connection is established in each scale feature mapping between the encoder network and the decoder network.
[0016] The encoder network is:
[0017] Z=f(X,X T )g(X)
[0018]
[0019] wherein Z represents a block operation, g, θ, all represent convolution after maximum pooling, ψ, σ relu respectively represent convolution operation and ReLU activation function, X∈R N×C×H×W represents input data, i.e. synthetic RGB images, wherein R represents a real number set, N represents a quantity, C represents a channel number, H represents a height, W represents a width, and output f represents a pixel-level relationship on X.
[0020] The decoder network is:
[0021] F=φ(Z)+X
[0022] where φ denotes the up-sampling operation and F denotes the decoder output.
[0023] The depth estimation network is trained by constructing a geometry-aware depth estimation loss function, including the following steps:
[0024] Calculate the direct L1 loss l of the predicted depth and the ground truth depth :
[0025]
[0026] where, denotes the true value in the synthetic data, denotes the estimated value of the depth;
[0027] Calculate the gradient loss l of the edge grad :
[0028]
[0029] where, and denote the gradient in the X and Y directions, respectively;
[0030] Calculate the cosine similarity loss l of the surface normal generated by the depth map normal :
[0031]
[0032] where, and denote the normal value of the predicted depth and the ground truth, and <·,·> denotes the inner product of the vector;
[0033] Calculate the smoothing loss l smooth :
[0034]
[0035] where, denotes the gradient operation on the i-th frame image;
[0036] The total loss function l of the depth estimation network is: D
[0037] l D = λ depth (l depth +l grad +l normal )+ λ smooth l smooth
[0038] where λ depth and λ smooth denote the corresponding weights.
[0039] The adaptive scale point cloud registration method is specifically: the positional relationship between the cavity point cloud model and the reconstructed global point cloud is calculated as a scale factor, the scale factor is directly integrated into a least square problem, and a rotation and translation matrix and a scale factor of registration of the cavity point cloud model and the reconstructed global point cloud are obtained.
[0040] By minimizing the function E(s,t), the translation t and the uniform scale factor s are optimized:
[0041]
[0042] Wherein p represents the cavity point cloud model, q represents the reconstructed global point cloud, s is in R, and t is in R 3×1 ;
[0043] That is:
[0044]
[0045] Wherein c and d represent coefficients,
[0046] The endoscopic mixed reality navigation system based on geometric structure perception comprises:
[0047] A dataset construction module is configured to construct an endoscopic dataset.
[0048] A depth estimation network construction module is configured to construct and train a depth estimation network, and to perform depth estimation on data in the endoscopic dataset by using the trained depth estimation network.
[0049] A point cloud matching module is configured to perform re-projection on the estimated endoscopic depth data, to complete point cloud to endoscopic continuous frames, and to perform adaptive scale point cloud registration on a cavity point cloud model in the endoscopic dataset and a point cloud reconstructed from the synthetic data, to obtain a registered point cloud model.
[0050] A visual navigation module is configured to visually display the registered point cloud model by using a mixed reality technology.
[0051] A computer readable storage medium, wherein a computer program is stored on the storage medium, and when the computer program is executed by a processor, the endoscopic mixed reality navigation method based on geometric structure perception is realized.
[0052] The present application has the following advantages and benefits:
[0053] The application provides a three-dimensional reconstruction method suitable for an endoscope scene without additional binocular endoscopes, electromagnetic navigation or optical navigation equipment. A global point cloud model that can be locally updated is formed, and adaptive scale registration is performed with a preoperative three-dimensional reconstruction model to realize mixed reality navigation display. BRIEF DESCRIPTION OF DRAWINGS
[0054] Figure 1 An endoscopic mixed reality navigation framework diagram;
[0055] Figure 2 A depth estimation network block diagram;
[0056] Figure 3 A point cloud registration block diagram;
[0057] Figure 4 A mixed reality visualization schematic diagram. DETAILED DESCRIPTION
[0058] The application will be further described in detail below in combination with the drawings and embodiments.
[0059] As Figure 1 shown, the mixed reality navigation method of the endoscope provided by the application includes the following steps:
[0060] 1. Segmentation and reconstruction are performed using preoperative human cavity CT images to obtain a preoperative reconstructed cavity point cloud model, synthetic data is constructed based on illumination rendering, and synthetic data sets are constructed for geometric structures under severe reflection and illumination changes.
[0061] 2. Synthetic endoscope data and synthetic endoscope data depth true values are used as inputs to design a depth estimation loss function based on geometric perception, supervised network training for depth estimation is realized, network weights are updated, and depth estimation reasoning is performed.
[0062] 3. The depth estimated by the unsupervised network is reprojected to obtain local point clouds of consecutive endoscopic frames. An adaptive scale point cloud registration algorithm is used to adaptively register the cavity point cloud model with the global point cloud reconstructed by the synthetic data to obtain a registered point cloud model, and the registered model is visually displayed through mixed reality technology.
[0063] The specific implementation steps are as follows:
[0064] (1) First, RGB images and corresponding depth maps were synthesized by light rendering. A 3D environment synthesizer was used to simulate the endoscopic imaging process. A virtual camera with an additional spotlight was set up to follow the endoscopic navigation process in the virtual model. To simulate the changes in imaging and lighting conditions, the rendering pass was iteratively run to generate several subsets of the dataset. Each subset used different colors, textures, and lighting intensities to obtain image and depth ground truths, with a minimum depth of 0.01 (calculated in simulation units) and a maximum depth of 100. The 3D environment synthesizer was the open-source blender software.
[0065] (2) Second, a depth estimation network with an encoder-decoder structure was designed. Figure 2 is the network diagram of depth estimation, which is implemented as follows:
[0066] The backbone of the encoder network is a ResNet18 network. The spatial attention module (SAB) in the disparity network can improve the perception of endoscopic image texture. Therefore, the encoder network was modified using SAB to enhance the output of the first layer. X∈R N×C×H×W is the input data, where R is the real set, N represents the number, C represents the number of channels, H represents the height, and W represents the width. The output f represents the pixel-level relationship on X, and the process of the spatial attention module is described as follows:
[0067]
[0068] where Z represents the block operation, g, θ, represents the convolution after max pooling, ψ, σ relu represent the convolution operation and the ReLU activation function, respectively, and φ represents the up-sampling operation. The result of f has been activated by the softmax operator σ softmax . The activation can be seen as a weight attached to the input, which tends to enhance the target features. The up-sampled output is performed using the following convolution:
[0069] F = φ(Z) + X (2)
[0070] The encoder network is followed by a decoder network formed by up-sampling convolutions. At the same time, skip connections are established between the feature maps at each scale in the encoder and decoder networks.
[0071] A depth estimation loss function with geometric structure perception was designed. First, the direct L1 loss of the predicted depth and the ground truth was calculated.
[0072]
[0073] where is the true value in the synthetic data, an estimate of depth.
[0074] Secondly, the gradient loss of the edge is calculated:
[0075]
[0076] where, and denote the gradient in X and Y direction respectively.
[0077] Thirdly, the cosine similarity loss of the surface normal generated from the depth map is calculated:
[0078]
[0079] where, and are the normal values of the predicted depth and ground truth, and <·,·> is the inner product of vectors.
[0080] Finally, the smoothness loss is calculated, which combines the input image and the predicted depth, aiming to keep the edge features appearing in the input image still visible in the predicted depth, denoted as:
[0081]
[0082] where, denotes the gradient operation on the i-th frame of image.
[0083] The total loss function of the depth estimation network is:
[0084] l D = λ depth (l depth + l grad + l normal ) + λ smooth l smooth (7)
[0085] where, λ depth and λ smooth are the corresponding weights.
[0086] (3) Finally, the matching of the preoperative and synthetic image generated point clouds is realized by point cloud registration, and the visual display of the registered model is realized by using the mixed reality technology.
[0087] Figure 3 is the block diagram of point cloud registration. As shown in Figure 3 , a scale adaptive nearest point iteration algorithm is proposed to calculate the positional relationship between the preoperative point cloud and the synthetic data point cloud. By directly integrating the scale factor into the least squares problem, the rotation and translation matrix are solved, and the scale factor of the preoperative and synthetic data point cloud registration is calculated.
[0088] In the ICP algorithm, the translation t and the uniform scale factor s are jointly optimized to minimize the following function:
[0089]
[0090] where p is the point cloud in the cavity point cloud model, q is the synthetic data point cloud, s∈R, t=[t0,t1,t2]∈R 3×1 .
[0091] First, E(s,t) is converted into the following form suitable for differentiation:
[0092]
[0093] Then, the partial derivatives are set to zero to minimize:
[0094]
[0095]
[0096] Rewriting equation (9) gives:
[0097]
[0098] where
[0099] Rewriting equation (11) gives
[0100]
[0101] The linear system of equations (12) and (13) is solved quickly in the following Ax=b form:
[0102]
[0103] where
[0104] Figure 4 is a mixed reality visualization display block diagram. As shown in Figure 4 , the display interface is divided into a mixed reality window, a synthetic image window and a preoperative three-dimensional model window. According to the real-time position of the endoscope, the corresponding virtual-real fusion image, the relative position view of the endoscope and the preoperative three-dimensional model are displayed, and are updated to the display screen for display. In the field of view observation area of the endoscope, virtual-real fusion is carried out, and augmented reality guidance is realized in the field of view observation area through rendering, which changes with the change of the position and direction of the endoscope, and is improved in distance perception and scene immersion.
Claims
1. An endoscope mixed reality navigation method based on geometric structure perception, characterized in that, Includes the following steps: Construct an endoscopic dataset; Construct and train a depth estimation network, and use the trained depth estimation network to estimate the depth of data in the endoscopy dataset. The estimated endoscopic depth data is reprojected to complete the point cloud of continuous endoscopic frames; The point cloud model of the cavity in the endoscope dataset is registered with the point cloud reconstructed from the synthetic data using an adaptive scaling point cloud registration method, resulting in a registered point cloud model. The registered point cloud model is visualized using mixed reality technology; The depth estimation network is trained by constructing a geometry-aware depth estimation loss function, including the following steps: Calculate the direct L1 loss for predicted depth and ground truth. : ； in, Indicates the truth value in the synthetic data. An estimated value representing depth; Calculate the gradient loss at the edge : ； in, and These represent calculating the gradients with respect to the X and Y directions, respectively. Calculate the cosine similarity loss of the surface normals generated from the depth map. : ； in, and This represents the normal value of the predicted depth and the ground truth. Represents the dot product of vectors; Calculate smoothing loss : ； in, This indicates that gradient calculation is performed on the i-th frame of the image; Total loss function of depth estimation network for: ； in, and Indicates the corresponding weight; The adaptive scale point cloud registration method specifically involves: calculating the positional relationship between the cavity point cloud model and the reconstructed global point cloud as a scaling factor; and obtaining the rotation and translation matrices and the scale scaling factor for registering the cavity point cloud model and the reconstructed global point cloud by directly integrating the scaling factor into the least squares problem. By minimizing the function For translation and uniform scaling factor Optimize: ； in, Represents the point cloud model of the cavity. Represents the reconstructed global point cloud. ; Right now: ； Where c and d represent coefficients, , .
2. The endoscopic mixed reality navigation method based on geometric structure perception according to claim 1, characterized in that, The construction of the endoscopic dataset includes the following steps: The human body cavity CT images are obtained from the database and then segmented and reconstructed to obtain the reconstructed cavity point cloud model. The endoscopic imaging process is simulated by iteratively rendering the channels to generate subsets of multiple datasets. Different colors, textures, and lighting intensities are used for each subset to obtain a synthetic RGB image of the endoscope and its corresponding ground truth depth value.
3. The endoscopic mixed reality navigation method based on geometric structure perception according to claim 1, characterized in that, The depth estimation network is constructed using an encoder-decoder structure, wherein: the backbone of the encoder network is a ResNet18 network, and the encoder network is modified by a spatial attention module; the decoder network is formed by upsampling convolutions; and skip connections are established in the feature maps at each scale between the encoder network and the decoder network.
4. The endoscopic mixed reality navigation method based on geometric structure perception according to claim 3, characterized in that, The encoder network is: ； ； Where Z represents a block operation. , , Both represent convolutions after max pooling. These represent the convolution operation and the ReLU activation function, respectively. This represents the input data, i.e., the synthesized RGB image, where R represents the set of real numbers, N represents the number of elements, C represents the number of channels, H represents the height, and W represents the width. The output... express The pixel-level relationship.
5. The endoscopic mixed reality navigation method based on geometric structure perception according to claim 3, characterized in that, The decoder network is: ； in, Indicates an upsampling operation. This indicates the decoder output.
6. A geometric structure-aware endoscopic mixed reality navigation system, used to implement the geometric structure-aware endoscopic mixed reality navigation method of claim 1, characterized in that, include: The dataset building module is used to build endoscopic datasets; The depth estimation network building module is used to build and train a depth estimation network, and then use the trained depth estimation network to estimate the depth of data in the endoscopy dataset. The point cloud matching module is used to reproject the estimated endoscope depth data to complete the point cloud of continuous endoscope frames; The point cloud model of the cavity in the endoscope dataset is registered with the point cloud reconstructed from the synthetic data using an adaptive scaling point cloud registration method, resulting in a registered point cloud model. The visualization navigation module is used to visualize the registered point cloud model using mixed reality technology.
7. A computer-readable storage medium, characterized in that, The storage medium stores a computer program that, when executed by a processor, implements the endoscopic mixed reality navigation method based on geometric structure perception as described in any one of claims 1-5.