An augmented reality surgical navigation method and system based on endoscopic images

Abstract

The application provides an augmented reality surgery navigation method and system based on an endoscope image, acquires multiple target images of a knee joint shot by an endoscope, performs depth estimation processing and semantic segmentation processing on each target image, and obtains corresponding depth images and semantic segmentation images; constructs a target SLAM model based on the multiple depth images and the multiple semantic segmentation images, extracts three-dimensional semantic point clouds of a femur region and a tibia region from the target SLAM model to obtain source point clouds; extracts target point clouds and a pre-calibrated surgery navigation tunnel from a three-dimensional digital twin model pre-constructed based on medical images of the knee joint; performs registration processing based on the source point clouds and the target point clouds, obtains a target coordinate transformation matrix between the models, converts the surgery navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and projects the surgery navigation tunnel into an image plane of the current endoscope, so that a non-invasive surgery navigation method is finally provided, and intuitive navigation guidance is provided.

Claims

1. An augmented reality surgical navigation method based on endoscopic images, characterized in that, include: Acquire a video stream captured by an endoscope at the knee joint, and extract multiple frames of target images from the video stream; Depth estimation processing is performed on the target image in each frame to obtain the corresponding depth image of each frame, and semantic segmentation processing is performed on the femoral region and tibia region in the target image in each frame to obtain the corresponding semantic segmentation image of each frame; Based on the semantic segmentation images of each frame, a mask image containing only the femoral and tibial regions is generated to invalidate the depth information corresponding to soft tissue outside the mask image. Local point clouds are constructed using the mask image and depth image corresponding to the target image of each frame, and the first target image is used as a keyframe. An initial target SLAM model is constructed based on the local point cloud corresponding to the first target image. For each subsequent target image frame, the SFM algorithm is used to calculate the correspondence between the local point clouds of the current target image and the target images of previous frames, fusing the local point clouds frame by frame to iteratively update the initial target SLAM model until the iteration is complete, thus obtaining the target SLAM model. The three-dimensional semantic point clouds of the femoral and tibial regions are then extracted from the target SLAM model to obtain the source point cloud. The target point cloud is obtained by extracting the three-dimensional semantic point cloud of the femoral region and tibia region from the three-dimensional digital twin model pre-constructed based on the medical image of the knee joint, and the surgical navigation tunnel pre-marked between the femoral region and tibia region is obtained from the three-dimensional digital twin model. Based on the source point cloud and the target point cloud, a registration process is performed to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; Based on the target coordinate transformation matrix, the surgical navigation tunnel is transformed into the target SLAM model, and the surgical navigation tunnel in the target SLAM model is projected onto the image plane of the current endoscope.

2. The augmented reality surgical navigation method based on endoscopic images according to claim 1, characterized in that, The registration process based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the 3D digital twin model includes: Surface sampling is performed on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points. Based on each pair of target feature points, the transformation matrix is ​​solved to perform coarse registration processing, and the initial coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model is obtained. Using the initial coordinate transformation matrix as the initial value for iteration, the ICP algorithm is used to iteratively optimize the point cloud matching error for fine registration, thereby obtaining the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model.

3. The augmented reality surgical navigation method based on endoscopic images according to claim 2, characterized in that, The process involves surface sampling of the source point cloud and the target point cloud to obtain multiple pairs of matching target feature points. A coarse registration process is then performed based on the transformation matrix of each pair of target feature points to obtain the initial coordinate transformation matrix between the target SLAM model and the 3D digital twin model. This includes: Obtain a pre-defined set of iconic anatomical feature points that are adapted to the anatomical structure of the knee joint, and extract anatomical feature points that correspond one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud, respectively. Calculate the geometric saliency score, anatomical variability score, and intraoperative visibility score for each anatomical feature point, and obtain the matching weight for each anatomical feature point by weighted summation based on the geometric saliency score, the anatomical variability score, and the intraoperative visibility score; Based on the matching weight, the weighted similarity of each of the corresponding anatomical feature points between the source point cloud and the target point cloud is calculated to filter out multiple pairs of matching target feature points; A weighted error function is constructed, and the matching weights of each pair of target feature points are substituted into the weighted error function. The rigid body transformation matrix is ​​obtained by minimizing the weighted error function. The rigid body transformation matrix is ​​then verified by anatomical constraints to obtain the initial coordinate transformation matrix.

4. The augmented reality surgical navigation method based on endoscopic images according to claim 3, characterized in that, The step of extracting anatomical feature points corresponding one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud respectively includes: Based on the geometric type of each feature point in the set of iconic anatomical feature points, the source point cloud is processed using the corresponding three-dimensional Hough transform kernel to complete the coarse localization of the candidate anatomical feature points. Calculate the three-dimensional shape descriptor of each candidate anatomical feature point and perform bidirectional matching verification with the three-dimensional shape descriptor of the corresponding anatomical feature point in the target point cloud to complete the fine screening of candidate anatomical feature points; The weighted random sampling consensus algorithm is used to process the selected feature point pairs to eliminate mismatched points generated during the feature point matching process, thereby obtaining multiple pairs of matched anatomical feature points.

5. The augmented reality surgical navigation method based on endoscopic images according to claim 1, characterized in that, The step of projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope includes: The surgical navigation tunnel in the target SLAM model is projected onto the image plane of the endoscope. Different graphic identifiers are used to represent the entrance, axis, and depth reference information of the surgical navigation tunnel, and a semi-transparent blending mode is used for drawing so that the generated navigation identifiers do not obscure the real tissue image of the knee joint acquired by the endoscope.

6. The augmented reality surgical navigation method based on endoscopic images according to claim 1, characterized in that, After projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope, the method further includes: The spatial pose information of the endoscope and surgical tools is collected in real time by optical positioning equipment, and combined with the pre-completed coordinate system calibration results to calculate the spatial pose of the surgical tools under the target SLAM model. The spatial deviation between the surgical tool and the surgical navigation tunnel is calculated, and the spatial deviation is superimposed on the image plane of the endoscope for real-time display.

7. An augmented reality surgical navigation system based on endoscopic images, characterized in that, include: An image acquisition module is used to acquire a video stream captured by an endoscope at the knee joint and extract multiple frames of target images from the video stream. The image processing module is used to perform depth estimation processing on each frame of the target image to obtain the corresponding depth image of each frame, and to perform semantic segmentation processing on the femoral region and tibia region in each frame of the target image to obtain the corresponding semantic segmentation image of each frame. The SLAM model construction module is used to generate a mask image containing only the femoral and tibial regions based on the semantic segmentation images of each frame, so as to invalidate the depth information corresponding to the soft tissue outside the mask image; to construct corresponding local point clouds based on the mask image and the depth image corresponding to the target image of each frame, and to construct an initial target SLAM model based on the local point cloud corresponding to the first frame of the target image as a keyframe; for each subsequent frame of the target image, the SFM algorithm is used to calculate the correspondence between the local point clouds of the current frame of the target image and the target images of each historical frame, so as to fuse each local point cloud frame by frame, and to iteratively update the initial target SLAM model until the iteration is completed, thus obtaining the target SLAM model, and extracting the three-dimensional semantic point clouds of the femoral and tibial regions from the target SLAM model to obtain the source point cloud; The information acquisition module is used to extract the three-dimensional semantic point cloud of the femoral region and the tibia region from a three-dimensional digital twin model pre-constructed based on the medical image of the knee joint to obtain the target point cloud, and to obtain the surgical navigation tunnel pre-marked between the femoral region and the tibia region from the three-dimensional digital twin model. The registration module is used to perform registration processing based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; The navigation display module is used to transform the surgical navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope.

8. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the augmented reality surgical navigation method based on endoscopic images as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the augmented reality surgical navigation method based on endoscopic images as described in any one of claims 1 to 6.

Images