An Augmented Reality Surgical Navigation Method and System Based on Endoscopic Images

Abstract

This invention provides an augmented reality surgical navigation method and system based on endoscopic images. It acquires multiple target images captured by an endoscope at the knee joint, performs depth estimation and semantic segmentation on each frame to obtain corresponding depth images and semantic segmentation images. A target SLAM model is constructed based on these images, and three-dimensional semantic point clouds of the femoral and tibial regions are extracted from the target SLAM model to obtain source point clouds. Target point clouds and pre-calibrated surgical navigation tunnels are extracted from a pre-constructed three-dimensional digital twin model based on medical images of the knee joint. Registration processing is performed on the source and target point clouds to obtain a target coordinate transformation matrix between the models. Based on this matrix, the surgical navigation tunnel is transformed into the target SLAM model and projected onto the current endoscopic image plane. Ultimately, this provides a non-invasive surgical navigation method with intuitive navigation guidance.

Description

Technical Field

[0001] This disclosure relates to the field of medical image processing technology, and in particular to an augmented reality surgical navigation method and system based on endoscopic images. Background Technology

[0002] Endoscopic anterior cruciate ligament reconstruction is a classic surgical procedure for treating knee joint sports injuries. The key to the success of the surgery lies in the precise positioning of the femoral and tibial tunnels. Clinical studies have shown that a large number of revision patients have problems with improper femoral tunnel positioning. Therefore, it is crucial to design a navigation system for precise positioning of the femoral and tibial tunnels.

[0003] However, in related technologies, traditional navigation systems require the installation of tracking markers on bone tissue to navigate between the femur and tibia. This interventional approach increases additional trauma and risks, as well as navigation time. Furthermore, due to the limited field of vision within the joint cavity, weak tissue texture, and complex and variable lighting, existing technologies are difficult to directly apply to real-time modeling of endoscopic images, thus making it difficult to provide intuitive navigation guidance. Summary of the Invention

[0004] The main objective of this disclosure is to propose an augmented reality surgical navigation method and system based on endoscopic images, which can provide a non-invasive surgical navigation method and offer intuitive navigation guidance.

[0005] To achieve the above objectives, a first aspect of this disclosure provides an augmented reality surgical navigation method based on endoscopic images, comprising: Acquire a video stream captured by an endoscope at the knee joint, and extract multiple frames of target images from the video stream; Depth estimation processing is performed on the target image in each frame to obtain the corresponding depth image of each frame, and semantic segmentation processing is performed on the femoral region and tibia region in the target image in each frame to obtain the corresponding semantic segmentation image of each frame; A target SLAM model is constructed based on multiple frames of the depth image and multiple frames of the semantic segmentation image, and the source point cloud is obtained by extracting the three-dimensional semantic point cloud of the femoral region and the tibia region from the target SLAM model. The target point cloud is obtained by extracting the three-dimensional semantic point cloud of the femoral region and tibia region from the three-dimensional digital twin model pre-constructed based on the medical image of the knee joint, and the surgical navigation tunnel pre-marked between the femoral region and tibia region is obtained from the three-dimensional digital twin model. Based on the source point cloud and the target point cloud, a registration process is performed to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; Based on the target coordinate transformation matrix, the surgical navigation tunnel is transformed into the target SLAM model, and the surgical navigation tunnel in the target SLAM model is projected onto the image plane of the current endoscope.

[0006] In some embodiments, constructing a target SLAM model based on multiple frames of the depth image and multiple frames of the semantic segmentation image includes: Based on the semantic segmentation images of each frame, a mask image containing only the femoral and tibial regions is generated to invalidate the depth information corresponding to the soft tissue outside the mask image; A corresponding local point cloud is constructed using the mask image and depth image corresponding to the target image in each frame, and an initial target SLAM model is constructed based on the local point cloud corresponding to the target image in the first frame, using the target image in the first frame as a key frame. For each subsequent frame of the target image, the SFM algorithm is used to calculate the correspondence between the local point clouds of the target image in the current frame and the target images in previous frames, so as to fuse the local point clouds frame by frame and iteratively update the initial target SLAM model until the iteration is completed, and the target SLAM model is obtained.

[0007] In some embodiments, the registration process based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the 3D digital twin model includes: Surface sampling is performed on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points. Based on each pair of target feature points, the transformation matrix is ​​solved to perform coarse registration processing, and the initial coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model is obtained. Using the initial coordinate transformation matrix as the initial value for iteration, the ICP algorithm is used to iteratively optimize the point cloud matching error for fine registration, thereby obtaining the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model.

[0008] In some embodiments, the step of performing surface sampling on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points, and performing coarse registration processing based on the transformation matrix solution of each pair of target feature points to obtain the initial coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model includes: Obtain a pre-defined set of iconic anatomical feature points that are adapted to the anatomical structure of the knee joint, and extract anatomical feature points that correspond one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud, respectively. Calculate the geometric saliency score, anatomical variability score, and intraoperative visibility score for each anatomical feature point, and obtain the matching weight for each anatomical feature point by weighted summation based on the geometric saliency score, the anatomical variability score, and the intraoperative visibility score; Based on the matching weight, the weighted similarity of each of the corresponding anatomical feature points between the source point cloud and the target point cloud is calculated to filter out multiple pairs of matching target feature points; A weighted error function is constructed, and the matching weights of each pair of target feature points are substituted into the weighted error function. The rigid body transformation matrix is ​​obtained by minimizing the weighted error function. The rigid body transformation matrix is ​​then verified by anatomical constraints to obtain the initial coordinate transformation matrix.

[0009] In some embodiments, extracting anatomical feature points corresponding one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud respectively includes: Based on the geometric type of each feature point in the set of iconic anatomical feature points, the source point cloud is processed using the corresponding three-dimensional Hough transform kernel to complete the coarse localization of the candidate anatomical feature points. Calculate the three-dimensional shape descriptor of each candidate anatomical feature point and perform bidirectional matching verification with the three-dimensional shape descriptor of the corresponding anatomical feature point in the target point cloud to complete the fine screening of candidate anatomical feature points; The weighted random sampling consensus algorithm is used to process the selected feature point pairs to eliminate mismatched points generated during the feature point matching process, thereby obtaining multiple pairs of matched anatomical feature points.

[0010] In some embodiments, projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope includes: The surgical navigation tunnel in the target SLAM model is projected onto the image plane of the endoscope. Different graphic identifiers are used to represent the entrance, axis, and depth reference information of the surgical navigation tunnel, and a semi-transparent blending mode is used for drawing so that the generated navigation identifiers do not obscure the real tissue image of the knee joint acquired by the endoscope.

[0011] In some embodiments, after projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope, the method further includes: The spatial pose information of the endoscope and surgical tools is collected in real time by optical positioning equipment, and combined with the pre-completed coordinate system calibration results to calculate the spatial pose of the surgical tools under the target SLAM model. The spatial deviation between the surgical tool and the surgical navigation tunnel is calculated, and the spatial deviation is superimposed on the image plane of the endoscope for real-time display.

[0012] To achieve the above objectives, a second aspect of this disclosure provides an augmented reality surgical navigation system based on endoscopic images, comprising: An image acquisition module is used to acquire a video stream captured by an endoscope at the knee joint and extract multiple frames of target images from the video stream. The image processing module is used to perform depth estimation processing on each frame of the target image to obtain the corresponding depth image of each frame, and to perform semantic segmentation processing on the femoral region and tibia region in each frame of the target image to obtain the corresponding semantic segmentation image of each frame. The SLAM model building module is used to build a target SLAM model based on multiple frames of the depth image and multiple frames of the semantic segmentation image, and extract the three-dimensional semantic point cloud of the femoral region and tibia region from the target SLAM model to obtain the source point cloud; The information acquisition module is used to extract the three-dimensional semantic point cloud of the femoral region and the tibia region from a three-dimensional digital twin model pre-constructed based on the medical image of the knee joint to obtain the target point cloud, and to obtain the surgical navigation tunnel pre-marked between the femoral region and the tibia region from the three-dimensional digital twin model. The registration module is used to perform registration processing based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; The navigation display module is used to transform the surgical navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope.

[0013] To achieve the above objectives, a third aspect of this disclosure provides an electronic device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the augmented reality surgical navigation method based on endoscopic images described in the first aspect embodiment.

[0014] To achieve the above objectives, a fourth aspect of the present disclosure provides a storage medium, which is a computer-readable storage medium storing a computer program that, when executed by a processor, implements the augmented reality surgical navigation method based on endoscopic images described in the first aspect embodiment.

[0015] The beneficial effects of the embodiments disclosed herein include: Using endoscopic video streams as data input eliminates the need for additional tracking markers on the patient's bone tissue, avoiding the additional trauma and surgical risks associated with invasive markers, and saving the time-consuming steps of marker installation and calibration. Subsequently, addressing the challenges of the complex joint cavity environment, depth estimation and semantic segmentation are simultaneously performed on the extracted endoscopic target images, acquiring depth information and precise semantic boundaries of the femoral and tibial regions for each frame, solving the problems of lack of depth perception in endoscopy and difficulty in identifying key bone structures in complex environments. Then, based on multiple frames of depth images and semantically segmented images, a target SLAM model fusing semantic information is constructed, enabling marker-free endoscopic imaging. Real-time tracking of the endoscope's pose and 3D semantic reconstruction of the joint cavity's internal structure are used to extract source point clouds. These are then combined with a pre-built 3D digital twin model based on the patient's knee joint medical images to extract femoral and tibial target point clouds and pre-planned surgical navigation tunnels. Through registration processing of the source and target point clouds, a coordinate mapping relationship is established between the real-time scene and the precise model. Finally, based on the coordinate transformation matrix, the planned surgical navigation tunnel is projected onto the current endoscope's image plane, superimposing virtual navigation information onto the real field of view in an augmented reality manner. This allows for a direct and clear view of the precise location of the femoral and tibial tunnels, ultimately achieving a non-invasive, precise, and intuitive surgical navigation effect. Attached Figure Description

[0016] Figure 1 This is a schematic diagram of an application environment for the augmented reality surgical navigation method based on endoscopic images provided in the embodiments of this disclosure; Figure 2 This is a schematic flowchart of the augmented reality surgical navigation method based on endoscopic images provided in the embodiments of this disclosure; Figure 3 This is a schematic diagram illustrating the surgical navigation process in a real surgical scenario provided in the embodiments of this disclosure; Figure 4 yes Figure 2 A flowchart further includes step S103; Figure 5 yes Figure 2 A flowchart further includes step S105; Figure 6 yes Figure 5 A flowchart further included in step S301; Figure 7 yes Figure 6 A flowchart further includes step S401; Figure 8 yes Figure 2 A flowchart illustrating the further steps following step S106; Figure 9This is a schematic diagram of the functional modules of the augmented reality surgical navigation system based on endoscopic images provided in an embodiment of this disclosure; Figure 10 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0017] The accompanying drawings in the embodiments clearly and completely describe the technical solutions in the embodiments of this disclosure. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0018] It is understood that in the specific embodiments of this disclosure, which involve retrieving video streams, target images and related data, when the above embodiments of this disclosure are applied to specific products or technologies, permission or consent can be obtained from the target, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0019] Furthermore, when the embodiments of this disclosure need to retrieve video streams, target images, and related data, separate permission or consent to the video streams, target images, and related data can be obtained through pop-up windows or redirection to a confirmation page. After obtaining separate permission or consent to the video streams, target images, and related data, the necessary video streams, target images, and related data for the normal operation of the embodiments of this disclosure can be obtained.

[0020] In this disclosure, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0021] Before providing a further detailed description of the embodiments of this disclosure, the terms and concepts used in these embodiments are explained, and they are subject to the following interpretations: Artificial intelligence (AI) is a new branch of computer science that studies, develops, and applies theories, methods, technologies, and systems to simulate, extend, and expand human intelligence. It aims to understand the essence of intelligence and produce intelligent machines that can react in a way similar to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing, and expert systems. AI can simulate the information processes of human consciousness and thought. Furthermore, AI utilizes digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceiving the environment, acquiring knowledge, and using that knowledge to achieve optimal results.

[0022] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0023] Simultaneous Localization and Mapping (SLAM) is a core technology that enables robots to locate themselves in unknown environments and build environmental maps in real time using sensors. Its core principle is to simultaneously estimate the robot's own trajectory and the coordinates of environmental features by fusing measurement data from sensors (such as LiDAR, cameras, and millimeter-wave radar).

[0024] Structure from Motion (SFM) is a photogrammetric technique that reconstructs the structure of a 3D scene by analyzing image sequences. Also known as motion reconstruction, this technique calculates a 3D model and camera pose from an unordered image set through multi-view geometry and camera parameter estimation. It is primarily used in computer vision and 3D reconstruction. The SFM algorithm process includes feature extraction (such as SIFT), feature matching, camera parameter estimation (including extrinsic and intrinsic parameter matrices), and bundle adjustment optimization. Its mathematical foundation is based on a pinhole camera model and projection matrix calculation, relying on geometric constraints (such as epipolar geometry) and rigid body motion assumptions.

[0025] The Iterative Closest Point (ICP) algorithm is a point set registration method used to calculate the optimal rotation and translation transformation between two point cloud datasets to achieve accurate alignment. Based on the least squares method, this algorithm iteratively optimizes the transformation parameters through two steps: "finding the nearest corresponding point" and "solving the optimal transformation," until the mean square error is less than a set threshold.

[0026] In various endoscopic-assisted procedures, such as arthroscopic surgery, laparoscopic surgery, natural orifice endoscopic surgery, virtual surgical simulation training, and remote surgical teaching, precise localization and navigation of the target area are core requirements. Traditional navigation systems require additional markers to be installed on the target tissue, increasing trauma and operation time. Furthermore, the intracavitary field of view is limited and texture is weak, making it difficult to directly apply existing SLAM technology. Therefore, embodiments of this disclosure provide an augmented reality navigation method and system based on endoscopic images, which can achieve high-precision, real-time, and intuitive augmented reality navigation without the need for markers.

[0027] Please see Figure 1 , Figure 1 A schematic diagram of the implementation environment for the augmented reality surgical navigation method based on endoscopic images provided in this embodiment of the disclosure includes: a terminal 11 and a server 12.

[0028] For example, server 12 can acquire a video stream captured by endoscope at the knee joint from terminal 11, and extract multiple target images from the video stream; perform depth estimation processing on each target image to obtain corresponding depth images, and perform semantic segmentation processing on the femoral and tibial regions in each target image to obtain corresponding semantic segmentation images; construct a target SLAM model based on the multiple depth images and multiple semantic segmentation images, and extract the three-dimensional semantic point clouds of the femoral and tibial regions from the target SLAM model to obtain source point clouds; extract the three-dimensional semantic point clouds of the femoral and tibial regions from a three-dimensional digital twin model pre-constructed based on the medical images of the knee joint to obtain target point clouds, and obtain surgical navigation tunnels pre-marked between the femoral and tibial regions from the three-dimensional digital twin model; perform registration processing based on the source point cloud and target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; transform the surgical navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope. Finally, server 12 can display an image plane containing the surgical navigation tunnel in terminal 11.

[0029] Terminal 11 can be a mobile phone, computer, smart voice interaction device, smart wearable device, smart home appliance, vehicle terminal, endoscope device terminal, etc., but is not limited to these. Terminal 11 can also independently execute the augmented reality surgical navigation method based on endoscopic images. Terminal 11 and server 12 can be directly or indirectly connected through wired or wireless communication, and this embodiment of the disclosure does not impose any limitations.

[0030] Server 12 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Additionally, server 12 can also be a node server in a blockchain network.

[0031] It should be noted that, Figure 1 The schematic diagram of the implementation environment shown is merely an example. The scenarios described in this disclosure are intended to more clearly illustrate the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided in this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new business scenarios, the technical solutions provided in this disclosure are also applicable to similar technical problems.

[0032] Please see Figure 2 , Figure 2 This is a flowchart illustrating the augmented reality surgical navigation method based on endoscopic images provided in this embodiment. This augmented reality surgical navigation method based on endoscopic images can be applied in an augmented reality surgical navigation system based on endoscopic images (hereinafter referred to as the system or navigation system), or in the server described in the above embodiments, or jointly executed by a terminal and a server. The augmented reality surgical navigation method based on endoscopic images includes steps S101 to S106: Step S101: Acquire the video stream captured by the endoscope at the knee joint, and extract multiple target images from the video stream; Step S102: Perform depth estimation processing on each frame of target image to obtain the corresponding depth image of each frame, and perform semantic segmentation processing on the femoral region and tibia region in each frame of target image to obtain the corresponding semantic segmentation image of each frame. Step S103: Construct a target SLAM model based on multiple frames of depth images and multiple frames of semantic segmentation images, and extract the three-dimensional semantic point clouds of the femoral and tibial regions from the target SLAM model to obtain the source point cloud; Step S104: Extract the three-dimensional semantic point cloud of the femoral region and tibia region from the three-dimensional digital twin model pre-constructed based on the medical image of the knee joint to obtain the target point cloud, and obtain the surgical navigation tunnel pre-marked between the femoral region and tibia region from the three-dimensional digital twin model. Step S105: Based on the source point cloud and the target point cloud, perform registration processing to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; Step S106: Based on the target coordinate transformation matrix, the surgical navigation tunnel is transformed into the target SLAM model, and the surgical navigation tunnel in the target SLAM model is projected onto the image plane of the current endoscope.

[0033] The overall application process of the embodiments disclosed herein can be referred to Figure 3 As shown, Figure 3 This is a schematic diagram illustrating the surgical navigation process in a real surgical scenario provided by the embodiments of this disclosure. Subsequent embodiments use this scenario as an example for illustration, but this does not imply limitation on the embodiments of this disclosure. Furthermore, the embodiments of this disclosure are not only applicable to real surgical scenarios, but also to virtual surgical simulators. They can generate three-dimensional anatomical models in real time based on real surgical video streams, overlay virtual surgical tools and operational feedback, providing an immersive training environment for medical students. They can also be applied to surgical teaching and remote guidance, constructing a three-dimensional model of the surgical area in real time during surgery, and overlaying expert operational guidance onto the endoscopic field of vision of primary care physicians through augmented reality, achieving remote and precise teaching.

[0034] Regarding step S101 above, this embodiment of the present disclosure can acquire a video stream captured by an endoscope at the knee joint and extract multiple target images from the video stream. Here, an endoscope refers to an optical imaging device used to observe the interior of body cavities or passages, such as an arthroscope, laparoscope, or gastroscope. The target area refers to the part requiring navigation, such as the knee joint cavity, abdominal cavity, or thoracic cavity. When this embodiment of the present disclosure is used for surgical navigation of the knee joint, the aforementioned endoscope can also be called an arthroscope. The video stream is a sequence of continuously acquired color images by the endoscope, transmitted to a computer system via an image acquisition card. Furthermore, the system can automatically sample the video stream at equal intervals, for example, sampling 5 frames per second to obtain multiple target images.

[0035] Furthermore, each frame of the target image usually needs to be preprocessed before subsequent processing, including distortion correction. Lens distortion can be eliminated using pre-calibrated camera intrinsic parameters and distortion coefficients. In addition, white balance adjustment and contrast enhancement can be performed to eliminate color deviation and uneven lighting caused by different light sources and angles. This disclosure does not impose specific limitations on these aspects.

[0036] It should be noted that the embodiments of this disclosure use endoscopic video streams directly as data input, eliminating the need to install any additional tracking markers on the target tissue. This avoids the additional trauma and operational risks associated with invasive markers, saves time-consuming steps in marker installation and calibration, and makes the method naturally suitable for virtual training and remote teaching scenarios.

[0037] Regarding step S102 above, this embodiment of the disclosure can perform depth estimation processing on each frame of target image to obtain corresponding depth images for each frame, and perform semantic segmentation processing on the target tissue regions in each frame of target image, namely the femur region and the tibia region, to obtain corresponding semantic segmentation images for each frame. Here, the depth image refers to a two-dimensional matrix with the same size as the color image, where each pixel value represents the distance from that point to the endoscope camera. This embodiment of the disclosure can employ a deep learning model, such as Metric3D, to predict depth values ​​from a single frame of color image. This model is pre-trained on a large amount of indoor and outdoor scene data and can adapt to common lighting changes, texture loss, and specular reflections in endoscopic images, outputting relative or absolute depth. The principle of depth estimation is based on the scale uncertainty of monocular SLAM vision. Through neural network learning, the mapping relationship from image features to depth is obtained, and the depth image provides spatial coordinate information for each pixel for subsequent 3D reconstruction.

[0038] Semantic image segmentation refers to labeled images obtained by classifying each pixel in a color image into a predefined category. In this embodiment, a neural network, such as nnU-Net, can be used to classify endoscopic images pixel-by-pixel, labeling each pixel as target tissue, soft tissue, or background. Target tissues include the femoral and tibial regions. nnU-Net is an adaptive medical image segmentation framework that can automatically adjust the network structure and preprocessing parameters based on dataset characteristics. Since endoscopic images often contain interfering elements such as blood, fat, synovium, and instruments, semantic segmentation can accurately separate the target structural regions requiring navigation, eliminating the influence of non-rigid or temporary objects. The significance of semantic segmentation lies in the fact that it allows subsequent 3D reconstruction to focus only on stable anatomical structures, thereby improving the robustness and mapping accuracy of the SLAM system.

[0039] It should be noted that, in response to the challenges of the complex environment of knee arthroscopy, the embodiments of this disclosure simultaneously perform depth estimation and semantic segmentation processing on the extracted target images, and obtain the depth information of each frame image and the precise semantic boundaries of the femoral and tibial regions, respectively. This solves the problem of the lack of depth perception in arthroscopy and the difficulty in recognizing bone structures in complex environments, and lays the data foundation for subsequent three-dimensional reconstruction of bone tissue alone.

[0040] Regarding step S103 above, this embodiment of the disclosure can construct a target SLAM model based on multiple frames of depth images and multiple frames of semantic segmentation images, and extract the three-dimensional semantic point cloud of the target tissue region from the target SLAM model to obtain the source point cloud. The target SLAM model refers to a three-dimensional map constructed in real time based on arthroscopic image sequences, which simultaneously includes the motion trajectory of the arthroscope and the three-dimensional structure of the femur and tibia in the scene. Furthermore, the construction process can utilize a mask generated from the semantic segmentation results to filter soft tissue, retaining only the depth information of bone tissue for three-dimensional reconstruction, thereby stably constructing a point cloud map in environments with weak textures and complex lighting.

[0041] The source point cloud is a set of three-dimensional points with semantic labels extracted from the target SLAM model. Each point contains spatial coordinates and a semantic category, such as femur or tibia. The point cloud represents the bone surface found during surgery. It is characterized by moderate data volume, real-time arthroscopic pose information, and the possibility of noise and point cloud missing due to synovial occlusion. The source point cloud is an input for subsequent registration operations.

[0042] Regarding step S104 above, this embodiment of the present disclosure can extract the three-dimensional semantic point cloud of the target tissue region from a three-dimensional digital twin model pre-constructed based on the medical images of the target region to obtain the target point cloud, and obtain pre-labeled navigation targets from the three-dimensional digital twin model. The three-dimensional digital twin model refers to a three-dimensional digital model created in a computer that corresponds one-to-one with the anatomical structure of the patient's knee joint. It is constructed by acquiring CT or MRI images of the patient's knee joint before surgery or in other pre-constructed methods, and generating a personalized, high-precision three-dimensional model through deep learning segmentation and surface reconstruction algorithms. This model contains complete geometric information, such as the femur, tibia, and meniscus, as well as semantic labels and navigation targets pre-labeled by sports medicine physicians, resulting in a surgical navigation tunnel containing tunnel point planning information, namely the entrance and exit points and axial directions of the femoral and tibial tunnels. In a virtual surgical simulator or teaching scenario, the three-dimensional digital twin model can be a standardized model constructed based on typical cases, and the navigation target is a predefined ideal tunnel location.

[0043] Furthermore, the surgical navigation tunnel is pre-marked by experienced sports medicine physicians on the medial wall of the lateral femoral condyle and the anteromedial region of the tibial plateau, and the tunnel diameter and direction are determined and saved as a planned path file for storage. This allows the relevant information of the surgical navigation tunnel to be extracted in intraoperative or other scenarios.

[0044] The target point cloud is a set of three-dimensional points on the bone surface extracted from the three-dimensional digital twin model that are semantically consistent with the source point cloud. It is also a set of femoral point clouds and tibial point clouds. The target point cloud has the characteristics of high precision, completeness and noise-free, and is an ideal reference for registration.

[0045] It should be noted that the embodiments disclosed herein obtain a personalized digital twin model of the patient's knee joint and precise tunnel planning through preoperative MRI / CT reconstruction, providing high-precision benchmark data for intraoperative navigation and avoiding intraoperative X-ray radiation and additional trauma.

[0046] Regarding step S105 above, this embodiment of the disclosure can perform registration processing based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model. Registration refers to finding a spatial transformation relationship, such as rigid body transformation, including rotation and translation, so that the reconstructed femoral and tibial point clouds during surgery achieve optimal alignment with the corresponding bone surfaces in the preoperative model after transformation. It should be noted that due to the limited field of view in knee arthroscopy, typically only a portion of the bone surface can be seen, and synovium and blood can cause obstruction, making point cloud matching difficult. Therefore, the registration process can be divided into two stages: coarse registration and fine registration. The purpose of coarse registration is to roughly align the source point cloud to the vicinity of the target point cloud, providing a good initial value for fine registration and avoiding ICP getting trapped in local optima. Fine registration further optimizes the transformation matrix, ensuring that the error between point clouds reaches the sub-millimeter level, meeting the accuracy requirements for tunnel positioning in anterior cruciate ligament reconstruction.

[0047] The target coordinate transformation matrix is ​​a homogeneous transformation matrix that includes rotation matrix and translation vector. This transformation matrix establishes the mapping relationship between the intraoperative target SLAM model coordinate system and the preoperative three-dimensional digital twin model coordinate system. It is the core of realizing the projection of the preoperative planned tunnel from the virtual model to the real-time arthroscopic video stream.

[0048] Regarding step S106 above, this embodiment of the present disclosure can transform the navigation target into the target SLAM model based on the target coordinate transformation matrix, and project the navigation target in the target SLAM model onto the current endoscope image plane. After obtaining the target coordinate transformation matrix, its inverse matrix can transform the preoperatively planned femoral tunnel and tibial tunnel, i.e., the surgical navigation tunnel, from the three-dimensional digital twin model coordinate system to the target SLAM model coordinate system, so that it is in the same spatial reference as the femur and tibia reconstructed in real time during surgery. Combined with the current frame arthroscopic camera pose provided by the target SLAM model, i.e., the transformation matrix from the SLAM model coordinate system to the camera coordinate system, and the pre-calibrated camera intrinsic parameter matrix, the three-dimensional tunnel points are projected onto the two-dimensional image plane through perspective projection, generating augmented reality navigation markers superimposed on the real arthroscopic image.

[0049] At this point, users can see on the arthroscopic screen that the virtual tunnel axis, entrance ring, depth scale, and other navigation markers accurately coincide with the actual femoral condyle and tibial plateau, thus intuitively guiding the drill direction of the surgical tools and significantly reducing tunnel positioning deviation.

[0050] In summary, this embodiment of the present disclosure, through the execution of the endoscopic image-based augmented reality surgical navigation method in steps S101 to S106, uses the video stream captured by the endoscope as data input, eliminating the need to install any additional tracking markers on the patient's bone tissue. This avoids the additional trauma and surgical risks associated with interventional markers and also saves the time-consuming steps of marker installation and calibration. Subsequently, addressing the challenges of the complex joint cavity environment, depth estimation and semantic segmentation are simultaneously performed on the extracted endoscopic target images, obtaining the depth information of each frame and the precise semantic boundaries of the femoral and tibial regions, thus solving the problems of lack of depth perception in endoscopes and difficulty in identifying key bone structures in complex environments. Finally, a fusion language is constructed based on multiple frames of depth images and semantically segmented images. The target SLAM model with semantic information enables real-time tracking of endoscopic pose and 3D semantic reconstruction of the internal structure of the joint cavity under markerless conditions, extracting source point clouds. Then, combined with a 3D digital twin model pre-constructed based on the patient's knee joint medical images, the femoral and tibial target point clouds and pre-planned surgical navigation tunnels are extracted. Through the registration processing of source and target point clouds, a coordinate mapping relationship between the real-time scene and the accurate model is established. Finally, based on the coordinate transformation matrix, the planned surgical navigation tunnel is projected onto the current endoscopic image plane, superimposing virtual navigation information onto the real field of view in an augmented reality manner. This allows for a direct and clear view of the precise location of the femoral and tibial tunnels, ultimately achieving a non-invasive, precise, and intuitive surgical navigation effect.

[0051] Please see Figure 4 , Figure 4 yes Figure 2 The flowchart further includes step S103. In some embodiments, the process of constructing a target SLAM model based on multi-frame depth images and multi-frame semantic segmentation images may also include steps S201 to S203: Step S201: Generate a mask image containing only the femoral and tibial regions based on the semantic segmentation images of each frame, so as to invalidate the depth information corresponding to the soft tissue outside the mask image; Step S202: Construct corresponding local point clouds using the mask images and depth images corresponding to each frame of the target image, and use the first frame of the target image as a key frame to construct an initial target SLAM model based on the local point cloud corresponding to the first frame of the target image. Step S203: For each subsequent frame of the target image, the SFM algorithm is used to calculate the correspondence between the local point clouds of the current frame target image and the target images of each historical frame, so as to fuse the local point clouds frame by frame and iteratively update the initial target SLAM model until the iteration is completed, and the target SLAM model is obtained.

[0052] In the above steps, this embodiment of the present disclosure can generate a mask image containing only the target tissue region based on each frame of semantic segmentation image, so as to invalidate the depth information corresponding to other regions outside the mask image. The mask image is a binary image with the same size as the original image, where pixels belonging to the femur or tibia have a value of 1, and other pixels, such as meniscus, synovium, blood, instruments, etc., have a value of 0. Since soft tissues and blood in knee arthroscopy change frequently during arthroscopic movement, incorporating their depth information into 3D reconstruction would severely interfere with modeling rigid bone structures such as the femur and tibia, leading to localization drift in the SLAM system. Therefore, the method for generating the mask is to set the pixels of the bone tissue category in the semantic segmentation result to 1, and the rest to 0. Then, this mask is multiplied pixel by pixel with the corresponding depth image. When the mask value is 1, the original depth value is retained; when the mask value is 0, the depth value is invalidated. Through mask filtering, the system only reconstructs stable bone surfaces, significantly improving the robustness and accuracy of mapping.

[0053] Next, local point clouds are constructed using the mask and depth images corresponding to each frame of the target image. The first frame of the target image is used as the keyframe, and an initial target SLAM model is built based on the local point cloud corresponding to the first frame. The local point cloud refers to a set of 3D points in the camera coordinate system generated from the effective depth map of a single frame. Specifically, for image coordinates, if the depth is valid, it is back-projected into the camera coordinate system using the camera intrinsic parameter matrix K. Each point also has a semantic label, i.e., femur or tibia. The initial SLAM model uses the local point cloud generated in the first frame as the initial map and records the arthroscopic pose of the first frame as the identity matrix. The first frame is set as the keyframe because it is the starting reference for the entire sequence; the poses of all subsequent frames are estimated relative to the first frame.

[0054] Subsequently, for each subsequent frame of the target image, the SFM algorithm is used to calculate the correspondence between the local point clouds of the current frame and the target images of previous frames. This allows for frame-by-frame fusion of the local point clouds, iteratively updating the initial target SLAM model until the iterations are complete, resulting in the target SLAM model. The Structure from Motion (SFM) algorithm is a technique for simultaneously estimating camera motion and 3D structure from an image sequence. In this embodiment, the SFM algorithm first extracts 2D feature points from the current frame, such as SIFT or ORB. These feature points are invariant to rotation, scaling, and illumination changes, and are matched with feature points from the previous keyframe. By matching point pairs, the relative pose of the current frame relative to the previous keyframe (i.e., rotation matrix and translation vector) is estimated using essential matrix factorization or the PnP algorithm. Then, combined with the effective depth map of the current frame, the local point cloud of the current frame is transformed to the world coordinate system and fused with the existing model point cloud.

[0055] Furthermore, the fusion strategy in this embodiment can be as follows: for a newly added point, calculate its Euclidean distance to existing points in the model. If it is less than a threshold, such as 1 mm, update the coordinates of the point using a weighted average, with the weights allocated according to the confidence level of the depth value; otherwise, add it to the model as a new point. In addition, the system determines the pose change between the current frame and the previous keyframe. When the rotation angle exceeds 5 degrees or the translation distance exceeds 5 mm, the current frame is set as a new keyframe for subsequent feature point matching and pose optimization. After iterative processing of all frames, a dense target SLAM model and a complete arthroscopic trajectory are obtained. In knee arthroscopy, due to the gentle curvature change of the bone surface and the relatively few feature points, the SFM algorithm can still maintain stable tracking by relying on the strong constraints of depth information.

[0056] It should be noted that the embodiments of this disclosure filter out non-bone tissue interference by using semantic masking, and only perform SLAM reconstruction on the point clouds of the femur and tibia, which significantly improves the mapping quality and positioning accuracy in the complex environment of arthroscopy, and provides a reliable source point cloud for subsequent registration.

[0057] Please see Figure 5 , Figure 5 yes Figure 2 The flowchart further includes step S105. In some embodiments, the process of obtaining the target coordinate transformation matrix between the target SLAM model and the 3D digital twin model through registration processing based on the source point cloud and the target point cloud may further include steps S301 to S302: Step S301: Surface sampling is performed on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points. The transformation matrix is ​​solved based on each pair of target feature points to perform coarse registration processing and obtain the initial coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model. Step S302: Using the initial coordinate transformation matrix as the initial value for iteration, the ICP algorithm is used to iteratively optimize the point cloud matching error for fine registration, thereby obtaining the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model.

[0058] In the above steps, embodiments of this disclosure can perform surface sampling on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points, and perform coarse registration processing based on the transformation matrix of each pair of target feature points to obtain the initial coordinate transformation matrix. Surface sampling refers to extracting representative feature points from the point cloud, rather than using the entire point cloud for registration, to reduce computational load and improve robustness. The purpose of coarse registration is to quickly find an approximate rigid body transformation so that the intraoperative point cloud roughly moves to the vicinity of the preoperative model point cloud. Due to the limited field of view of arthroscopy, the reconstructed femoral and tibial point clouds during surgery are usually only partial surfaces, for example, only the lateral femoral condyle and the anterior edge of the tibial plateau can be seen, and there are holes caused by synovial obstruction. Directly using the entire point cloud for ICP can easily lead to local optima; therefore, it is necessary to obtain good initial values ​​through anatomically guided feature point matching.

[0059] Next, to meet the high-precision requirements of anterior cruciate ligament reconstruction, the initial coordinate transformation matrix was used as the initial value for iteration. The ICP algorithm was employed to iteratively optimize the point cloud matching error for fine registration, resulting in the target coordinate transformation matrix. The Iterative Closest Point (ICP) algorithm is a classic point cloud registration method. Its core idea is to transform the source point cloud to the target point cloud coordinate system according to the current transformation matrix in each iteration, find the nearest neighbor of each point in the source point cloud in the target point cloud, construct corresponding point pairs, and then update the transformation matrix by minimizing the sum of squared Euclidean distances of all corresponding point pairs.

[0060] Specifically, the algorithm can set a maximum number of iterations, such as 200, and a convergence threshold of 0.01 mm. In the k-th iteration, for each point in the transformed source point cloud, the algorithm finds the point in the target point cloud with the closest Euclidean distance, constructs a set of corresponding point pairs, and then solves to minimize the objective function to obtain the optimal rotation matrix and translation vector. The coordinate transformation matrix is ​​then updated, and the source point cloud is updated after the transformation. This process is repeated. The algorithm stops when the average distance change is less than the threshold or the maximum number of iterations is reached, finally obtaining the registered target coordinate transformation matrix.

[0061] It should be noted that the embodiments disclosed herein provide good initial values ​​through coarse registration, followed by fine registration via ICP, which balances convergence speed and registration accuracy, making it suitable for the real-time requirements of knee arthroscopic surgery.

[0062] Please see Figure 6 , Figure 6 yes Figure 5 The flowchart further includes step S301. In some embodiments, during the process of sampling the surface of the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points, and solving the transformation matrix based on each pair of target feature points to perform coarse registration processing and obtain the initial coordinate transformation matrix between the target SLAM model and the 3D digital twin model, steps S401 to S404 may also be included: Step S401: Obtain a set of landmark anatomical feature points for the pre-defined anatomical structure adapted to the knee joint, and extract anatomical feature points that correspond one-to-one with the set of landmark anatomical feature points from the source point cloud and the target point cloud respectively. Step S402: Calculate the geometric saliency score, anatomical variability score, and intraoperative visibility score for each anatomical feature point, and obtain the matching weight for each anatomical feature point by weighted summation based on the geometric saliency score, anatomical variability score, and intraoperative visibility score. Step S403: Calculate the weighted similarity of each anatomical feature point between the source point cloud and the target point cloud based on the matching weight, so as to filter out multiple pairs of matching target feature points; Step S404: Construct a weighted error function, substitute the matching weights of each pair of target feature points into the weighted error function, solve for the rigid body transformation matrix with the goal of minimizing the weighted error function, and verify the rigid body transformation matrix by anatomical constraints to obtain the initial coordinate transformation matrix.

[0063] In the above steps, embodiments of this disclosure can obtain a pre-defined set of landmark feature points adapted to the anatomical structure of the target region, and extract anatomical feature points corresponding one-to-one with the set of landmark feature points from the source point cloud and the target point cloud, respectively. The set of landmark feature points refers to a group of key points that are clearly defined in knee joint anatomy, have relatively stable positions in different individuals, and are easily identifiable in the arthroscopic field of view. For example, it can include 12 feature points: the apex of the medial femoral condyle, the apex of the lateral femoral condyle, the medial apex of the intercondylar spine of the tibia, the lateral apex of the intercondylar spine of the tibia, the apex of the tibial tuberosity, the midpoint of the anterior border of the intercondylar fossa, the midpoint of the posterior border of the intercondylar fossa, the midpoint of the anterior medial border of the tibial plateau, the midpoint of the anterior lateral border of the tibial plateau, the midpoint of the posterior border of the medial tibial plateau, and the apex of the posterior border of the medial femoral condyle. These feature points are divided into three categories according to their geometric shape: apex type, which is the local curvature maxima and convex points; edge midpoint type, which is the midpoint of the intersection line of two curved surfaces; and concave type, which is the local curvature minima and concave points. In the knee joint, these feature points are directly related to tunnel positioning. For example, the midpoint of the anterior border of the intercondylar fossa determines the location of the femoral tunnel exit. Therefore, their accurate extraction is the key to successful registration.

[0064] Next, the geometrical significance score, anatomical variability score, and intraoperative visibility score for each anatomical feature point are calculated. The matching weight for each anatomical feature point is then obtained by weighted summation of these scores. The geometrical significance score represents the uniqueness of the feature point in its local geometry, determined by the variation in Gaussian curvature within its surrounding neighborhood. For example, the femoral condyle apex has a large curvature variation and a high score; the midpoint of the anterior edge of the tibial plateau has a smaller curvature variation and a relatively low score. The anatomical variability score represents the consistency of the feature point's position across different patients, obtained by normalizing the standard deviation of the point's position across a large number of knee joint samples. For example, the midpoint of the anterior and posterior edges of the intercondylar fossa of the femur has a smaller variability and a high score; the tibial tuberosity apex has a larger variability and a lower score. The intraoperative visibility score is pre-calibrated based on arthroscopic surgical experience and represents the probability of the feature point appearing in standard knee arthroscopic approaches, such as the anteromedial and anterolateral approaches. For example, the posterior edge apex of the medial femoral condyle is difficult to expose under arthroscopy and has a low score. The final matching weights are assigned preset coefficients to the geometric significance score, anatomical variability score, and intraoperative visibility score, for example, 0.4, 0.3, and 0.3 respectively. These are then weighted and summed to obtain the matching score. Feature points with high weights play a more important role in subsequent matching and registration because they are more stable, easier to identify, and more discriminative. In knee arthroscopy, the midpoint of the anterior and posterior margins of the intercondylar fossa of the femur is usually assigned the highest weight due to its strong geometric features, low variability, and clear visibility under the microscope, thus dominating the registration process.

[0065] Subsequently, weighted similarity scores for each anatomical feature point between the source and target point clouds are calculated based on matching weights to filter out multiple pairs of matching target feature points. Specifically, for each feature point extracted from the source point cloud, the descriptor cosine similarity between it and each candidate corresponding feature point in the target point cloud is calculated. The descriptor is a 64-dimensional SHOT vector, and the weighted similarity is calculated as follows: ,in The matching weights for the source feature points, Here, θ represents the matching weights for the target candidate feature points, and θ is the angle between the two descriptor vectors. , and These are the source descriptor and target descriptor vectors, respectively.

[0066] Furthermore, in this embodiment of the present disclosure, a bidirectional matching verification strategy is adopted. First, for each point in the source point cloud, the point with the highest weighted similarity is found in the target point cloud to form a candidate matching pair. Then, for each point in the target point cloud, the point with the highest weighted similarity is found in the source point cloud. Only pairs with consistent bidirectional matching results and similarity exceeding a preset threshold, such as 0.7, are retained as valid matches. This bidirectional verification can effectively eliminate false matches caused by local similarity of point clouds, such as cross-matches that occur when the medial and lateral femoral condyles have similar shapes.

[0067] Finally, a weighted error function is constructed. The matching weights of each pair of target feature points are substituted into the weighted error function, and the rigid body transformation matrix is ​​obtained by minimizing the weighted error function. The rigid body transformation matrix is ​​then verified by anatomical constraints to obtain the initial coordinate transformation matrix. The weighted error function considers the weights of each pair of matched feature points and has the following form: ; Where n is the total number of matching feature points. Let be the matching weight of the i-th pair of matching feature points, R be the rotation matrix to be determined (e.g., a 3×3 orthogonal matrix), and t be the translation vector to be determined (e.g., a 3×1 column vector). Let i be the three-dimensional coordinates of the i-th feature point in the source point cloud. Given the 3D coordinates of the feature points in the target point cloud that match it, this function assigns greater influence to feature points with higher weights. The standard method for solving this least squares problem is based on weighted singular value decomposition. First, calculate the weighted centroids: ; ; in, The weighted centroid coordinates of the source feature points. We obtain the weighted centroid coordinates of the target feature points, then decenter each point and calculate the weighted covariance matrix: ; Where H is a 3×3 weighted covariance matrix, and the superscript T denotes matrix transpose, the singular value decomposition of H is: H = UΣV^T, where U and V are orthogonal matrices, and Σ is a diagonal matrix. Then the rotation matrix R = VU^T, and the translation vector is... Finally, verify that the determinant of R is 1; if it is -1, it needs to be corrected. Anatomical constraint verification includes checking whether the relative positions of the transformed femur and tibia are within the physiological range, such as ensuring that there should be no perforation, and whether the tunnel direction is at a reasonable angle to the anatomical axis of the knee joint.

[0068] Finally, after successful verification, the initial transformation matrix was obtained. Then, the ICP algorithm is used to... As the initial value, This is the initial pose obtained by sampling points, as described in the coarse registration description above. It is the initial state obtained by initially aligning corresponding points collected from the point cloud, serving as the initial state for subsequent ICP fine registration. The transformation matrix is ​​then iteratively optimized. This minimizes the distance between the source point cloud and the target point cloud. It is the process registration matrix, the transformation matrix from the origin point cloud to the target point cloud generated during the ICP iteration process, which is also the matrix we ultimately want to optimize. The transformation matrix is ​​obtained. After registration, the target coordinate transformation matrix between the target SLAM model coordinate system and the 3D digital twin model coordinate system is obtained. .

[0069] It should be noted that the embodiments disclosed herein make feature point matching more robust by introducing matching weights and anatomical priors, which is especially suitable for situations where the point cloud is partially missing in knee arthroscopy, such as when the intercondylar fossa region is incomplete due to synovial occlusion.

[0070] Please see Figure 7 , Figure 7 yes Figure 6 The flowchart further includes step S401. In some embodiments, the process of extracting anatomical feature points corresponding one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud respectively may also include steps S501 to S503: Step S501: Based on the geometric type of each feature point in the set of iconic anatomical feature points, the source point cloud is processed using the corresponding three-dimensional Hough transform kernel to complete the coarse localization of the candidate anatomical feature points. Step S502: Calculate the three-dimensional shape descriptor of each candidate anatomical feature point and perform bidirectional matching verification with the three-dimensional shape descriptor of the corresponding anatomical feature point in the target point cloud to complete the fine screening of candidate anatomical feature points. Step S503: The weighted random sampling consensus algorithm is used to process the selected feature point pairs to remove mismatched points generated during the feature point matching process, and obtain multiple pairs of matched anatomical feature points.

[0071] In the above steps, embodiments of this disclosure can process the source point cloud using the corresponding 3D Hough transform kernel according to the geometric type of each feature point in the set of landmark feature points, in order to complete the coarse localization of candidate anatomical feature points. The 3D Hough transform is a voting mechanism used to detect point sets with specific geometric shapes in a point cloud, specifically for vertex-type feature points.

[0072] Furthermore, for features such as the apex of the medial and lateral femoral condyles and the apex of the tibial tuberosity, a spherical Hough transform kernel is used. Each point p in the point cloud is used as the center of a sphere with a radius R = 8 mm. The number of points falling within the sphere is counted. The point cloud around an ideal convex point will be densely distributed on the sphere surface, resulting in a high number of votes. By suppressing the vote count using non-maximum values, local maxima are selected as candidate convex points. For edge midpoint features, such as the midpoint of the anterior and posterior edges of the intercondylar fossa and the midpoint of the anterior edge of the tibial plateau, a randomized Hough transform (RHT) is used to extract straight line segments. This involves randomly sampling two points, determining the straight line parameters, counting the number of points that conform to the straight line, and retaining line segments with a length exceeding a first preset length and a fitting error less than a second preset length. The first preset length can be 10 mm, and the second preset length can be 0.5 mm. Then, the midpoint of the line segment is calculated. For concave features, such as the apex of the posterior edge of the medial femoral condyle, a concave spherical kernel is used. Similar to the spherical kernel, but the normal vector of the point inside the sphere must point outwards, i.e., the local surface is concave. The number of points meeting the conditions is counted.

[0073] The purpose of coarse localization is to quickly filter out a small number of candidate points in the entire point cloud, greatly reducing the computational burden of subsequent fine matching. In knee arthroscopy, because the curvature of the bone surface changes gradually, the parameters of the Hough transform, such as the radius, need to be adapted according to the typical size of the femoral condyle, which is about 20-30 mm.

[0074] Next, a three-dimensional shape descriptor for each candidate anatomical feature point is calculated, and bidirectional matching verification is performed with the three-dimensional shape descriptor of the corresponding anatomical feature point in the target point cloud to complete the fine screening of candidate anatomical feature points. The three-dimensional shape descriptor is used to quantify the local geometry around the feature point; in this embodiment, the SHOT descriptor (Signature of Histograms of Orientations) is used. The calculation steps are as follows: First, with feature point p as the center, select the support radius R according to the feature point type, for example, 8mm for vertices, 12mm for edges, and 10mm for depressions. In the knee joint, the support radius needs to cover sufficient anatomical structures around the feature point. For example, the midpoint of the anterior edge of the intercondylar fossa needs to include the morphological information of the bone walls on both sides of the intercondylar fossa. Establish a local reference frame (LRF) within the support radius. Calculate the covariance matrix of the neighborhood points, obtain three orthogonal directions using principal component analysis, and determine the Z-axis direction using normal vector sign consistency constraints. Divide the local space into 32 spatial bins, for example, the radial direction is divided into 2 layers, the azimuth angle into 8 equal parts, and the polar angle into 2 equal parts. Within each spatial bin, calculate the histogram of the normal vector direction of the internal points, quantizing the normal vector direction into 11 direction bins. Obtain the original histogram of 32×11=352 dimensions, and then reduce the dimension to 64 dimensions through principal component analysis (PCA) to obtain the final descriptor vector.

[0075] The descriptors possess rotational invariance and partial translational invariance; even with noise and slight deformation in the point cloud, similar feature points will have similar descriptors. During fine-tuning, a descriptor is calculated for each candidate point, and then compared with the cosine similarity of the descriptor to the corresponding anatomical point in the target point cloud. Two-way matching verification is used, retaining only points that match consistently and have a similarity higher than a threshold (e.g., above 0.7) as the final extracted anatomical feature points. In the knee joint, the descriptor for the intercondylar fossa region of the femur exhibits high specificity and can effectively distinguish between the medial and lateral condyles.

[0076] Finally, a weighted random sampling consensus algorithm is used to process the selected feature point pairs to eliminate mismatches generated during the feature point matching process, resulting in multiple pairs of matched anatomical feature points. The weighted random sampling consensus algorithm is an improvement on the traditional RANSAC algorithm. Let the initial set of matching pairs be M, and each matching pair have a weight. The algorithm process includes setting the maximum number of iterations to 100. In each iteration, based on weighted probabilities, the higher the weight of the matching pair, the greater the probability of it being selected. Four non-coplanar matching pairs are selected without replacement, and the rigid body transformation matrix is ​​calculated using these four matching pairs. Then, for each matching pair, calculate... Reprojection error under transformation , that is, the Euclidean distance between the transformed source point and the target point, where and These represent the rotation and translation components of the candidate transformation matrix, respectively. The inlier error threshold is dynamically set based on the feature point weights: high-weight points (greater than 0.8) require an error ≤1.5mm; medium-weight points (between 0.6 and 0.8) require an error ≤2.0mm; and low-weight points (less than 0.6) require an error ≤2.5mm. Subsequently, the inlier points are counted, and the inlier score is calculated: Score = Σ That is, the sum of the inlier weights. If the score is greater than the current best score, the best transformation matrix and the best inlier set are updated. If the score exceeds 80% of the total weight or the iteration reaches the preset number of iterations, the iteration is terminated early.

[0077] Finally, the final transformation matrix is ​​recalculated using weighted least squares based on all inliers, and the matching pairs corresponding to the inliers are output as multiple pairs of matched anatomical feature points. This algorithm, through weighted guided sampling and dynamic thresholding, can stably solve for the correct transformation even with a high proportion of mismatches. In knee arthroscopy, synovial occlusion may prevent some feature points from being detected; weighted RANSAC can still achieve correct registration using the remaining high-weight feature points, demonstrating strong robustness.

[0078] It should be noted that the embodiments of this disclosure achieve high-accuracy identification of knee joint anatomical feature points in sparse and noisy point clouds by using multi-level Hough transform coarse localization, shape descriptor fine matching, and weighted RANSAC to eliminate mismatches, providing high-quality corresponding point pairs for subsequent registration.

[0079] In some embodiments, step S106 above, in which the surgical navigation tunnel in the target SLAM model is projected onto the image plane of the current endoscope, may further include step S601: Step S601: Project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope. Use different graphic symbols to represent the entrance, axis and depth reference information of the surgical navigation tunnel respectively, and draw it in a semi-transparent blending mode so that the drawn navigation symbols do not obscure the real tissue image of the knee joint acquired by the endoscope.

[0080] Furthermore, in this embodiment of the present disclosure, the projection process can be divided into two steps. First, the planned surgical navigation tunnel is transformed from the coordinate system of the three-dimensional digital twin model to the coordinate system of the target SLAM model. Since the path of the surgical navigation tunnel consists of a series of three-dimensional points, each point... Transformation matrix of target coordinates Mapping the inverse matrix to the target SLAM model coordinate system: ,in The three-dimensional coordinates of the surgical navigation tunnel in the target SLAM model.

[0081] Then, the arthroscopic camera pose of the current frame is used by the target SLAM model. That is, the transformation matrix from the target SLAM model coordinate system to the endoscope camera coordinate system, and the pre-calibrated camera intrinsic parameter matrix K, which projects each 3D tunnel point of the surgical navigation tunnel onto the image plane, as shown in the formula. , This is the camera extrinsic parameter matrix for the current frame, used to transform points in the target SLAM model coordinate system to the camera coordinate system. These are the homogeneous coordinates of the final image.

[0082] Finally, all projection points are connected in their original order to form a line, creating the superposition of the virtual tunnel on the arthroscopic image. To enhance visualization, different graphic markers can be used; for example, a blue ring marks the tunnel entrance, a green dashed line represents the tunnel axis, and yellow scale lines mark every 2 millimeters of depth. During rendering, a semi-transparent blending function in OpenGL, such as glBlendFunc, is used to semi-transparently overlay the virtual markers with the real bone tissue, ensuring that the user can simultaneously see virtual information and real anatomical structures, avoiding obstruction of key areas such as the intercondylar fossa and tibial plateau. In knee arthroscopy, because the arthroscopic field of view is usually small, the size and density of the projection markers need to be automatically adjusted according to the field of view to avoid excessive density affecting observation.

[0083] Please see Figure 8 , Figure 8 yes Figure 2 The flowchart further includes steps S106. In some embodiments, after projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope, steps S701 to S702 may also be included: Step S701: The spatial pose information of the endoscope and surgical tools is collected in real time through optical positioning equipment, and combined with the pre-completed coordinate system calibration results to calculate the spatial pose of the surgical tools in the target SLAM model. Step S702: Calculate the spatial deviation between the surgical tool and the surgical navigation tunnel, and superimpose the spatial deviation onto the image plane of the endoscope for real-time display.

[0084] In the above steps, this embodiment of the present disclosure can acquire the spatial pose information of the endoscope and surgical tools in real time using an optical positioning device, and combine it with the pre-completed coordinate system calibration results to calculate the spatial pose of the surgical tools in the target SLAM model. The optical positioning device is a real-time three-dimensional measurement system based on infrared or visible light, which can track the multi-degree-of-freedom pose of optical markers mounted on arthroscopy and surgical tools, such as drills and guides, to indicate their position and orientation. Through pre-calibration, an optical tracker can be pre-installed on the endoscope, and the transformation relationship between the tracker coordinate system and the endoscope camera coordinate system can be calibrated using an eye-to-hand calibration method via a calibration plate. Then, an optical tracker is pre-installed on the surgical instrument. Using an optical probe, key geometric feature points of the surgical instrument are calibrated, and the transformation relationship between the tracker coordinate system and the surgical instrument coordinate system is obtained. .

[0085] Since the target SLAM model provides the camera pose for the current frame. This refers to the transformation matrix from the target SLAM model coordinate system to the camera coordinate system. Additionally, the optical positioning device outputs the tracker's pose in the world coordinate system in real time. and The pose of the surgical tool's end effector in the target SLAM model coordinate system can be obtained through a combination transformation: ; in, The transformation matrix from the world coordinate system to the endoscope tracker coordinate system is provided in real time by the optical positioning instrument. The transformation matrix from the world coordinate system to the surgical tool tracker coordinate system is provided in real time by the optical positioning instrument. The transformation matrix from the surgical tool tracker coordinate system to the endoscope tracker coordinate system is obtained through calibration. The transformation matrix from the surgical tool tracker coordinate system to the tool end is obtained through calibration. This is the transformation matrix from the target SLAM model coordinate system to the endoscope camera coordinate system, provided in real-time by the target SLAM model. The superscript -1 indicates the inverse of the matrix. The derivation of this formula is based on a coordinate system transformation chain, that is, the coordinates of the surgical tool's end-point point change from the surgical tool coordinate system, to the tracker coordinate system, to the world coordinate system, to the endoscope tracker coordinate system, to the endoscope camera coordinate system, and back to the target SLAM model coordinate system. Through this transformation, the real-time position of the surgical tool can be unified into the target SLAM model coordinate system, thus enabling spatial comparison with the pre-planned tunnel. In anterior cruciate ligament reconstruction of the knee joint, it is usually necessary to track the drill bit and tunnel locator, displaying their deviation from the planned axis in real time.

[0086] Subsequently, the spatial deviation between the surgical instrument and the surgical navigation tunnel is calculated and superimposed onto the endoscopic image plane for real-time display. First, key points of the navigation tunnel, such as the entry point and axial direction, and the current position of the surgical instrument's tip are transformed to the same coordinate system. Then, the shortest distance *d* from the surgical instrument's tip to the tunnel axis and the depth deviation *Δs* along the axial direction are calculated; these are the directed distances between the projection point of the tool's tip and the tunnel entry point. These two parameters reflect the degree of deviation of the drill or locator from the planned path. *d* and *Δs* are displayed digitally in real-time at the image edge and color-coded, for example, green for deviation <1mm, yellow for 1-3mm, and red for >3mm. Simultaneously, a virtual projection point of the surgical instrument's tip, such as a small dot, and an indicator line showing the angle between the tool's axis and the tunnel axis can be plotted on the image. Furthermore, in knee arthroscopy, since the arthroscopic field of view and the tool operation direction are often off-axis, this deviation is particularly important. Users can adjust the drill direction in real time based on this visual feedback to ensure that the tunnel position and direction are consistent with the preoperative plan, control the positioning error within 2mm, and significantly reduce the revision rate.

[0087] It should be noted that the embodiments disclosed herein achieve real-time navigation of surgical tools by fusing optical tracking with the SLAM coordinate system, and intuitively quantify and feed back spatial deviations to the doctor, thereby further improving the accuracy and safety of knee arthroscopy.

[0088] Furthermore, in virtual surgical simulator scenarios, optical positioning devices can be replaced with virtual input devices, such as force feedback handles, whose poses are simulated by software, and the position of the tool in virtual space can also be calculated. In remote teaching scenarios, the poses of the expert's operating tools can be transmitted over the network and displayed as guidance information in the field of vision of primary care physicians, which is particularly suitable for surgeries such as knee arthroscopy that are difficult to operate and have a long learning curve.

[0089] Please see Figure 9 This disclosure also provides an augmented reality surgical navigation system based on endoscopic images, which can implement the above-described augmented reality surgical navigation method based on endoscopic images. The augmented reality surgical navigation system based on endoscopic images includes: The image acquisition module 901 is used to acquire the video stream captured by the endoscope at the knee joint and extract multiple frames of target images from the video stream. The image processing module 902 is used to perform depth estimation processing on each frame of target image to obtain the corresponding depth image of each frame, and to perform semantic segmentation processing on the femoral region and tibia region in each frame of target image to obtain the corresponding semantic segmentation image of each frame. The SLAM model building module 903 is used to build a target SLAM model based on multi-frame depth images and multi-frame semantic segmentation images, and extract the three-dimensional semantic point clouds of the femoral and tibial regions from the target SLAM model to obtain the source point cloud. The information acquisition module 904 is used to extract the three-dimensional semantic point cloud of the femoral region and the tibia region from the three-dimensional digital twin model pre-constructed based on the medical image of the knee joint to obtain the target point cloud, and to obtain the surgical navigation tunnel pre-marked between the femoral region and the tibia region from the three-dimensional digital twin model. The registration module 905 is used to perform registration processing based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; The navigation display module 906 is used to convert the surgical navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope.

[0090] In summary, the endoscopic image-based augmented reality surgical navigation system executes the endoscopic image-based augmented reality surgical navigation method described in the above embodiments. Using the video stream captured by the endoscope as data input, it eliminates the need for any additional tracking markers on the patient's bone tissue, avoiding the additional trauma and surgical risks associated with invasive markers, and saving the time-consuming steps of marker installation and calibration. Subsequently, addressing the challenges of the complex joint cavity environment, depth estimation and semantic segmentation are simultaneously performed on the extracted endoscopic target images, acquiring depth information and precise semantic boundaries of the femoral and tibial regions for each frame. This solves the problems of lack of depth perception in endoscopes and difficulty in identifying key bone structures in complex environments. Finally, a fusion model is constructed based on multiple frames of depth images and semantically segmented images. The target SLAM model, which integrates semantic information, enables real-time tracking of endoscopic pose and 3D semantic reconstruction of the internal structure of the joint cavity under markerless conditions, extracting source point clouds. Then, combined with a pre-constructed 3D digital twin model based on the patient's knee joint medical images, the femoral and tibial target point clouds are extracted along with pre-planned surgical navigation tunnels. Through registration processing of the source and target point clouds, a coordinate mapping relationship is established between the real-time scene and the precise model. Finally, based on the coordinate transformation matrix, the planned surgical navigation tunnel is projected onto the current endoscopic image plane, superimposing virtual navigation information onto the real field of view in an augmented reality manner. This allows for a direct and clear view of the precise location of the femoral and tibial tunnels, ultimately achieving a non-invasive, precise, and intuitive surgical navigation effect.

[0091] The specific implementation of this endoscopic image-based augmented reality surgical navigation system is basically the same as the specific embodiment of the endoscopic image-based augmented reality surgical navigation method described above, and will not be repeated here. While meeting the requirements of the embodiments of this disclosure, the endoscopic image-based augmented reality surgical navigation system may also be equipped with other functional modules to implement the endoscopic image-based augmented reality surgical navigation method described above.

[0092] This disclosure also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned augmented reality surgical navigation method based on endoscopic images. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0093] Please see Figure 10 , Figure 10 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes: The processor 1001 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this disclosure. The memory 1002 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 1002 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 1002 and is called and executed by the processor 1001 to execute the augmented reality surgical navigation method based on endoscopic images according to the embodiments of this disclosure. Input / output interface 1003 is used to implement information input and output; The communication interface 1004 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.). Bus 1005 transmits information between various components of the device (e.g., processor 1001, memory 1002, input / output interface 1003, and communication interface 1004); The processor 1001, memory 1002, input / output interface 1003 and communication interface 1004 are connected to each other within the device via bus 1005.

[0094] This disclosure also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described augmented reality surgical navigation method based on endoscopic images.

[0095] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0096] The embodiments described in this disclosure are for the purpose of more clearly illustrating the technical solutions of this disclosure and do not constitute a limitation on the technical solutions provided by this disclosure. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by this disclosure are also applicable to similar technical problems.

[0097] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this disclosure, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0098] The system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0099] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0100] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in this disclosure and the foregoing drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequence. It should be understood that such orders can be interchanged where appropriate so that embodiments of this disclosure described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover a non-exclusive inclusion, for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0101] It should be understood that in this disclosure, "at least one item" means one or more, and "more than one" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0102] In the embodiments provided in this disclosure, it should be understood that the disclosed systems and methods can be implemented in other ways. For example, the system embodiments described above are merely illustrative, and the division of the units described above is only a logical functional division. In actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between systems or units may be electrical, mechanical, or other forms.

[0103] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0104] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0105] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this disclosure, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this disclosure. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0106] The preferred embodiments of the present disclosure have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present disclosure. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and spirit of the present disclosure shall be within the scope of the claims of the present disclosure.

Claims

1. An augmented reality surgery navigation method based on an endoscope image, characterized by, include: Acquire a video stream captured by an endoscope at the knee joint, and extract multiple frames of target images from the video stream; Depth estimation processing is performed on the target image in each frame to obtain the corresponding depth image of each frame, and semantic segmentation processing is performed on the femoral region and tibia region in the target image in each frame to obtain the corresponding semantic segmentation image of each frame; A target SLAM model is constructed based on multiple frames of the depth image and multiple frames of the semantic segmentation image, and the source point cloud is obtained by extracting the three-dimensional semantic point cloud of the femoral region and the tibia region from the target SLAM model. The target point cloud is obtained by extracting the three-dimensional semantic point cloud of the femoral region and tibia region from the three-dimensional digital twin model pre-constructed based on the medical image of the knee joint, and the surgical navigation tunnel pre-marked between the femoral region and tibia region is obtained from the three-dimensional digital twin model. Based on the source point cloud and the target point cloud, a registration process is performed to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; Based on the target coordinate transformation matrix, the surgical navigation tunnel is transformed into the target SLAM model, and the surgical navigation tunnel in the target SLAM model is projected onto the image plane of the current endoscope.

2. The endoscope image-based augmented reality surgery navigation method according to claim 1, characterized by, The construction of the target SLAM model based on multiple frames of the depth image and multiple frames of the semantic segmentation image includes: Based on the semantic segmentation images of each frame, a mask image containing only the femoral and tibial regions is generated to invalidate the depth information corresponding to the soft tissue outside the mask image; A corresponding local point cloud is constructed using the mask image and depth image corresponding to the target image in each frame, and an initial target SLAM model is constructed based on the local point cloud corresponding to the target image in the first frame, using the target image in the first frame as a key frame. For each subsequent frame of the target image, the SFM algorithm is used to calculate the correspondence between the local point clouds of the target image in the current frame and the target images in previous frames, so as to fuse the local point clouds frame by frame and iteratively update the initial target SLAM model until the iteration is completed, and the target SLAM model is obtained.

3. The endoscope image-based augmented reality surgery navigation method of claim 1, wherein, The registration process based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the 3D digital twin model includes: Surface sampling is performed on the source point cloud and the target point cloud respectively to obtain multiple pairs of matching target feature points. Based on each pair of target feature points, the transformation matrix is ​​solved to perform coarse registration processing, and the initial coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model is obtained. Using the initial coordinate transformation matrix as the initial value for iteration, the ICP algorithm is used to iteratively optimize the point cloud matching error for fine registration, thereby obtaining the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model.

4. The endoscope image-based augmented reality surgery navigation method according to claim 3, characterized by, The process involves surface sampling of the source point cloud and the target point cloud to obtain multiple pairs of matching target feature points. A coarse registration process is then performed based on the transformation matrix of each pair of target feature points to obtain the initial coordinate transformation matrix between the target SLAM model and the 3D digital twin model. This includes: Obtain a pre-defined set of iconic anatomical feature points that are adapted to the anatomical structure of the knee joint, and extract anatomical feature points that correspond one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud, respectively. Calculate the geometric saliency score, anatomical variability score, and intraoperative visibility score for each anatomical feature point, and obtain the matching weight for each anatomical feature point by weighted summation based on the geometric saliency score, the anatomical variability score, and the intraoperative visibility score; Based on the matching weight, the weighted similarity of each of the corresponding anatomical feature points between the source point cloud and the target point cloud is calculated to filter out multiple pairs of matching target feature points; A weighted error function is constructed, and the matching weights of each pair of target feature points are substituted into the weighted error function. The rigid body transformation matrix is ​​obtained by minimizing the weighted error function. The rigid body transformation matrix is ​​then verified by anatomical constraints to obtain the initial coordinate transformation matrix.

5. The endoscope image-based augmented reality surgery navigation method according to claim 4, characterized by, The step of extracting anatomical feature points corresponding one-to-one with the set of iconic anatomical feature points from the source point cloud and the target point cloud respectively includes: Based on the geometric type of each feature point in the set of iconic anatomical feature points, the source point cloud is processed using the corresponding three-dimensional Hough transform kernel to complete the coarse localization of the candidate anatomical feature points. Calculate the three-dimensional shape descriptor of each candidate anatomical feature point and perform bidirectional matching verification with the three-dimensional shape descriptor of the corresponding anatomical feature point in the target point cloud to complete the fine screening of candidate anatomical feature points; The weighted random sampling consensus algorithm is used to process the selected feature point pairs to eliminate mismatched points generated during the feature point matching process, thereby obtaining multiple pairs of matched anatomical feature points.

6. The endoscope image-based augmented reality surgery navigation method of claim 1, wherein, The step of projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope includes: The surgical navigation tunnel in the target SLAM model is projected onto the image plane of the endoscope. Different graphic identifiers are used to represent the entrance, axis, and depth reference information of the surgical navigation tunnel, and a semi-transparent blending mode is used for drawing so that the generated navigation identifiers do not obscure the real tissue image of the knee joint acquired by the endoscope.

7. The augmented reality surgical navigation method based on endoscopic images according to claim 1, characterized in that, After projecting the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope, the method further includes: The spatial pose information of the endoscope and surgical tools is collected in real time by optical positioning equipment, and combined with the pre-completed coordinate system calibration results to calculate the spatial pose of the surgical tools under the target SLAM model. The spatial deviation between the surgical tool and the surgical navigation tunnel is calculated, and the spatial deviation is superimposed on the image plane of the endoscope for real-time display.

8. An augmented reality surgical navigation system based on endoscopic images, characterized in that, include: An image acquisition module is used to acquire a video stream captured by an endoscope at the knee joint and extract multiple frames of target images from the video stream. The image processing module is used to perform depth estimation processing on each frame of the target image to obtain the corresponding depth image of each frame, and to perform semantic segmentation processing on the femoral region and tibia region in each frame of the target image to obtain the corresponding semantic segmentation image of each frame. The SLAM model building module is used to build a target SLAM model based on multiple frames of the depth image and multiple frames of the semantic segmentation image, and extract the three-dimensional semantic point cloud of the femoral region and tibia region from the target SLAM model to obtain the source point cloud; The information acquisition module is used to extract the three-dimensional semantic point cloud of the femoral region and the tibia region from a three-dimensional digital twin model pre-constructed based on the medical image of the knee joint to obtain the target point cloud, and to obtain the surgical navigation tunnel pre-marked between the femoral region and the tibia region from the three-dimensional digital twin model. The registration module is used to perform registration processing based on the source point cloud and the target point cloud to obtain the target coordinate transformation matrix between the target SLAM model and the three-dimensional digital twin model; The navigation display module is used to transform the surgical navigation tunnel into the target SLAM model based on the target coordinate transformation matrix, and project the surgical navigation tunnel in the target SLAM model onto the image plane of the current endoscope.

9. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the augmented reality surgical navigation method based on endoscopic images as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the augmented reality surgical navigation method based on endoscopic images as described in any one of claims 1 to 7.

Images