A markerless anti-occlusion ventricle and intracranial hematoma puncture AR navigation method and system based on monocular vision

Abstract

The application discloses a marker-free anti-occlusion ventricle and intracranial hematoma puncture AR navigation method and system based on monocular vision, and belongs to the technical field of medical augmented reality. The method comprises the following steps: acquiring a preoperative three-dimensional medical image model of a patient; collecting a video in real time by using a monocular camera of a consumer-level device, and constructing a three-dimensional model of the body surface of the patient by using a three-dimensional reconstruction algorithm such as neural rendering; eliminating visual interference such as a surgical drape and strong light reflection by using an optical correction technology; adopting a time sequence hybrid registration mechanism comprising a global locking period, an inertial maintenance period and a local correction period, so as to accurately align the reconstructed model with the image model, and realize continuous tracking when the visual signal is lost; and finally generating an augmented reality view for navigation. The method does not need an external tracker and a surgical marker, and can realize high-precision, anti-occlusion and real-time navigation by using a portable device, so that the cost and use threshold of surgical navigation are significantly reduced, and the method has a very high clinical application value.

Description

Technical Field

[0001] This invention belongs to the field of medical augmented reality technology, specifically, it relates to a markerless anti-occlusion AR navigation method and system for ventricular and intracranial hematoma puncture based on monocular vision. Background Technology

[0002] Ventricular puncture and intracranial hematoma puncture (especially minimally invasive evacuation of hypertensive intracerebral hemorrhage) are the most basic and common emergency procedures in neurosurgery. Currently, clinical practice mainly relies on doctors' experience-based "blind puncture" based on anatomical landmarks (such as the external auditory canal and brow ridge). This method is greatly affected by individual differences (such as ventricular displacement and irregular hematoma morphology) and carries significant risks: for ventricular puncture, it may lead to multiple puncture failures; for hematoma puncture, it is very easy to deviate from the target of the hematoma or accidentally damage important blood vessels and functional areas, causing secondary injury. Existing neuronavigation systems are mainly divided into two categories: one is large workstations based on infrared optical or magnetic positioning, which are bulky, have long preparation times, and are expensive; the other is the recently emerged robotic vision-assisted system, which, although incorporating AI, often relies on complex hardware such as structured light projectors, binocular camera modules, or robotic arms, making it difficult to deploy flexibly in the confined space of the emergency room. In addition, existing mobile navigation solutions mostly only function as "electronic levels," lacking perception of the three-dimensional structure of the patient's head, and are unable to solve the problem of registration failure caused by facial features being obscured by sterile surgical drapes. For example, the existing patent CN202011155529.2 discloses a guiding device for cerebral hemorrhage puncture surgery, which only uses a mobile phone digital display tilt meter APP and a mechanical support to assist in setting the angle. It relies entirely on the doctor to manually adjust the tilt angle of the mobile phone and lacks the ability to automatically reconstruct and match the three-dimensional structure of the patient's head. It cannot cope with situations where the patient's head moves slightly or is covered by a drape.

[0003] In recent years, neurosurgical navigation systems based on AR technology have emerged (such as systems that recognize QR code markers using the Vuforia SDK), employing marker-based registration with an accuracy of up to 1.333 mm. However, such solutions suffer from fatal flaws in clinical applications: conflicts between marker placement and surgical incisions, marker obstruction by sterile drapes leading to recognition failure, marker shedding due to skin oils, and the potential for the markers themselves to become sources of contamination. In contrast, markerless registration directly utilizes the patient's own anatomical features as a natural benchmark, achieving registration through monocular visual 3D reconstruction and surface matching technology, fundamentally avoiding the clinical application obstacles caused by markers. This is the core technical difference between existing marker-based AR navigation systems and the present invention.

[0004] Although neural rendering technologies such as Neural Radiation Field (NeRF) and 3D Gaussian Splatting have been applied in the field of computer vision, their application in real-time medical navigation in the operating room environment still faces the following challenges: (1) The lighting in the operating room is complex, with strong reflections, shadows, and direct exposure to shadowless lamps; (2) The occlusion caused by sterile drapes leads to feature loss, and traditional NeRF algorithms cannot handle transparent media occlusion; (3) The computing power of mobile devices is limited, and existing NeRF solutions mostly run on high-performance GPU servers, which cannot meet the real-time requirements of mobile devices (usually requiring a rendering frame rate of more than 60fps).

[0005] In summary, there is an urgent need for a high-precision mobile navigation solution that is label-free, low-cost, and can effectively solve the problem of obstruction in sterile drapes. Summary of the Invention

[0006] To address the above shortcomings, this invention provides a markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision, comprising:

[0007] S1: Obtain a three-dimensional medical image model of the patient before surgery;

[0008] S2: During the surgery, a monocular camera from a consumer-grade computing device is used to capture video sequences of the surgical area in real time, and a three-dimensional surface model of the surgical area is constructed in real time through a three-dimensional reconstruction algorithm.

[0009] S3: Perform optical correction on the video sequence to eliminate the obstruction and reflection interference of the surgical drape, and perform temporal hybrid registration between the corrected three-dimensional surface model and the three-dimensional medical image model;

[0010] S4: The registered 3D medical image model is overlaid with the real-time video footage to generate an augmented reality view, providing navigation for surgery.

[0011] Furthermore, in step S2, the three-dimensional reconstruction algorithm specifically employs a neural rendering algorithm.

[0012] Furthermore, the neural rendering algorithm employs a lightweight neural rendering algorithm, such as 3D Gaussian Splatting or a lightweight neural radiation field (NeRF) algorithm.

[0013] Furthermore, in step S2, the three-dimensional reconstruction algorithm specifically adopts a regression reconstruction algorithm based on a parameterized model.

[0014] Furthermore, the optical correction in step S3 includes:

[0015] An optical flow separation mechanism is used to separate the surgical drape layer from the patient's body surface layer in the video sequence; multi-view geometric constraints are used to eliminate specular highlights on the patient's body surface; and an image enhancement algorithm is used to improve the clarity of the image under the semi-transparent medium.

[0016] Furthermore, the temporal hybrid registration mechanism in step S3 includes:

[0017] Global Lock-in Period: Before laying the surgical drapes, multi-angle video sequences are acquired, and visual feature points are used to perform a one-time global initial alignment between the three-dimensional surface model and the three-dimensional medical image model.

[0018] Inertial maintenance period: After global locking, when the visual signal is lost or the visual feature quality is lower than a preset threshold, the system switches to using the inertial measurement unit data of the computing device for attitude tracking to maintain the registration state.

[0019] Local correction period: After the visual signal is recovered, the accumulated error of inertial tracking is corrected and compensated online using real-time visual features.

[0020] Furthermore, it also includes real-time tracking of the position and orientation of surgical instruments using image recognition algorithms, and displaying their virtual extensions and safety paths in an augmented reality view.

[0021] This invention also discloses a label-free augmented reality brain puncture navigation system based on monocular vision, characterized in that it includes:

[0022] The image acquisition module is used to acquire video sequences of the surgical area in real time;

[0023] An inertial measurement module is used to collect inertial data from the equipment.

[0024] The storage module is used to store the patient's preoperative 3D medical image model;

[0025] The optical correction module is used to perform adaptive optical correction processing on the video sequence to eliminate visual interference caused by surgical drape obstruction (including optical flow separation, multi-view geometric reflection elimination, and translucent enhancement);

[0026] The registration module is used to perform temporal hybrid registration, which aligns the corrected intraoperative 3D surface model with the 3D medical image model according to the three-stage state machine of global locking-inertia maintenance-local correction.

[0027] The processor, electrically connected to the above modules, is used to construct a three-dimensional surface model of the surgical area in real time based on the video sequence using a three-dimensional reconstruction algorithm (3D Gaussian sputtering or lightweight NeRF-type algorithm), and to coordinate the modules to complete the complete navigation process.

[0028] And a display module, used to overlay the registered 3D medical image model with real-time video footage to generate and display an augmented reality view.

[0029] Compared with the prior art, the present invention has the following advantages:

[0030] 1. Low cost and high portability: No need for dedicated large optical or magnetic positioning equipment; high-precision navigation can be achieved with just a smartphone.

[0031] 2. Strong anti-occlusion capability: The innovative transparent drape optical correction and temporal registration mechanism effectively solves the problem of feature loss in aseptic operating environments;

[0032] 3. Fully local computing: All data processing is completed locally on the terminal without the need for an internet connection, thus protecting patient data privacy;

[0033] 4. Wide range of applications: It is not only applicable to conventional scalp surface localization, but also to localization based on brain tissue surface after craniotomy.

[0034] In summary, by introducing advanced real-time neural graphics primitives or implicit / semi-implicit representation algorithms such as 3D Gaussian sputtering, the system directly utilizes the patient's own anatomical features (facial skin texture, bony landmarks) as a natural registration benchmark to achieve rapid and high-precision three-dimensional reconstruction of the patient's head during surgery. Combined with general complex optical environment correction technology, the system can not only penetrate transparent drapes, but also achieve accurate navigation in non-ideal environments with strong light reflection or local occlusion without drapes. Attached Figure Description

[0035] Figure 1 This is the overall flowchart of the navigation method in Example 2.

[0036] Figure 2 This is a schematic diagram of the three stages of the registration process in step S3 of the present invention.

[0037] Figure 3 This is a schematic diagram illustrating the optical correction and reflection elimination principle of the transparent drape in this invention.

[0038] Figure 4 This is a schematic diagram of the navigation interface in this invention. Detailed Implementation

[0039] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0040] Example 1

[0041] This embodiment provides a label-free augmented reality brain puncture navigation system based on monocular vision, including:

[0042] The image acquisition module is used to acquire video sequences of the surgical area in real time;

[0043] An inertial measurement module is used to collect inertial data from the equipment.

[0044] The storage module is used to store the patient's preoperative 3D medical image model;

[0045] The optical correction module is used to perform adaptive optical correction processing on the video sequence to eliminate visual interference caused by surgical drape obstruction (including optical flow separation, multi-view geometric reflection elimination, and translucent enhancement);

[0046] The registration module is used to perform temporal hybrid registration, which aligns the corrected intraoperative 3D surface model with the 3D medical image model according to the three-stage state machine of global locking-inertia maintenance-local correction.

[0047] The processor, electrically connected to the above modules, is used to construct a three-dimensional surface model of the surgical area in real time based on the video sequence using a three-dimensional reconstruction algorithm (3D Gaussian sputtering or lightweight neural radiation field NeRF-type algorithm), and to coordinate the modules to complete the complete navigation process.

[0048] And a display module, used to overlay the registered 3D medical image model with real-time video footage to generate and display an augmented reality view.

[0049] Example 2

[0050] This embodiment provides a markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision, which is based on the system in Embodiment 1, such as... Figure 1 As shown, the monocular vision-based label-free augmented reality brain puncture navigation system in Example 1 will be referred to as the system in the following text, and specifically includes the following steps:

[0051] S1. Obtain the patient's preoperative 3D medical imaging model.

[0052] S101. Data Acquisition: Read the patient's head medical computed tomography images (including CT, MRI or PET-CT, DICOM format).

[0053] S102, Reference Model Reconstruction: Three 3D surface models are extracted using the Marching Cubes algorithm.

[0054] Skin surface model ( ): Used for registration with data captured by the mobile phone during surgery;

[0055] Skull surface model ( ): Used to calculate the recommended puncture site (bone hole location);

[0056] Brain tissue / target model ( ): Includes internal structures such as ventricles, hematomas, or tumors;

[0057] Step S103, Intelligent Path Planning and Bone Hole Recommendation:

[0058] Intelligent Path Planning: Brain Tissue / Target Model Extracted from S102 ( (Setting the puncture entry point in three-dimensional space) and target Generate the ideal puncture path vector ;

[0059] Bone hole location decision suggestion: based on the skull surface model extracted from S102 ( Based on preset medical rules (such as avoiding important blood vessels, functional areas, hairline, etc.), the system automatically calculates and recommends the optimal puncture point (bone hole location). This recommendation is directly overlaid on the patient's head model (including the skull surface model) via an AR interface during the preoperative stage. ) and skin surface model ( The data is stored in the system to help doctors quickly determine the location of the markers.

[0060] S2. Real-time construction of a 3D surface model of the surgical area

[0061] During the surgery, the operator holds a consumer-grade computing device (such as a smartphone or tablet) and uses its rear-mounted monocular camera to capture video sequences in real time, and uses an efficient 3D reconstruction algorithm to build a 3D surface model of the patient's head in real time and incrementally.

[0062] In this embodiment, the 3D Gaussian Splatting (3DGS) algorithm is employed. The core idea of ​​this algorithm is to represent the scene as a collection of numerous three-dimensional Gaussian ellipsoids, each possessing attributes such as position, rotation, scaling, color, and opacity. Compared to traditional NeRF (Neural Radiation Field), which requires time-consuming volume rendering via neural networks, 3DGS achieves extremely high rendering speed by projecting three-dimensional Gaussian ellipsoids onto a two-dimensional image plane and performing efficient fragment sorting and alpha blending. To adapt to the computing power of mobile devices, the optimization strategies employed include:

[0063] 1) Online training and incremental mapping optimize the model only for newly acquired keyframes, rather than global optimization;

[0064] 2) Model pruning and compression: Regularly remove Gaussian points with low opacity or excessive size, and compress the model volume using quantization, encoding, and other methods;

[0065] 3) A simplified spherical harmonic function (SH) order is used to reduce the complexity of color calculations. These optimizations enable high-precision surface reconstruction of patient heads on mainstream mobile devices, supporting real-time rendering at over 60fps.

[0066] S3. Perform optical correction and temporal hybrid registration (where S301-S303 are called adaptive optical correction processing, and S303 is called cross-modal surface registration processing, as detailed below):

[0067] S301, Dynamic Interference Stripping (Optical Flow Separation Mechanism, i.e., for flexible or non-rigid micro-flickering isolation media):

[0068] In environments with transparent drapes (whether tightly attached) or airflow disturbances, optical flow field separation is performed in the frequency domain by utilizing the characteristics of non-rigid random tremors generated by the interfering medium (such as a thin film blown by laminar airflow) and the low-frequency rigid motion of the patient's head (skin texture). Although the absolute frequency of the tremors may not be high, they have a significantly different spectral distribution relative to the quasi-static characteristics of the head. Based on this, the system eliminates the dynamic interference layer, as follows:

[0069] P1, for two consecutive frames of images and The optical flow field was calculated using the Lucas-Kanade algorithm. ;

[0070] P2. Perform frequency domain analysis (FFT transform) on the optical flow field to separate the non-rigid micro-vibration component (corresponding to the medium) and the rigid motion component (corresponding to the head).

[0071] P3, Set frequency threshold (e.g., 10Hz), the rejection frequency is higher than The optical flow vectors are used to retain only the low-frequency rigid optical flow layer for subsequent reconstruction.

[0072] P4. Using motion model assumptions (such as rigid body motion model), perform consistency verification on the retained low-frequency optical flow and further eliminate outliers.

[0073] S302, Static artifact removal (multi-view geometric consistency, i.e., for rigid media or exposed skin):

[0074] In environments with strong specular reflection / highlighting (such as the reflection of skin oil / water stains and skin areas with strong reflection caused by direct exposure to surgical lights without draping), or in environments with rigid / semi-rigid transparent isolation media that are not in close contact with the skin (such as rigid sterile covers or suspended drapes), the position of the image will shift under different viewing angles (virtual image), while the actual anatomical texture will remain fixed (real image), as detailed below:

[0075] P1. Acquire image sequences from multiple viewpoints and estimate the camera pose for each viewpoint using visual odometry (VIO). ;

[0076] P2. For each suspected reflective area, calculate its reprojection error at multiple viewing angles. The formula for calculating the reprojection error is:

[0077]

[0078] in, 3D point coordinates For projection function, For the first 2D observation points from various perspectives;

[0079] P3. If the reprojection error of a certain area is greater than the threshold (e.g., 5 pixels), it is determined to be a reflective area.

[0080] P4. Using PatchMatch-based image inpainting techniques to remove reflective areas and restore the underlying skin texture, a priority-based texture synthesis algorithm proposed by Criminisi et al. is used, which first fills the boundary areas and then diffuses inwards.

[0081] S303, Semi-transparency enhancement mechanism: In environments with semi-transparent or frosted materials, image contrast may decrease and details may become blurred. Therefore, the following steps are used to achieve enhancement:

[0082] P1. Contrast Enhancement: Histogram equalization is performed on the acquired image to enhance overall contrast. The transformation function for histogram equalization is... Input grayscale level Mapped to output grayscale levels The mapping relationship is as follows:

[0083]

[0084] in, To input grayscale level, To output grayscale levels, grayscale The number of pixels, This represents the total number of pixels. The total number of gray levels in an image (e.g., in the most common 8-bit grayscale image). );

[0085] P2. Sharpening Filter: Sharpening filtering is performed using Unsharp Masking technology. The image after histogram equalization. To The low-pass filtered image obtained after Gaussian blurring is the enhanced image. for:

[0086]

[0087] in, This is the sharpening intensity factor (usually 1.5-2.0).

[0088] P3, Based on the enhanced image SIFT or ORB algorithms are used for feature point extraction: the system detects the extreme points in the local scale space of the image and generates a descriptor with scale and rotation invariance for each key point; effective feature points with response values ​​higher than a preset threshold are selected to form a set of verified feature points, which are directly used as the visual feature input for the subsequent temporal hybrid registration stage;

[0089] S304, Three-Stage Temporal Registration State Machine:

[0090] To address the issue of visual tracking failure caused by prolonged and large-area occlusion due to surgical draping, the registration process consists of the following three stages (e.g., Figure 2 (as shown)

[0091] Phase I: Global Lockdown Period (Before Laying the Towel or in a Visually Positive Environment)

[0092] This stage is performed before the start of surgery and before the sterile surgical drapes are laid out. The operator holds the device and scans around the patient's head (covering feature-rich areas such as the nose, eyes, and ears), acquiring a short video (approximately 5-10 seconds) containing multi-angle information. Then, using the ICP (Iterative Closest Point) algorithm or a feature-point-based registration method (such as RANSAC+SVD), an initial transformation matrix is ​​established from the medical imaging coordinate system to the intraoperative real-world coordinate system. During this process, the reprojection error is less than 1mm and the number of feature point matches is greater than 100 (which can be preset to a threshold).

[0093] It should be noted that when establishing the initial transformation matrix... At the same time, the system will register all 3D feature points within the current field of view that are involved in the registration. Its descriptors are stored as a global feature point map. This is used for local closed-loop correction in subsequent stage III;

[0094] Phase II: Inertia Maintenance Period (After the towel is laid out or in an environment where vision is lost)

[0095] After global locking is completed, the surgery proceeds normally, and the surgeon begins laying out the surgical drapes. At this point, the camera's field of view is completely or mostly obstructed, resulting in the loss of visual features. When the system detects that the tracking quality is below a preset threshold (e.g., when the number of feature points is less than 50, or the reprojection error is greater than 2mm), it automatically switches to the inertial maintenance phase. The preset threshold can be set empirically; for example, it can be set to the number of effective visual feature points being less than a certain value, or the reprojection error of the 3D point in the current frame being greater than a certain millimeter value. During this phase, the system pauses visual processing and relies entirely on data from the device's built-in inertial measurement unit (IMU). Through a tightly coupled vision-inertial odometry (VIO) framework (such as VINS-Fusion or ORB-SLAM3), it uses only the IMU's angular velocity and acceleration data for integration to continuously calculate the device's pose changes. This ensures that even without visual signals, the virtual model in the AR view will not be immediately lost or experience violent fluctuations, providing continuity for the surgical procedure.

[0096] set up Let R be the pose matrix of the device at time t (including the rotation matrix R). Translation vector t Then the VIO pose update formula is:

[0097]

[0098] in, The pose increment is obtained by integrating the IMU angular velocity and acceleration data, including translation and rotation components.

[0099] During this stage, the system maintains the coordinate system without drifting, but the accuracy will decrease (the error accumulation rate is usually 0.1-0.5 mm / s).

[0100] The above switching logic also applies to brief accidental occlusions (such as a doctor's hand momentarily obscuring the view or a strong light sweeping across, lasting less than 2 seconds). In this case, the system automatically switches to VIO inertial maintenance mode; once the visual signal recovers (the number of feature points exceeds the threshold), the system does not need to go through the complete Stage III local correction process, but directly uses the matching results of the newly detected feature points and the existing model to quickly re-lock the pose, reducing the impact of brief occlusions on navigation continuity, which is defined as accidental interference handling;

[0101] Phase III: Local Correction Period (After Towel Placement or in the Environment of Visual Recovery)

[0102] Once the surgical drapes are in place, exposing only the surgical incision area, or after obstructions such as the surgeon's hands have been removed and some visual information has been restored to the camera's field of view, the system enters a local correction phase. At this time, the system re-detects and tracks visible patient surface features (such as the nose and ear outlines) within the field of view, and then uses a Local Bundle Adjustment (BA) algorithm for closed-loop correction.

[0103] First, extract sparse feature points (such as SIFT and ORB feature points) in the locally exposed areas.

[0104] Subsequently, these feature points are compared with the global feature point map established in Phase I. Perform matching and establish corresponding relationships;

[0105] Reconstruct the optimization objective function:

[0106]

[0107] in, For the number of viewpoints, The number of feature points, For robust kernel functions (such as Huber loss). The regularization coefficient is . This is the locally corrected transformation matrix. Let be the global transformation matrix for stage I. For the first Camera pose matrix from each viewpoint For the first The coordinates of a 3D feature point in the world coordinate system For perspective projection functions, project 3D points (That is, after transforming from world coordinates to camera coordinates) projected onto the image plane, For three-dimensional points The two-dimensional observation coordinates (feature point pixel positions) in the first viewpoint image, therefore This is called reprojection error, which measures the deviation between the projected position of a three-dimensional point and its actual observed position.

[0108] Finally, the Levenberg-Marquardt algorithm is used to optimize the objective function and find the optimal solution. This eliminates the cumulative drift generated during the VIO phase.

[0109] The corrected registration accuracy was restored to near the level of Stage I (reprojection error <1.5mm). This process was continuous, ensuring registration accuracy during long surgical procedures.

[0110] The principles of optical correction and reflection elimination for transparent drapes can be found in [reference needed]. Figure 3 .

[0111] S4. Generate and display the augmented reality navigation view.

[0112] like Figure 4 As shown, the navigation interface uses a Video Transspective (VST) mode. The screen background is a real camera view, with a precisely overlaid semi-transparent virtual ventricle / hematoma model, a highlighted crosshair (recommended puncture site), and a red target point in the foreground. When the doctor holds the needle for puncture, the system identifies the needle axis. Once the needle tip is obscured after piercing the skin, the system draws a green visual guide line (virtual extension line) in real time based on the direction of the needle tail, intuitively displaying the "transspective" trajectory of the puncture needle. The screen edge features color-coded angle deviation indicators (green-yellow-red) and a digital reading of the distance to the target point. In addition, the system supports contactless interactive control: for general portable terminals, it can be operated via voice commands (such as "lock target point") and hovering gestures / head postures; for dedicated medical terminals, it can be used in conjunction with physical sterile buttons to ensure that doctors can efficiently control the device in a sterile environment.

[0113] Example 3

[0114] The difference from Example 2 is that the 3D reconstruction algorithm in step S2 adopts the neural radiation field algorithm based on Instant-NGP (Instant Neural Graphics Primitives). Its basic principle is to use multi-resolution hash encoding in conjunction with a miniature multilayer perceptron (MLP) to implicitly represent the scene. Multi-resolution hash encoding can efficiently map spatial points to high-dimensional feature vectors, greatly reducing the size and training overhead of the MLP. The advantage of this approach is that the generated surface model is smoother, and the model file occupies less space. To cope with interference from dynamic elements during surgery (such as the surgeon's hand), the system can combine a semantic segmentation network to ignore pixels on non-patient surfaces during training, thereby constructing a clean, static patient head model.

[0115] Example 4

[0116] The difference from Example 2 lies in the fact that the 3D reconstruction algorithm in step S2 employs a regression reconstruction algorithm based on a parametric model as a robust baseline to cope with scenarios where computing power is severely limited or lighting conditions are extremely poor. Its basic principle is to utilize a pre-built 3D deformable face / head model (3DMM) or an existing open-source model library (such as Media Pipe Face Mesh). The algorithm uses a lightweight convolutional neural network to regress the shape and pose parameters of the model from a single 2D image, thereby quickly fitting the 3D shape of the patient's head. Although this approach has slightly lower accuracy than the previous two, it requires minimal computation, is highly robust, and can ensure basic navigation usability under extreme conditions.

[0117] Example 5

[0118] The steps of Example 2 can be applied to localization during craniotomy. In this scenario, the anatomical surface features in step S1 are replaced with the brain tissue surface. During intraoperative reconstruction, the system uses the sulci, gyri, and morphological features of the exposed brain tissue to perform feature matching with the preoperative MRI-reconstructed cortical model. This allows the present invention not only to guide puncture but also to assist in locating deep subcortical lesions (such as gliomas and cavernous hemangiomas).

[0119] In addition, to strictly adhere to the aseptic principles of the operating room, and especially to address the issue of the inability to sterilize the touchscreen in Example 2 (general-purpose portable terminal), this system provides the following contactless interaction mechanism (for Example 5, this mechanism can serve as a supplement to physical aseptic buttons):

[0120] 1. Voice command control: Integrates an offline voice recognition engine to support specific medical commands (such as "lock target", "switch view", "start navigation", "reset"), enabling hands-free operation throughout the entire process.

[0121] 2. Hover gesture recognition: Using the front or rear camera to capture the doctor's hand gestures in the air (such as waving to turn a page, clenching a fist to confirm), the transparency or zoom of the AR view can be adjusted without physical contact with the screen.

[0122] 3. Head Gaze: When the front camera is enabled, it tracks the doctor's head movement or gaze point to enable cursor movement and dwell confirmation (Dwell Click).

[0123] It should be noted that the structure described in this invention can be implemented in many different forms and is not limited to the embodiments described. Any equivalent transformations made by those skilled in the art based on the description and drawings of this invention, or direct or indirect applications in other related technical fields, such as the loading and unloading of other items, are included within the protection scope of this invention.

Claims

1. A markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision, characterized in that, include: S1: Obtain a three-dimensional medical image model of the patient before surgery; S2: During the operation, a monocular camera is used to collect video sequences of the surgical area in real time, and a three-dimensional surface model of the surgical area is constructed in real time through a three-dimensional reconstruction algorithm. S3: Perform optical correction on the video sequence to eliminate the obstruction and reflection interference of the surgical drape, and perform temporal hybrid registration between the corrected three-dimensional surface model and the three-dimensional medical image model; S4: The registered 3D medical image model is overlaid with the real-time video footage to generate an augmented reality view, providing navigation for surgery.

2. The markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in claim 1, characterized in that: In step S2, the three-dimensional reconstruction algorithm specifically employs a neural rendering algorithm.

3. The markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in claim 2, characterized in that: The neural rendering algorithm is a lightweight neural rendering algorithm, which is a 3D Gaussian sputtering or lightweight neural radiation field algorithm.

4. The markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in claim 1, characterized in that: In step S2, the three-dimensional reconstruction algorithm specifically adopts a regression reconstruction algorithm based on a parameterized model.

5. A markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in any one of claims 1, 2, or 4, characterized in that: The optical correction in step S3 includes: The method employs optical flow separation to separate the surgical drape layer from the patient's body surface layer in the video sequence, multi-view geometric constraints to eliminate specular highlights on the patient's body surface, and image enhancement algorithms.

6. The markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in any one of claims 5, characterized in that: The temporal hybrid registration mechanism in step S3 includes: Global Lock-in Period: Before laying the surgical drapes, multi-angle video sequences are acquired, and visual feature points are used to perform a one-time global initial alignment between the three-dimensional surface model and the three-dimensional medical image model. Inertial maintenance period: After global locking, when the visual signal is lost or the visual feature quality is lower than a preset threshold, the system switches to using the inertial measurement unit data of the computing device for attitude tracking to maintain the registration state. Local correction period: After the visual signal is recovered, the accumulated error of inertial tracking is corrected and compensated online using real-time visual features.

7. The markerless, anti-occlusion AR navigation method for ventricular and intracranial hematoma puncture based on monocular vision as described in claim 1, characterized in that: It also includes real-time tracking of the position and orientation of surgical instruments using image recognition algorithms, and displaying their virtual extensions and safety paths in an augmented reality view.

8. A label-free augmented reality brain puncture navigation system based on monocular vision, characterized in that, include: The image acquisition module is used to acquire video sequences of the surgical area in real time; An inertial measurement module is used to collect inertial data from the equipment. The storage module is used to store the patient's preoperative 3D medical image model; An optical correction module is used to perform adaptive optical correction processing on video sequences to eliminate visual interference caused by surgical drapes obscuring the view. The registration module is used to perform temporal hybrid registration, which aligns the corrected intraoperative 3D surface model with the 3D medical image model according to the three-stage state machine of global locking-inertia maintenance-local correction. The processor, electrically connected to the above modules, is used to construct a three-dimensional surface model of the surgical area in real time based on the video sequence using a three-dimensional reconstruction algorithm, and to coordinate the modules to complete the complete navigation process. And a display module, used to overlay the registered 3D medical image model with real-time video footage to generate and display an augmented reality view.

Images