A neural puncture dynamic obstacle avoidance navigation method and system based on multi-modal image fusion

Abstract

The application relates to a kind of neural puncture dynamic obstacle avoidance navigation method and system based on multi-modal image fusion, belong to medical image navigation and surgical operation technical field, by fusing preoperative high-resolution static image and intraoperative real-time dynamic ultrasound data, construct unified space-time reference model, and utilize artificial intelligence technology to automatically segment and reconstruct the three-dimensional model of key neural vascular structure, can be based on real-time image data dynamic calculation optimal puncture path and real-time monitoring collision risk in the process of needle insertion, provide multi-sensory fusion intuitive navigation guide for operator through augmented reality technology, through recording and analyzing the whole process data of operation, utilize big data platform to continuously iterate optimization to segmentation and obstacle avoidance algorithm, form self-improving intelligent closed loop.The application effectively solves the problems that traditional neural puncture surgery depends on the experience of operator, cannot real-time dynamic obstacle avoidance and system lacks learning ability, significantly improves the precision, safety and intelligent level of operation.

Description

Technical Field

[0001] This application relates to the fields of medical image navigation and surgical technology, and in particular to a method and system for dynamic obstacle avoidance navigation for nerve puncture based on multimodal image fusion. Background Technology

[0002] Currently, nerve puncture surgeries such as nerve blocks or biopsies mainly rely on single-modality imaging guidance, such as ultrasound, CT, or MRI. Ultrasound can provide real-time dynamic images, but its resolution is limited, making it difficult to clearly display deep nerves and complex anatomical relationships; while CT / MRI can provide high-resolution static images, it cannot reflect the dynamic interaction between the needle and the tissue in real time. This single-image guidance method forces the surgeon to complete the fusion of multimodal information and path planning in their mind, which is highly dependent on personal experience and can easily lead to problems such as inaccurate path planning, increased number of punctures, and a long learning curve.

[0003] In recent years, although some studies have attempted to fuse preoperative CT / MRI with intraoperative ultrasound to assist navigation, existing methods mostly focus on rigid registration in static space, failing to effectively address dynamic deviations caused by patient breathing, body position changes, or tissue deformation. Furthermore, traditional obstacle avoidance planning typically relies on pre-set fixed paths based on preoperative static images, making dynamic adjustments and real-time risk warnings during surgery impossible based on actual conditions (such as vascular pulsation or organ movement), thus posing potential surgical risks.

[0004] Furthermore, existing navigation systems lack the ability to learn and optimize based on big data. Multimodal image data, operational trajectories, and outcome information generated during surgery are not systematically collected and used for model iteration. Their segmentation, planning, and obstacle avoidance algorithms are often rigid and unable to learn from historical surgical cases to adapt to complex anatomical variations or rare situations. Therefore, developing an intelligent navigation method that can integrate multimodal images, achieve dynamic real-time obstacle avoidance, and continuously self-optimize has become an urgent need in this field. Summary of the Invention

[0005] To achieve the above objectives, this application provides the following technical solution:

[0006] According to a first aspect of the present invention, the present invention claims protection for a dynamic obstacle avoidance navigation method for neural puncture based on multimodal image fusion, comprising the following steps:

[0007] S1. High-resolution static image data is acquired before surgery using medical imaging equipment, and dynamic image stream data is acquired during surgery using real-time ultrasound imaging equipment. A spatiotemporal synchronous coordinate system is constructed, and the static image data and dynamic image stream data are registered and fused in time and space to generate a multimodal fusion image base with a unified spatiotemporal reference.

[0008] S2, based on a pre-trained deep learning segmentation model, performs pixel-level semantic segmentation on the multimodal fusion image substrate, identifies and reconstructs the target neural structure, key blood vessels, bones, and dangerous areas to be avoided in three dimensions; and displays the segmentation results overlaid with the preoperative planned path.

[0009] S3, based on the segmented three-dimensional anatomical structure, uses a path search algorithm to calculate an optimal initial path from the puncture site to the target nerve region; receives the dynamic image stream data in real time, calculates the motion vector of the intraoperative medium using optical flow method, and monitors the deviation from the initial path or danger zone in real time. When a potential collision risk is detected, an obstacle avoidance guidance vector is calculated in real time based on the motion vector and the spatial relationship of the anatomical structure.

[0010] S4. The initial path, real-time instrument pose, obstacle avoidance guidance vector, and 3D reconstruction model of key structures are used to generate dynamic visual and auditory guidance information based on the obstacle avoidance guidance vector, prompting the operator to adjust the needle insertion angle, depth, or direction.

[0011] S5 continuously records multimodal imaging data, instrument movement trajectory, surgeon operation details, and final result data throughout the entire puncture process, building a big data warehouse for nerve puncture surgery.

[0012] Furthermore, step S1 also includes:

[0013] High-resolution static image data includes one or more combinations of MRI images, CT images, or PET-CT images; the establishment of the spatiotemporal synchronization coordinate system is achieved by attaching positioning markers to the patient's body surface that are visible in multimodal images.

[0014] Furthermore, step S2 also includes:

[0015] The pre-trained deep learning segmentation model was trained using a large-scale historical desensitized neural image dataset within a distributed computing framework. Data parallelism was employed during the model training process to accelerate the processing of massive amounts of training data.

[0016] Furthermore, step S3 also includes:

[0017] The path search algorithm is an improved algorithm based on the A* algorithm or the Fast Random Exploration Tree (RRT) algorithm. Its cost function integrates multiple factors such as path length, distance from the danger zone, and the permeability of the organization.

[0018] Furthermore, step S3 also includes:

[0019] The real-time obstacle avoidance vector calculation is specifically as follows: a real-time local obstacle avoidance potential field is constructed with the puncture needle tip as the center, a repulsive field is generated in the danger zone, and an attractive field is generated in the target zone;

[0020] The obstacle avoidance guidance vector is the resultant force direction vector of the potential field at the current needle tip position.

[0021] Furthermore, in step S4, the dynamic guidance information includes, but is not limited to:

[0022] A dynamic guide arrow indicating the correct needle insertion direction is overlaid on the real-time video, and the area is highlighted and a graded audible alarm is emitted when approaching a danger zone.

[0023] Furthermore, step S5 also includes:

[0024] The big data warehouse for nerve puncture surgery is built using a big data platform based on Hadoop or Spark. The data warehouse not only stores the raw data, but also stores high-quality feature datasets after feature extraction, which are used for subsequent analysis and model optimization.

[0025] Furthermore, step S5 also includes:

[0026] Iterative optimization and incremental learning specifically involve periodically sampling new surgical data from the big data warehouse, fine-tuning the segmentation model and path planning algorithm, and continuously updating and enriching the obstacle avoidance rule base using obstacle avoidance cases in the new data.

[0027] Furthermore, the method also includes:

[0028] S6, Surgical plan simulation and prognosis prediction: Based on surgical data and prognosis information of similar cases in the historical big data warehouse, multiple simulated puncture paths are generated for the current patient before surgery, and the success rate, risk index and expected recovery effect of each path are predicted to assist doctors in making preoperative decisions.

[0029] The method operates within a navigation system that integrates a high-performance computing unit and a real-time data processing unit. The real-time data processing unit continuously processes dynamic image stream data using stream processing technology to ensure the real-time nature of obstacle avoidance guidance.

[0030] According to a second aspect of the present invention, the present invention claims protection for a dynamic obstacle avoidance navigation system for neural puncture based on multimodal image fusion, comprising:

[0031] One or more processors;

[0032] A memory storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the aforementioned method for dynamic obstacle avoidance navigation of neural puncture based on multimodal image fusion.

[0033] This invention relates to a dynamic obstacle avoidance navigation method and system for nerve puncture based on multimodal image fusion, belonging to the field of medical image navigation and surgical technology. By fusing preoperative high-resolution static images with intraoperative real-time dynamic ultrasound data, a unified spatiotemporal reference model is constructed. Artificial intelligence technology is used to automatically segment and reconstruct three-dimensional models of key neurovascular structures. Based on real-time image data, the optimal puncture path can be dynamically calculated and collision risks during needle insertion can be monitored in real time. Augmented reality technology provides surgeons with intuitive navigation guidance through multi-sensory fusion. By recording and analyzing data from the entire surgical process, the segmentation and obstacle avoidance algorithms are continuously iterated and optimized using a big data platform, forming a self-improving intelligent closed loop. This invention effectively solves the problems of reliance on surgeon experience, inability to achieve real-time dynamic obstacle avoidance, and lack of learning ability in traditional nerve puncture surgery, significantly improving the accuracy, safety, and intelligence level of the surgery. Attached Figure Description

[0034] Figure 1 A flowchart illustrating the workflow of a dynamic obstacle avoidance navigation method for nerve puncture based on multimodal image fusion, as claimed in this application embodiment;

[0035] Figure 2 The diagram shows the structural modules of a neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion, which is claimed in the embodiments of this application. Detailed Implementation

[0036] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0037] The terms "first," "second," and "third" in this application are for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined as "first," "second," or "third" may explicitly or implicitly include at least one of that feature. In the description of this application, "multiple" means at least two, such as two, three, etc., unless otherwise explicitly specified. All directional indications (such as up, down, left, right, front, back, etc.) in the embodiments of this application are only used to explain the relative positional relationships and movements between components in a specific orientation (as shown in the figures). If the specific orientation changes, the directional indications also change accordingly. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or devices.

[0038] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0039] According to a first embodiment of the present invention, the present invention claims protection for a dynamic obstacle avoidance navigation method for neural puncture based on multimodal image fusion, referring to... Figure 1 This includes the following steps:

[0040] S1. High-resolution static image data is acquired before surgery using medical imaging equipment, and dynamic image stream data is acquired during surgery using real-time ultrasound imaging equipment. A spatiotemporal synchronous coordinate system is constructed, and the static image data and dynamic image stream data are registered and fused in time and space to generate a multimodal fusion image base with a unified spatiotemporal reference.

[0041] S2, based on a pre-trained deep learning segmentation model, performs pixel-level semantic segmentation on the multimodal fusion image substrate, identifies and reconstructs the target neural structure, key blood vessels, bones, and dangerous areas to be avoided in three dimensions; and displays the segmentation results overlaid with the preoperative planned path.

[0042] S3, based on the segmented three-dimensional anatomical structure, uses a path search algorithm to calculate an optimal initial path from the puncture site to the target nerve region; receives the dynamic image stream data in real time, calculates the motion vector of the intraoperative medium using optical flow method, and monitors the deviation from the initial path or danger zone in real time. When a potential collision risk is detected, an obstacle avoidance guidance vector is calculated in real time based on the motion vector and the spatial relationship of the anatomical structure.

[0043] S4. The initial path, real-time instrument pose, obstacle avoidance guidance vector, and 3D reconstruction model of key structures are used to generate dynamic visual and auditory guidance information based on the obstacle avoidance guidance vector, prompting the operator to adjust the needle insertion angle, depth, or direction.

[0044] S5 continuously records multimodal imaging data, instrument movement trajectory, surgeon operation details, and final result data throughout the entire puncture process, building a big data warehouse for nerve puncture surgery.

[0045] In this embodiment, S1, multimodal image data acquisition and synchronous fusion, involves: First, acquiring high-resolution static images of the patient before surgery, which clearly distinguish structures such as nerves, blood vessels, bones, and soft tissues. Simultaneously, during the surgery, a dynamic real-time image sequence containing the puncture needle and surrounding tissues is continuously acquired using an ultrasound probe. Then, by identifying and matching reference markers pre-attached to the patient's body surface and visible in all image modalities, a unified spatial coordinate system shared by all image data is established. Finally, the detailed anatomical structures of the high-resolution static images are fused with the real-time ultrasound images to generate a fused image model that displays both deep, complex structures and reflects real-time dynamic changes.

[0046] S2. Intelligent Recognition and 3D Reconstruction of Key Anatomical Structures: A deep learning network model, pre-trained on a large amount of neuroimaging data, automatically analyzes the fused image model obtained in step S1. This model identifies target nerve bundles, various blood vessels, bone boundaries, and important functional areas that need to be avoided pixel by pixel, and automatically outlines their contours. Based on this identified 2D contour information, a 3D model of the key anatomical structures is reconstructed using 3D volume rendering technology. This model is then spatially aligned with the fused image and overlaid on the navigation interface.

[0047] S3. Optimal Path Planning and Real-Time Dynamic Obstacle Avoidance Calculation: Based on the 3D anatomical model generated in step S2, the system automatically calculates a theoretically optimal path from the preset skin puncture point to the target nerve region. This path must minimize the risk to dangerous structures while considering tissue safety. During the actual advancement of the puncture needle, the system continuously analyzes the real-time ultrasound image stream, estimating the movement trend and direction of minute movements within the tissue by comparing pixel changes between consecutive frames. The system continuously monitors the relative position of the puncture needle tip with the planned path and the danger zone. Once a collision is predicted if the needle is advanced along the current trajectory, the system immediately integrates the current needle tip position, movement trend, and spatial orientation of the danger zone to calculate a new needle advancement direction guide to avoid risks.

[0048] S4. Augmented Reality Navigation and Multi-Sensory Guidance: The 3D path, real-time obstacle avoidance direction guidance, and reconstructed anatomical model calculated in step S3 are precisely integrated into the surgeon's real-time surgical video or microscope field of view using graphics overlay technology, forming an augmented reality view. The system generates intuitive visual guidance information, such as an arrow that dynamically adjusts its direction according to changes in risk, and a highlighted area that changes color when approaching danger; simultaneously, it generates sound signals of different tones and frequencies to guide the surgeon in adjusting the needle insertion angle, depth, and direction, ensuring that the puncture process remains on a safe path.

[0049] S5. Surgical Process Data Recording and Adaptive Optimization: The system automatically archives multimodal image data of the entire surgical procedure, instrument movement trajectories, all planning and obstacle avoidance decision data, and surgical results. This data, after being anonymized, is stored in a distributed big data platform, forming a continuously growing surgical case library. These new case data are periodically used to retrain and optimize the recognition model in step S2 and the path planning and obstacle avoidance logic in step S3, enabling the system's recognition accuracy and decision-making intelligence to continuously improve with increased usage.

[0050] Furthermore, step S1 also includes:

[0051] High-resolution static image data includes one or more combinations of MRI images, CT images, or PET-CT images; the establishment of the spatiotemporal synchronization coordinate system is achieved by attaching positioning markers to the patient's body surface that are visible in multimodal images.

[0052] In this embodiment, the resolution static image data is T2-weighted magnetic resonance imaging (MRI) data or CT angiography (CTA) data; the spatiotemporal synchronization is achieved by fixing at least three markers containing barium sulfate or vitamin E on the patient's body, which can be clearly identified in all imaging modes and used for accurate registration.

[0053] Furthermore, step S2 also includes:

[0054] The pre-trained deep learning segmentation model was trained using a large-scale historical desensitized neural image dataset within a distributed computing framework. Data parallelism was employed during the model training process to accelerate the processing of massive amounts of training data.

[0055] In this embodiment, in step S2, the training process of the deep learning segmentation model is completed on a distributed computing cluster. This cluster processes massive amounts of historical neural image data in parallel and continuously adjusts millions of parameters within the model to enable it to accurately distinguish subtle features of different anatomical structures.

[0056] Furthermore, step S3 also includes:

[0057] The path search algorithm is an improved algorithm based on the A* algorithm or the Fast Random Exploration Tree (RRT) algorithm. Its cost function integrates multiple factors such as path length, distance from the danger zone, and the permeability of the organization.

[0058] In this embodiment, in step S3, the optimal path planning strategy prioritizes the route with the shortest path and the largest safety margin from dangerous structures such as blood vessels and bones; the real-time dynamic obstacle avoidance calculation infers motion by analyzing the displacement of tissue texture in real-time images and simulates a thrust to move the needle tip away from the danger zone to generate a guidance vector.

[0059] Furthermore, step S3 also includes:

[0060] The real-time obstacle avoidance vector calculation is specifically as follows: a real-time local obstacle avoidance potential field is constructed with the puncture needle tip as the center, a repulsive field is generated in the danger zone, and an attractive field is generated in the target zone;

[0061] The obstacle avoidance guidance vector is the resultant force direction vector of the potential field at the current needle tip position.

[0062] In this embodiment, step S3 specifically involves the following: the system assigns a repulsive force field that changes over time to the dangerous area and a fixed attractive force field to the target point within the software; calculates the direction in which the robotic arm or needle tip should move under the current resultant force, and converts this direction into an intuitive visual signal for the operator.

[0063] Furthermore, in step S4, the dynamic guidance information includes, but is not limited to:

[0064] A dynamic guide arrow indicating the correct needle insertion direction is overlaid on the real-time video, and the area is highlighted and a graded audible alarm is emitted when approaching a danger zone.

[0065] In this embodiment, step S4 includes the following dynamic guidance information: in the augmented reality field of view, a semi-transparent green arrow that always points in a safe direction turns yellow and flashes when a risk is detected, while the danger zone model turns red; audibly, a prompt is emitted that changes from an intermittent beeping sound to a continuous alarm sound.

[0066] Furthermore, step S5 also includes:

[0067] The big data warehouse for nerve puncture surgery is built using a big data platform based on Hadoop or Spark. The data warehouse not only stores the raw data, but also stores high-quality feature datasets after feature extraction, which are used for subsequent analysis and model optimization.

[0068] In this embodiment, in step S5, the big data platform uses columnar storage and a distributed file system to manage massive amounts of surgical procedure data, and utilizes its parallel computing capabilities to extract valuable feature patterns from the massive data for incremental learning of the model.

[0069] Furthermore, step S5 also includes:

[0070] Iterative optimization and incremental learning specifically involve periodically sampling new surgical data from the big data warehouse, fine-tuning the segmentation model and path planning algorithm, and continuously updating and enriching the obstacle avoidance rule base using obstacle avoidance cases in the new data.

[0071] In this embodiment, in step S5, the adaptive optimization refers to: the system periodically extracts the latest accumulated successful and risky cases, conducts targeted reinforcement training on them without changing the core knowledge of the original model, and uses new obstacle avoidance scenarios to improve its obstacle avoidance rule base, thereby continuously improving the system performance.

[0072] Furthermore, the method also includes:

[0073] S6, Surgical plan simulation and prognosis prediction: Based on surgical data and prognosis information of similar cases in the historical big data warehouse, multiple simulated puncture paths are generated for the current patient before surgery, and the success rate, risk index and expected recovery effect of each path are predicted to assist doctors in making preoperative decisions.

[0074] The method operates within a navigation system that integrates a high-performance computing unit and a real-time data processing unit. The real-time data processing unit continuously processes dynamic image stream data using stream processing technology to ensure the real-time nature of obstacle avoidance guidance.

[0075] In this embodiment, the method further includes a pre-planning step before step S1:

[0076] S0. Surgical Plan Simulation and Prognostic Analysis: The system retrieves complete data of similar past cases from the big data platform, including images, pathways, results, and rehabilitation records. It generates several alternative surgical plans for the current patient and performs virtual puncture simulation. It also predicts the possible surgical effects and risks of each plan, assisting doctors in preoperative planning and decision-making.

[0077] The entire computational process of the method is supported by a computer system that integrates a graphics processing unit and a high-performance central processing unit. The processing of real-time ultrasound video streams is completed by a dedicated streaming data processing engine to ensure that the latency from image acquisition to the generation of guidance information is extremely low, meeting the requirements of real-time surgery.

[0078] According to a second embodiment of the present invention, the present invention claims protection for a dynamic obstacle avoidance navigation system for neural puncture based on multimodal image fusion, referring to... Figure 2 ,include:

[0079] One or more processors;

[0080] A memory storing one or more programs, which, when executed by one or more processors, enable the one or more processors to implement the aforementioned method for dynamic obstacle avoidance navigation of neural puncture based on multimodal image fusion.

[0081] This embodiment aims to describe in detail the practical application of a dynamic obstacle avoidance navigation method for nerve puncture based on multimodal image fusion. By combining multimodal image data, artificial intelligence segmentation, real-time dynamic obstacle avoidance, and big data optimization, it achieves accurate and safe navigation for nerve puncture surgery. This embodiment will gradually demonstrate the specific implementation of the method and provide experimental test data to verify its effectiveness.

[0082] Experimental subjects: Fifty patients who needed lumbar nerve root block surgery were randomly divided into two groups: the experimental group (25 cases) underwent navigation puncture using this method, and the control group (25 cases) underwent traditional ultrasound-guided puncture.

[0083] Preoperative imaging equipment: 3.0T MRI scanner (model: Philips Achieva) and 128-slice CT scanner (model: Siemens Somatom Definition AS+).

[0084] Intraoperative imaging equipment: Portable color Doppler ultrasound diagnostic instrument (model: Mindray M9).

[0085] Navigation system: A dedicated workstation equipped with an NVIDIA RTX A6000 graphics card and an Intel Xeon Gold 6330 processor.

[0086] Location markers: Medical skin markers containing barium sulfate.

[0087] Software environment:

[0088] Image fusion and registration software: A customized platform developed based on the ITK and VTK libraries.

[0089] Deep learning segmentation model: U-Net++ model trained on the PyTorch framework, with training data from 5000 anonymized neural image datasets.

[0090] Path planning and obstacle avoidance algorithm: implemented based on the ROS (Robot Operating System) framework.

[0091] Big Data Platform: A distributed storage and computing cluster built on Hadoop 3.3.4 and Spark 3.3.1.

[0092] Step S1: Multimodal image data acquisition and synchronous fusion

[0093] Procedure: Preoperative T2-weighted MRI and CT scans of the lumbar spine were performed on each patient in the experimental group, with a slice thickness of 1 mm. One hour preoperatively, four barium sulfate-containing markers were affixed to the patient's lumbar region. During the procedure, the patient was placed in a prone position, and the ultrasound probe, after being coated with coupling gel, was placed over the marker area to acquire real-time ultrasound images. The navigation system automatically identified the positions of the markers in the MRI, CT, and ultrasound images. Through affine transformation and a non-rigid registration algorithm, the high-resolution structures of MRI and CT were fused with the real-time ultrasound images to form a fused image model in a unified coordinate system. The registration error was controlled within 1.5 mm.

[0094] Test results: The fused image model was successfully generated, and the average registration error of the marker points was 1.2 mm (range 0.8-1.5 mm), which meets the clinical accuracy requirements.

[0095] Step S2: Intelligent Recognition and 3D Reconstruction of Key Anatomical Structures

[0096] Implementation process: The fused image model is input into a pre-trained U-Net++ segmentation model, which automatically outputs segmentation masks for nerve roots, dural sacs, vertebrae, and blood vessels (such as lumbar arteries). Based on the segmentation results, the system uses the moving cubes algorithm to perform 3D reconstruction, generating an interactive 3D anatomical model, which is then overlaid on real-time ultrasound images for display.

[0097] Test results: The model segmentation accuracy (Dice coefficient) reached 94.5%, the 3D reconstruction time was 3.2 seconds on average, and the model rendering frame rate was stable at 30fps.

[0098] Step S3: Optimal path planning and real-time dynamic obstacle avoidance calculation

[0099] Implementation Process: The operator selects the skin entry point and target nerve root (e.g., L4 nerve root) via the interface. The system calculates the initial path based on the A* algorithm, prioritizing the shortest route that is >5mm from the blood vessel. During the puncture, the system analyzes the ultrasound video stream (30 frames / second) in real time and calculates the tissue motion vector using optical flow. When the needle tip is <3mm from the blood vessel, the system triggers an obstacle avoidance mechanism, calculating and displaying an obstacle avoidance guidance vector (e.g., "Adjust 15° ventrally").

[0100] Test results: Initial path planning took an average of 0.8 seconds. Obstacle avoidance calculation latency was <100ms. In the experimental group of 25 punctures, the system triggered obstacle avoidance prompts 37 times, of which 35 successfully avoided blood vessels, achieving an obstacle avoidance success rate of 94.6%.

[0101] Step S4: Augmented Reality Navigation and Multi-Sensory Guidance Prompts

[0102] Implementation process: The operator wears a head-mounted AR display (Microsoft HoloLens 2), which displays the needle position, 3D anatomical model, planned path (green dashed line), and obstacle avoidance guide arrows (normally green, flashing yellow during warnings) in real time within their field of vision. When the needle tip is less than 2mm from the danger zone, the system emits a buzzer alarm, with the tone becoming more urgent the closer the distance.

[0103] Test results: The surgeon reported that the AR guidance was intuitive, and the visual overlay error was <1.5mm. The auditory alarm sensitivity was 100%, with no missed alarms.

[0104] Step S5: Recording and Adaptive Optimization of Surgical Process Data

[0105] Implementation process: The fused images, segmentation results, path planning, needle trajectory, and surgeon's operation records for each surgery are stored in JSON format to the Hadoop Distributed File System (HDFS). Every week, the system automatically extracts new data, incrementally trains the segmentation model (learning rate 0.0001, 50 iterations), and updates the obstacle avoidance rule base.

[0106] Test results: After incremental training with 10 surgical data cases, the Dice coefficient of the segmentation model improved to 95.8%. 12 new rules for high-risk scenarios were added to the obstacle avoidance rule base.

[0107] Regarding the image data and markers, the implementation process is as follows: In this embodiment, static images were obtained using T2-MRI (TR=3200ms, TE=102ms) and CT (120kV, 250mAs). The markers were 5mm diameter barium sulfate dots, which were affixed to the spinous processes of L3, L4, and L5 and the upper edge of the sacrum.

[0108] Test results: All markers were clearly visible in MRI, CT and ultrasound, with no detachment or displacement.

[0109] Distributed model training implementation process: The segmentation model is trained on a cluster of 8 servers, each equipped with 4 NVIDIA A100 GPUs. The training data consists of 5000 anonymized images, employing a data parallel strategy and synchronous gradient descent.

[0110] Test results: Training time was reduced from 120 hours for a single machine to 18 hours for a cluster, a speedup of 6.7 times.

[0111] Path planning strategy implementation process: The path cost function is defined as: f = path length + 10 × (1 / minimum safe distance) + 5 × organization penetration difficulty coefficient. The safe distance threshold is set to 3mm.

[0112] Test results: The average safe distance of the planned path was 4.8mm, and no path crossed bones or major blood vessels.

[0113] The process of obstacle avoidance using a potential field: In the obstacle avoidance potential field, the intensity of the repulsive force field of the blood vessels is inversely proportional to the square of the distance, while the gravitational field at the target point is constant. The direction of the resultant force is obtained by calculating the potential field gradient.

[0114] Test results: The potential field calculation took an average of 15ms, and the average deviation between the steering vector and the expert's expected direction was 4.3°.

[0115] Multi-sensory guidance implementation details: Visual alarm: The arrow flashes yellow when the distance to the danger zone is 3-2mm, and turns red and flashes at a high frequency when the distance is <2mm. Auditory alarm: Correspondingly emits a 400Hz and 800Hz buzzing sound.

[0116] Test results: The surgeon's reaction time was 0.5 seconds shorter than that of pure visual cues.

[0117] Big data platform architecture implementation process: The HDFS cluster consists of 5 nodes with a total storage capacity of 800TB. Surgical data is stored in Parquet columnar format, and Spark is used for feature extraction (such as extracting needle trajectory curvature, speed, etc.).

[0118] Test results: The average data storage size for a single surgical case was 2.1GB, and the average time for feature extraction was 3 minutes per case.

[0119] Incremental learning process: Every Monday morning, the Spark job automatically loads new data from HDFS and fine-tunes the segmentation model (freezing the encoder layer and training only the decoder), which takes about 2 hours.

[0120] Test results: After incremental learning, the model's Dice coefficient improved by an average of 1.3% on the new data.

[0121] Preoperative simulation and prediction process: The system retrieves 50 similar cases (same segment, similar age, and anatomical variations) from the big data platform, simulates 3 pathways, and predicts the success rate. Predictive features include pathway length, safe distance, and historical complication rate of similar pathways.

[0122] Test results: The correlation coefficient between the predicted success rate and the actual success rate reached 0.89.

[0123] Real-time performance assurance implementation process: The ultrasonic video stream is accessed through a GStreamer pipeline and processed by a dedicated thread with real-time priority. The obstacle avoidance calculation task is bound to a high-performance CPU core.

[0124] Test results: The end-to-end latency from image acquisition to AR display is 85ms, which meets the requirements for real-time interaction.

[0125] The comprehensive test results include:

[0126] Puncture accuracy: The error of the needle tip reaching the target in the experimental group was (1.3±0.6) mm, while that in the control group was (2.8±1.2) mm (p<0.01).

[0127] Operation time: The average puncture time was 8.5 minutes in the experimental group and 12.3 minutes in the control group (p<0.05).

[0128] Complication rate: No vascular injury or nerve stimulation events occurred in the experimental group, while 2 cases of minor vascular perforation occurred in the control group.

[0129] System performance: AR rendering frame rate 30fps, obstacle avoidance calculation latency <100ms, data archiving throughput 1.2GB / s.

[0130] This embodiment fully verifies the effectiveness, accuracy, and safety of the navigation method. Through multimodal image fusion, AI intelligent segmentation, real-time dynamic obstacle avoidance, and big data optimization closed loop, the success rate and safety of nerve puncture surgery are significantly improved, demonstrating significant clinical value and technological innovation.

[0131] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, or indirect coupling or communication connection between apparatuses or units, and may be electrical, mechanical, or other forms.

[0132] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated units described above can be implemented in hardware or as software functional units. The above are merely embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made based on the description and drawings of this application, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

[0133] The specific embodiments of the invention have been described in detail above, but they are only examples, and this application is not limited to the specific embodiments described above. For those skilled in the art, any equivalent modifications or substitutions to the invention are also within the scope of this application. Therefore, all equivalent changes, modifications, and improvements made without departing from the spirit and principles of this application should be covered within the scope of this application.

Claims

1. A dynamic obstacle avoidance navigation system for neural puncture based on multimodal image fusion, characterized in that, include: One or more processors; A memory storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement a neural puncture dynamic obstacle avoidance navigation method based on multimodal image fusion, including: S1. High-resolution static image data is acquired before surgery using medical imaging equipment, and dynamic image stream data is acquired during surgery using real-time ultrasound imaging equipment. A spatiotemporal synchronous coordinate system is constructed, and the static image data and dynamic image stream data are registered and fused in time and space to generate a multimodal fusion image base with a unified spatiotemporal reference. S2, based on a pre-trained deep learning segmentation model, performs pixel-level semantic segmentation on the multimodal fusion image substrate, identifies and reconstructs the target neural structure, key blood vessels, bones, and dangerous areas to be avoided in three dimensions; and displays the segmentation results overlaid with the preoperative planned path. S3, based on the segmented three-dimensional anatomical structure, uses a path search algorithm to calculate an optimal initial path from the puncture site to the target nerve region; receives the dynamic image stream data in real time, calculates the motion vector of the intraoperative medium using optical flow, and monitors the deviation from the initial path or danger zone in real time. When a potential collision risk is detected, an obstacle avoidance guidance vector is calculated in real time based on the motion vector and the spatial relationship of the anatomical structure; the real-time obstacle avoidance vector calculation specifically involves constructing a real-time local obstacle avoidance potential field centered on the puncture needle tip, generating a repulsive field in the danger zone and an attractive field in the target zone; the obstacle avoidance guidance vector is the resultant force direction vector of this potential field at the current needle tip position; S4. The initial path, real-time instrument pose, obstacle avoidance guidance vector, and 3D reconstruction model of key structures are used to generate dynamic visual and auditory guidance information based on the obstacle avoidance guidance vector, prompting the operator to adjust the needle insertion angle, depth, or direction. S5 continuously records multimodal imaging data, instrument movement trajectory, surgeon operation details, and final result data throughout the entire puncture process, building a big data warehouse for nerve puncture surgery; S6, Surgical plan simulation and prognosis prediction: Based on surgical data and prognosis information of similar cases in the historical big data warehouse, multiple simulated puncture paths are generated for the current patient before surgery, and the success rate, risk index and expected recovery effect of each path are predicted to assist doctors in making preoperative decisions. In S3, during the actual advancement of the puncture needle, the system continuously analyzes the real-time ultrasound image stream and estimates the movement trend and direction of minute movements in the tissue by comparing pixel changes between consecutive frames. The system continuously monitors the relative position of the puncture needle tip with the planned path and the danger zone. Once it is predicted that a collision will occur if the needle is advanced along the current trajectory, it immediately integrates the current needle tip position, movement trend and spatial orientation of the danger zone to calculate a new needle advancement direction guide to avoid risks.

2. The neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion according to claim 1, characterized in that, Step S1 also includes: High-resolution static image data includes one or more combinations of MRI images, CT images, or PET-CT images; the establishment of the spatiotemporal synchronization coordinate system is achieved by attaching positioning markers to the patient's body surface that are visible in multimodal images.

3. The neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion according to claim 1, characterized in that, Step S2 also includes: The pre-trained deep learning segmentation model was trained using a large-scale historical desensitized neural image dataset within a distributed computing framework. Data parallelism was employed during the model training process to accelerate the processing of massive amounts of training data.

4. The neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion according to claim 1, characterized in that, Step S3 also includes: The path search algorithm is an improved algorithm based on the A* algorithm or the Fast Random Exploration Tree (RRT) algorithm. Its cost function integrates multiple factors such as path length, distance from the danger zone, and the permeability of the organization.

5. The neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion according to claim 1, characterized in that, In step S4, the dynamic visual and auditory guidance information includes, but is not limited to: A dynamic guide arrow indicating the correct needle insertion direction is overlaid on the real-time video, and the area is highlighted and a graded audible alarm is emitted when approaching a danger zone.

6. The neural puncture dynamic obstacle avoidance navigation system based on multimodal image fusion according to claim 1, characterized in that, Step S5 also includes: The big data warehouse for nerve puncture surgery is built using a big data platform based on Hadoop or Spark. The data warehouse not only stores the raw data, but also stores high-quality feature datasets after feature extraction, which are used for subsequent analysis and model optimization.

7. A dynamic obstacle avoidance navigation system for neural puncture based on multimodal image fusion according to claim 6, characterized in that, Step S5 also includes: Iterative optimization and incremental learning specifically involve periodically sampling new surgical data from the big data warehouse, fine-tuning the segmentation model and path planning algorithm, and continuously updating and enriching the obstacle avoidance rule base using obstacle avoidance cases in the new data.

Images