Neurosurgery navigation system and method based on light projection

Abstract

The invention relates to the technical field of medicine, in particular to a neurosurgery navigation system and method based on light projection, the system comprises an image acquisition unit, a data processing unit and a projection unit, the data processing unit is integrated with a deformation simulation engine, a first AI module, a second AI module and a third AI module; compared with the prior art that navigation depends on static preoperative images, and guide information distortion is caused by intraoperative tissue displacement, according to the scheme, soft tissue deformation is dynamically compensated by establishing a real-time digital twin model and integrating a physical deformation simulation engine, so that navigation information can continuously and accurately cover a real anatomical position, and the navigation accuracy is improved. The problem of guiding errors is effectively solved, and the reliability and accuracy of an operation are greatly improved.

Description

Technical Field

[0001] This invention relates to the field of medical technology, and in particular to a neurosurgical navigation system and method based on light projection. Background Technology

[0002] Neurosurgical navigation systems are crucial tools in modern precision medicine. By establishing a mapping between preoperative images and the actual surgical space, they provide surgeons with visual guidance of deep anatomical structures, aiming to improve surgical safety, reduce tissue damage, and enhance patient outcomes. Traditional and current mainstream navigation systems primarily rely on optical or electromagnetic tracking technologies to register reconstructed 3D image data with the patient's position and present virtual navigation information on a monitor.

[0003] However, existing technologies have significant limitations. First, these systems generally rely on preoperative static images and cannot compensate in real time for brain shifts caused by instrument manipulation, gravity, cerebrospinal fluid loss, and tissue resection during surgery. This results in severe deviations between navigation information and the actual anatomical location, with accuracy continuously decreasing as the surgery progresses. Second, surgeons must frequently switch their gaze between the surgical field and external monitors, disrupting the surgical rhythm and increasing cognitive load and operational errors. Furthermore, the identification of key structures heavily depends on the surgeon's subjective experience and visual judgment, lacking real-time objective histological feedback; the switching of surgical stages and risk warnings are also largely determined manually, lacking intelligent decision support. These shortcomings collectively limit the reliability and practicality of navigation systems in complex surgeries.

[0004] This invention aims to solve the aforementioned problems through an integrated system that combines multimodal perception, artificial intelligence, and augmented reality projection. Its core lies in constructing a real-time updated digital twin model of the surgical area. This model continuously perceives the surgical field through binocular vision and hyperspectral imaging, utilizes a physics engine to simulate soft tissue deformation and perform reverse compensation to correct the projected coordinates, and integrates multiple AI modules to achieve automatic tissue recognition, surgical phase perception, and predictive risk alerts. Finally, the dynamically adjusted navigation information is precisely projected directly into the surgical field as light, creating an augmented reality navigation experience that is deeply integrated with the real surgical environment, intelligently adaptive, and intuitive. Summary of the Invention

[0005] To overcome the problems mentioned in the background art, the present invention proposes a neurosurgical navigation system and method based on light projection.

[0006] The technical solution of this invention is: a neurosurgical navigation system based on light projection, comprising: The image acquisition unit is used to acquire real-time image data of the surgical area; A data processing unit, communicatively connected to the image acquisition unit, is used to process the image data; The projection unit, which is communicatively connected to the data processing unit, is used to project navigation information onto the surgical area in the form of light rays.

[0007] Preferably, the image acquisition unit includes: A binocular vision camera, positioned above the surgical field of view, is used to continuously capture a sequence of binocular images of the surgical area in order to establish and update a three-dimensional digital twin model of the surgical area. Hyperspectral imaging camera, used to capture hyperspectral image data of the surgical area.

[0008] Preferably, the data processing unit is configured as follows: Based on binocular image sequences, a real-time three-dimensional surface model of the surgical area is reconstructed using stereo vision algorithms, serving as the initial digital twin model; It integrates a physics-based deformation simulation engine, which receives real-time instrument tracking data and tissue displacement data as input to simulate soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model. The navigation information based on preoperative image planning will be transformed and reverse-mapped according to the updated digital twin model to generate corrected projection commands to compensate for actual deformation.

[0009] Preferably, when the data processing unit reconstructs a real-time three-dimensional surface model of the surgical area based on a stereoscopic vision algorithm using a binocular image sequence as the initial digital twin model, it specifically includes: S11: Image acquisition, the left and right views are acquired simultaneously by a binocular vision camera to form a binocular image pair; S12: Camera calibration, based on the internal and external parameters of the binocular camera obtained in advance through the checkerboard calibration method, to establish the imaging model of the camera; S13: Image preprocessing and correction. The original binocular image pairs acquired are distorted using the camera's internal and external parameters, and stereo correction is performed to ensure that the epipolar lines of the two images are strictly aligned in the horizontal direction, generating a corrected image pair for stereo matching. S14: Stereo matching. Pixel matching is performed on the corrected image pairs. A matching algorithm is used to calculate the corresponding pixel in the right view for each pixel in the left view, thereby calculating a disparity map representing the disparity value of each pixel. The formula for calculating the disparity value is as follows: ;in, For parallax, The x-coordinate of the pixels in the left view. The x-coordinate of the pixels in the right view; S15: 3D coordinate calculation. Based on the principle of triangulation, using the disparity map and camera parameters, the 3D coordinates of each pixel in the image are calculated in the binocular camera coordinate system. The principle formula is as follows: ; in, This represents the actual physical distance between the pixel and the camera. For the camera's focal length, The baseline distance between the two cameras; S16: Point cloud generation and surface reconstruction. This process generates a dense 3D point cloud from a set of pixels with 3D coordinates, and then applies filtering and surface reconstruction algorithms to the point cloud to generate a continuous meshed 3D surface model.

[0010] The stereo matching step employs a semi-global matching algorithm or a deep learning-based optical flow / stereo matching neural network to calculate the disparity map.

[0011] The surface reconstruction algorithm used is either the Poisson reconstruction algorithm or the moving cube algorithm.

[0012] This includes model optimization after point cloud generation and surface reconstruction. The reconstructed 3D surface model is initially registered with the 3D organ model segmented from preoperative images. Fine registration is then performed using an iterative nearest point algorithm or a similar registration algorithm to spatially align the preoperatively planned navigation information with the real-time reconstructed visual model.

[0013] Preferably, the deformation simulation engine, when receiving real-time instrument tracking data and tissue displacement data as input, simulates soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model, specifically includes: S21: Data input, continuously receiving real-time data streams from the binocular vision camera; S22: Physical modeling, which transforms the reconstructed initial digital twin 3D surface model into a mechanical model with physical properties; in this process, soft tissue is discretized into a mass-spring system, and biomechanical material properties are assigned to the units in the model, including Young's modulus, Poisson's ratio, viscoelastic parameters, and tissue-type related density. S23: External force calculation and application. Based on the input real-time data, calculate and apply the external forces acting on the mechanical model. S24: Deformation simulation calculation, based on the applied external force and the constitutive equation of the material, calculates the new equilibrium state of the mechanical model under the action of the new external force by solving the dynamic equation and the static equilibrium equation, and generates a digital twin model to predict the deformation. S25: Model update, update the new vertex coordinates obtained from deformation simulation to the geometric data of the digital twin model, and complete the dynamic update of the digital twin model.

[0014] The principle formula for solving the dynamic equations is as follows: ; in, For the quality matrix, Here is the damping matrix. Here is the stiffness matrix. Let be the displacement vector of the node. Let be the velocity vector of the node. Let be the acceleration vector of the node. The force vector applied from the outside.

[0015] Preferably, when the deformation simulation engine calculates and applies external forces acting on the mechanical model based on real-time input data, it includes: For instrument operation, based on the instrument's pose and motion trajectory, the collision detection between the instrument and the mechanical model is determined, and the calculated instrument contact force and displacement boundary conditions are applied to the corresponding elements of the model. For gravity, a vertically downward volume force is continuously applied to all elements of the model; For cerebrospinal fluid loss, the buoyancy and constraint forces of surrounding tissues on the model are dynamically reduced to simulate the effect using a time-dependent predefined model.

[0016] Preferably, when the data processing unit performs coordinate transformation and reverse mapping on the navigation information planned based on preoperative images according to the updated digital twin model to generate corrected projection commands and compensate for actual deformation, it specifically includes: S31: Establish a mapping relationship. Before any deformation occurs, establish a corresponding mapping relationship between each pre-planned navigation point in the preoperative image coordinate system and the nearest vertex on the surface of the initial digital twin model. S32: Deformation field transfer. After the deformation simulation engine calculates the deformation of the digital twin model from the initial state to the current state, it obtains the displacement vector of each vertex and obtains the overall deformation field of the model. S33: Reverse mapping. For each pre-planned navigation point, find its corresponding initial model vertex according to the mapping relationship and read the displacement of the vertex in the current deformation state. Then, perform reverse calculation, apply the displacement vector in reverse to the pre-planned navigation point, and calculate the theoretical position point of the current deformed field. S34: Coordinate system transformation transforms the coordinates of the corrected theoretical position points from the preoperative image coordinate system to the projector coordinate system. This is achieved through a composite transformation matrix, which is obtained by multiplying the transformation matrices from the preoperative image coordinate system to the world coordinate system, from the world coordinate system to the camera coordinate system, and from the camera coordinate system to the projector coordinate system. The principle formula is as follows: ; in, These are the coordinates after the projector coordinates have been transformed. It is a composite transformation matrix. These are the coordinates of the corrected theoretical position point; S35: Projection command generation. The coordinates transformed to the projector coordinate system are converted into pixel coordinates on the projector image plane. Combined with the navigation information, the final digital projection command is generated and sent to the projection unit for execution.

[0017] The mapping relationship is established by using a mesh surface fusion or nearest point search algorithm to combine the 3D model of the pre-planned navigation information with the initial digital twin surface model.

[0018] When deformation causes significant shearing or rotation of the tissue, an inverse mapping algorithm based on affine transformation or higher-order deformation field is used during inverse mapping.

[0019] Preferably, before generating the projection command, it also includes determining whether the target projection point is occluded from the current viewpoint. If it is occluded, it is projected onto the edge of the occluder as a reminder, specifically including: S41: Scene depth map construction. Based on the current 3D point cloud data of the surgical area, a depth map representing the surface depth of objects in the scene is generated from the perspective of the projector through a surface reconstruction algorithm. S42: Projection point depth acquisition. For a point to be projected, calculate the depth value in the projector coordinate system. S43: Depth comparison and occlusion judgment: Project the point onto the projector image plane to obtain its corresponding pixel coordinates; read the depth value corresponding to the pixel coordinates in the scene depth map, and make occlusion judgment by comparing the two depth values. S44: Edge alert, identify the outline of the object causing the occlusion on the projector image plane; calculate the set of edge pixels of the outline, and project the original projection content onto the edge area of ​​the occluding object's outline.

[0020] The principle behind occlusion detection is as follows: like If so, the judgment point is occluded; in, This is the calculated depth value of the point to be projected in the projector coordinate system. These are the pixel coordinates obtained by projecting a point onto the image plane of the projector. These are the actual depth coordinates. This is the tolerance threshold.

[0021] The construction of the scene depth map is efficiently completed on the graphics processor using rasterization technology.

[0022] Preferably, the data processing unit also integrates a first AI module, used for: The preoperative MRI and CT images of patients are automatically segmented to identify key structures, including the three-dimensional contours of tumors, blood vessels, nerves and functional areas. Based on the segmentation results and a pre-defined surgical safety rule base, the system automatically calculates and recommends the optimal surgical approach and safe resection boundaries.

[0023] Preferably, the data processing unit also integrates a second AI module for: Real-time analysis of the spectral data captured by the hyperspectral imaging camera; Based on the reflectance spectral characteristics of tissues, real-time pixel-level classification and identification of surgical field tissues can be performed to distinguish tissue types including normal brain tissue, tumor tissue, blood vessels and nerves. The recognition results are converted into augmented reality (AR) projection content, and the projection unit is controlled to project different tissues in specific colors or patterns onto the corresponding positions in the surgical field.

[0024] Preferably, the data processing unit also integrates a third AI module for: Continuous analysis of the surgical video stream acquired by the image acquisition unit; The system automatically identifies the current surgical stage based on visual features in the video, including dural incision, cortical fistula, tumor resection, and hemostasis. Based on the identified surgical stage, the projection unit automatically controls the switching and adjustment of the projected information's layer, content, and display mode.

[0025] Preferably, the data processing unit also integrates a fourth AI module for: Access a data pool containing massive amounts of surgical data; Based on real-time surgical data, including instrument location, tissue classification results, and surgical stage, comparative analysis is performed with the data pool. When the current operation is identified as conforming to a preset high-risk pattern, a predictive alarm signal is generated.

[0026] A neurosurgical navigation method based on light projection includes the following steps: S51: Acquire real-time image data of the surgical area through the image acquisition unit; S52: Processing image data through the data processing unit, including: Establish and update a digital twin model of the surgical area in real time, and simulate deformation to correct the projected coordinates in reverse; analyze hyperspectral data in real time to identify and classify tissues; analyze surgical videos to identify surgical stages; perform risk prediction and generate alerts; S53: The corrected navigation content is projected onto the surgical area in real time through the projection unit based on the data processing results.

[0027] The beneficial effects of this invention are: 1. Compared with existing technologies that rely on static preoperative images for navigation, which suffer from the disadvantage of distorted guidance information due to intraoperative tissue displacement, this solution establishes a real-time digital twin model and integrates a physical deformation simulation engine to dynamically compensate for soft tissue deformation, so that the navigation information can continuously and accurately cover the real anatomical position, effectively solving the guidance error problem and greatly improving the reliability and accuracy of the surgery. 2. Compared with existing technologies that mainly rely on doctors to identify tissue types with their naked eyes, which has the disadvantages of strong subjectivity and easy to miss small lesions, this solution uses hyperspectral imaging and AI recognition technology to achieve real-time pixel-level classification of surgical field tissues, and intuitively distinguishes normal and diseased tissues through augmented reality projection, which significantly improves the integrity of tumor resection and the protection rate of key tissues. 3. Compared with existing technologies that require doctors to manually switch navigation interfaces and display modes, which are cumbersome and distracting, this solution uses AI to automatically identify the surgical stage and intelligently adjust the projection information layer and content, realizing context-adaptive presentation of navigation information. This allows doctors to focus entirely on the surgical operation, greatly improving the smoothness of the operation and the efficiency of human-computer interaction. 4. Compared with existing technologies that mainly rely on doctors' personal experience to judge surgical risks, which have the disadvantages of delayed early warning and susceptibility to subjective factors, this solution connects to a massive surgical database to perform real-time risk pattern matching and generate predictive alerts, enabling the surgical team to take preventive measures in advance and significantly reducing the probability of intraoperative complications. 5. Compared with existing technologies, which suffer from information interruption when the instrument obstructs the navigation target, thus affecting the continuity of the surgery, this solution uses depth perception and edge projection technology to intelligently project navigation information onto the outline edge of the obstruction when obstruction occurs. This avoids misleading display and maintains navigation continuity, ensuring the consistency of surgical decisions. 6. Compared with existing technologies that require doctors to manually perform preoperative image segmentation and path planning, which are time-consuming and lack standardized procedures, this solution uses AI to automatically segment key structures and intelligently recommend surgical plans based on a safety rule base, achieving standardization and intelligence in preoperative planning, and greatly improving preparation efficiency and planning quality. 7. Compared with existing technologies that rely on multiple independent systems, resulting in insufficient information integration and low collaborative efficiency, this solution organically integrates functions such as deformation compensation, tissue identification, stage judgment, and risk warning through a multimodal data fusion and unified processing platform. This creates an intelligent navigation environment that can dynamically adapt to the surgical process, thereby improving the overall safety and operational efficiency of the surgery. Attached Figure Description

[0028] Figure 1 The diagram shown is a schematic representation of the structure of the neurosurgical navigation system based on light projection of the present invention. Figure 2 The diagram shown is a flowchart of the neurosurgical navigation method based on light projection of the present invention. Detailed Implementation

[0029] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0030] Please see Figure 1 The present invention provides an embodiment: a neurosurgical navigation system based on light projection, comprising: The image acquisition unit is used to acquire real-time image data of the surgical area; A data processing unit, communicatively connected to the image acquisition unit, is used to process the image data; The projection unit, which is communicatively connected to the data processing unit, is used to project navigation information onto the surgical area in the form of light rays.

[0031] This embodiment provides a neurosurgical navigation system based on light projection. Its core lies in acquiring real-time surgical area data through an image acquisition unit, followed by automatic segmentation, tissue identification, surgical stage assessment, and deformation simulation via a data processing unit integrating artificial intelligence algorithms. Finally, the processed navigation information is precisely superimposed onto the real surgical field through a projection unit. This system creates an augmented reality surgical environment, significantly improving surgical accuracy and safety: it can dynamically compensate for brain tissue deformation to ensure continuous accuracy of navigation information, distinguish lesions from normal tissue beyond the limits of the naked eye, and intelligently predict risks, thereby greatly reducing over-reliance on surgeon experience, shortening surgical time, and reducing intraoperative decision-making uncertainty.

[0032] Preferably, the image acquisition unit includes: A binocular vision camera, positioned above the surgical field of view, is used to continuously capture a sequence of binocular images of the surgical area in order to establish and update a three-dimensional digital twin model of the surgical area. Hyperspectral imaging camera, used to capture hyperspectral image data of the surgical area.

[0033] Preferably, the data processing unit is configured as follows: Based on binocular image sequences, a real-time three-dimensional surface model of the surgical area is reconstructed using stereo vision algorithms, serving as the initial digital twin model; It integrates a physics-based deformation simulation engine, which receives real-time instrument tracking data and tissue displacement data as input to simulate soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model. The navigation information based on preoperative image planning will be transformed and reverse-mapped according to the updated digital twin model to generate corrected projection commands to compensate for actual deformation.

[0034] In this embodiment, the data processing unit reconstructs a 3D model of the surgical area in real time using binocular vision to establish an initial digital twin. It also integrates a physics-based deformation simulation engine, receiving instrument tracking and tissue displacement data to dynamically simulate soft tissue deformation caused by manipulation, gravity, and cerebrospinal fluid loss, thereby updating the digital twin model in real time. Finally, by transforming and mapping the preoperatively planned navigation information to the updated model, a corrected projection command is generated. The core benefit of this technical solution lies in achieving dynamic real-time compensation for tissue deformation, ensuring that the projection navigation information continuously and accurately covers the actual anatomical location of the surgical field. This effectively solves the guidance error problem caused by tissue displacement in traditional navigation systems, significantly improving the accuracy and reliability of the surgery.

[0035] Preferably, when the data processing unit reconstructs a real-time three-dimensional surface model of the surgical area based on a stereoscopic vision algorithm using a binocular image sequence as the initial digital twin model, it specifically includes: S11: Image acquisition, the left and right views are acquired simultaneously by a binocular vision camera to form a binocular image pair; S12: Camera calibration, based on the internal and external parameters of the binocular camera obtained in advance through the checkerboard calibration method, to establish the imaging model of the camera; S13: Image preprocessing and correction. The original binocular image pairs acquired are distorted using the camera's internal and external parameters, and stereo correction is performed to ensure that the epipolar lines of the two images are strictly aligned in the horizontal direction, generating a corrected image pair for stereo matching. S14: Stereo matching. Pixel matching is performed on the corrected image pairs. A matching algorithm is used to calculate the corresponding pixel in the right view for each pixel in the left view, thereby calculating a disparity map representing the disparity value of each pixel. The formula for calculating the disparity value is as follows: ;in, For parallax, The x-coordinate of the pixels in the left view. The x-coordinate of the pixels in the right view; S15: 3D coordinate calculation. Based on the principle of triangulation, using the disparity map and camera parameters, the 3D coordinates of each pixel in the image are calculated in the binocular camera coordinate system. The principle formula is as follows: ; in, This represents the actual physical distance between the pixel and the camera. For the camera's focal length, The baseline distance between the two cameras; S16: Point cloud generation and surface reconstruction. This process generates a dense 3D point cloud from a set of pixels with 3D coordinates, and then applies filtering and surface reconstruction algorithms to the point cloud to generate a continuous meshed 3D surface model.

[0036] The stereo matching step employs a semi-global matching algorithm or a deep learning-based optical flow / stereo matching neural network to calculate the disparity map.

[0037] In this embodiment, the data processing unit constructs a real-time digital twin model of the surgical area through a sophisticated and automated stereo vision process: First, images are simultaneously acquired by binocular cameras and calibrated, distortion corrected, and epipolar aligned. Then, high-precision stereo matching is performed using semi-global matching or deep learning neural network algorithms to calculate the key disparity map. Subsequently, based on the principle of triangulation, the disparity data is converted into a three-dimensional point cloud. Finally, a high-fidelity meshed three-dimensional model is generated through filtering and surface reconstruction. The beneficial effect of this technical solution is that it achieves millisecond-level, high-precision three-dimensional digital reproduction of the surgical area's geometry through a fully automated process, providing an accurate and reliable real-time geometric reference for subsequent deformation simulation and augmented reality projection, fundamentally ensuring the timeliness and authenticity of the core registration data of the entire navigation system.

[0038] The surface reconstruction algorithm used is either the Poisson reconstruction algorithm or the moving cube algorithm.

[0039] This includes model optimization after point cloud generation and surface reconstruction. The reconstructed 3D surface model is initially registered with the 3D organ model segmented from preoperative images. Fine registration is then performed using an iterative nearest point algorithm or a similar registration algorithm to spatially align the preoperatively planned navigation information with the real-time reconstructed visual model.

[0040] In this embodiment, when generating the 3D surface model, a continuous and complete organ surface model is constructed from the point cloud using Poisson reconstruction or moving cubes algorithm. Furthermore, iterative nearest-point registration algorithms are used to precisely register the reconstructed real-time model with the accurate organ model segmented from preoperative images. This achieves high-precision fusion of intraoperative real-time geometric data and preoperative multimodal planning information, completely unifying the coordinate system. This lays a solid spatial correspondence foundation for the subsequent accurate projection of key navigation information such as tumor boundaries and vascular pathways onto the dynamically changing real surgical field, ensuring the accuracy and reliability of augmented reality navigation.

[0041] Preferably, the deformation simulation engine, when receiving real-time instrument tracking data and tissue displacement data as input, simulates soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model, specifically includes: S21: Data input, continuously receiving real-time data streams from the binocular vision camera; S22: Physical modeling, which transforms the reconstructed initial digital twin 3D surface model into a mechanical model with physical properties; in this process, soft tissue is discretized into a mass-spring system, and biomechanical material properties are assigned to the units in the model, including Young's modulus, Poisson's ratio, viscoelastic parameters, and tissue-type related density. S23: External force calculation and application. Based on the input real-time data, calculate and apply the external forces acting on the mechanical model. S24: Deformation simulation calculation, based on the applied external force and the constitutive equation of the material, calculates the new equilibrium state of the mechanical model under the action of the new external force by solving the dynamic equation and the static equilibrium equation, and generates a digital twin model to predict the deformation. S25: Model update, update the new vertex coordinates obtained from deformation simulation to the geometric data of the digital twin model, and complete the dynamic update of the digital twin model.

[0042] The principle formula for solving the dynamic equations is as follows: ; in, For the quality matrix, Here is the damping matrix. Here is the stiffness matrix. Let be the displacement vector of the node. Let be the velocity vector of the node. Let be the acceleration vector of the node. The force vector applied from the outside.

[0043] In this embodiment, the deformation simulation engine uses real-time acquired instrument tracking and tissue displacement data as input to construct a soft tissue model with biomechanical properties based on a mass-spring system. It then precisely simulates the tissue deformation process under the coupled effects of multiple factors, such as surgical instrument operation, gravity, and cerebrospinal fluid loss, by solving dynamic equations, ultimately dynamically updating the geometry of the digital twin model. This achieves physically accurate prediction and real-time digital representation of dynamic brain tissue deformation, enabling the navigation system to proactively compensate for anatomical displacement caused by surgical procedures. It completely solves the problem of guidance information lag and distortion caused by tissue displacement in traditional rigid navigation, providing surgeons with precise augmented reality guidance that is truly aligned with the actual surgical field space.

[0044] Preferably, when the deformation simulation engine calculates and applies external forces acting on the mechanical model based on real-time input data, it includes: For instrument operation, based on the instrument's pose and motion trajectory, the collision detection between the instrument and the mechanical model is determined, and the calculated instrument contact force and displacement boundary conditions are applied to the corresponding elements of the model. For gravity, a vertically downward volume force is continuously applied to all elements of the model; For cerebrospinal fluid loss, the buoyancy and constraint forces of surrounding tissues on the model are dynamically reduced to simulate the effect using a time-dependent predefined model.

[0045] In this embodiment, the deformation simulation engine accurately simulates the mechanical effects during surgery through multiphysics coupling calculations: collision detection is performed based on real-time instrument pose and contact forces are applied, a gravity field is continuously applied to all tissue units, and buoyancy and constraint forces are dynamically adjusted through a time-dependent model to simulate the cerebrospinal fluid loss effect. This achieves full-element digital simulation of the complex mechanical environment within the surgical area. By using the physics engine to solve tissue deformation under the coupling of multiple external forces in real time, the digital twin model can reflect the dynamic response of real tissues with high fidelity, providing an accurate deformation data foundation for subsequent projection compensation and greatly enhancing the robustness and reliability of the navigation system in real surgical environments.

[0046] Preferably, when the data processing unit performs coordinate transformation and reverse mapping on the navigation information planned based on preoperative images according to the updated digital twin model to generate corrected projection commands and compensate for actual deformation, it specifically includes: S31: Establish a mapping relationship. Before any deformation occurs, establish a corresponding mapping relationship between each pre-planned navigation point in the preoperative image coordinate system and the nearest vertex on the surface of the initial digital twin model. S32: Deformation field transfer. After the deformation simulation engine calculates the deformation of the digital twin model from the initial state to the current state, it obtains the displacement vector of each vertex and obtains the overall deformation field of the model. S33: Reverse mapping. For each pre-planned navigation point, find its corresponding initial model vertex according to the mapping relationship and read the displacement of the vertex in the current deformation state. Then, perform reverse calculation, apply the displacement vector in reverse to the pre-planned navigation point, and calculate the theoretical position point of the current deformed field. S34: Coordinate system transformation transforms the coordinates of the corrected theoretical position points from the preoperative image coordinate system to the projector coordinate system. This is achieved through a composite transformation matrix, which is obtained by multiplying the transformation matrices from the preoperative image coordinate system to the world coordinate system, from the world coordinate system to the camera coordinate system, and from the camera coordinate system to the projector coordinate system. The principle formula is as follows: ; in, These are the coordinates after the projector coordinates have been transformed. It is a composite transformation matrix. These are the coordinates of the corrected theoretical position point; S35: Projection command generation. The coordinates transformed to the projector coordinate system are converted into pixel coordinates on the projector image plane. Combined with the navigation information, the final digital projection command is generated and sent to the projection unit for execution.

[0047] The mapping relationship is established by using a mesh surface fusion or nearest point search algorithm to combine the 3D model of the pre-planned navigation information with the initial digital twin surface model.

[0048] When deformation causes significant shearing or rotation of the tissue, an inverse mapping algorithm based on affine transformation or higher-order deformation field is used during inverse mapping.

[0049] In this embodiment, the data processing unit establishes a vertex mapping relationship between preoperative planning points and the digital twin model, performs reverse mapping calculations using the displacement field calculated by the deformation simulation engine, and inversely compensates the preoperative navigation coordinates to the theoretical position after deformation. Then, it accurately transforms the coordinates to the projector coordinate system through a multi-coordinate system transformation matrix chain to generate projection commands. This achieves inverse mathematical compensation for tissue deformation, effectively solves the mapping error during large shear deformation through affine transformation and high-order deformation field algorithms, and ensures that the preoperative planned navigation information maintains spatial consistency with the real-time deformed anatomical structure. Ultimately, this enables the projection system to accurately project key information such as tumor boundaries and vascular pathways to their corresponding physiological positions, providing surgeons with augmented reality navigation that is truly synchronized with the dynamic surgical field.

[0050] Preferably, before generating the projection command, it also includes determining whether the target projection point is occluded from the current viewpoint. If it is occluded, it is projected onto the edge of the occluder as a reminder, specifically including: S41: Scene depth map construction. Based on the current 3D point cloud data of the surgical area, a depth map representing the surface depth of objects in the scene is generated from the perspective of the projector through a surface reconstruction algorithm. S42: Projection point depth acquisition. For a point to be projected, calculate the depth value in the projector coordinate system. S43: Depth comparison and occlusion judgment: Project the point onto the projector image plane to obtain its corresponding pixel coordinates; read the depth value corresponding to the pixel coordinates in the scene depth map, and make occlusion judgment by comparing the two depth values. S44: Edge alert, identify the outline of the object causing the occlusion on the projector image plane; calculate the set of edge pixels of the outline, and project the original projection content onto the edge area of ​​the occluding object's outline.

[0051] The principle behind occlusion detection is as follows: like If so, the judgment point is occluded; in, This is the calculated depth value of the point to be projected in the projector coordinate system. These are the pixel coordinates obtained by projecting a point onto the image plane of the projector. These are the actual depth coordinates. This is the tolerance threshold.

[0052] The construction of the scene depth map is efficiently completed on the graphics processor using rasterization technology.

[0053] In this embodiment, a scene depth map is constructed before projection and compared in real time with the depth of the projection point to accurately determine whether the target point is obscured by instruments or tissue. When obscuration is detected, the system automatically and intelligently shifts the original projection content to the edge contour area of ​​the obscured object for display. The beneficial effect of this technical solution is that it effectively solves the problem of missing navigation information caused by surgical instruments or tissue obstruction. By using edge projection reminders, doctors can continuously perceive the anatomical structure information of the obscured area, which avoids misleading projection and maintains the continuity of navigation, significantly improving the smoothness of the surgical procedure and the coherence of decision-making.

[0054] Preferably, the data processing unit also integrates a first AI module, used for: The preoperative MRI and CT images of patients are automatically segmented to identify key structures, including the three-dimensional contours of tumors, blood vessels, nerves and functional areas. Based on the segmentation results and a pre-defined surgical safety rule base, the system automatically calculates and recommends the optimal surgical approach and safe resection boundaries.

[0055] In this embodiment, the data processing unit integrates a first AI module, which uses deep learning algorithms to perform fully automated, high-precision segmentation of preoperative MRI / CT images. This accurately identifies the three-dimensional contours of key structures such as tumors, blood vessels, and nerves. Based on the segmentation results and a pre-set surgical safety rule base, it intelligently calculates and recommends the optimal surgical approach and safe resection boundaries. This achieves intelligent and standardized preoperative planning, significantly improving the efficiency and accuracy of anatomical structure segmentation. Simultaneously, data-driven safety rules provide doctors with scientifically reliable surgical path decision support, effectively reducing reliance on subjective experience and significantly improving the safety and reliability of surgical planning.

[0056] Preferably, the data processing unit also integrates a second AI module for: Real-time analysis of the spectral data captured by the hyperspectral imaging camera; Based on the reflectance spectral characteristics of tissues, real-time pixel-level classification and identification of surgical field tissues can be performed to distinguish tissue types including normal brain tissue, tumor tissue, blood vessels and nerves. The recognition results are converted into augmented reality (AR) projection content, and the projection unit is controlled to project different tissues in specific colors or patterns onto the corresponding positions in the surgical field.

[0057] In this embodiment, the data processing unit integrates a second AI module. By analyzing the tissue reflectance spectral characteristics captured by hyperspectral imaging in real time, it performs pixel-level classification and identification of the surgical field, accurately distinguishing different tissue types such as normal brain tissue, tumor tissue, blood vessels, and nerves. The identification results are then converted into augmented reality projection content in real time, and the projection unit is controlled to project different tissues in specific colors or patterns onto the corresponding surgical field locations. This overcomes the physiological limitations of naked-eye observation, achieving real-time visualization of microscopic tissue features. This allows doctors to clearly distinguish tumor boundaries and key structures, significantly improving the integrity of tumor resection and the protection rate of important tissues, and providing real-time, histologically objective evidence for surgical decisions.

[0058] Preferably, the data processing unit also integrates a third AI module for: Continuous analysis of the surgical video stream acquired by the image acquisition unit; The system automatically identifies the current surgical stage based on visual features in the video, including dural incision, cortical fistula, tumor resection, and hemostasis. Based on the identified surgical stage, the projection unit automatically controls the switching and adjustment of the projected information's layer, content, and display mode.

[0059] In this embodiment, the data processing unit integrates a third AI module. By continuously analyzing the visual features in the surgical video stream, it automatically identifies different surgical stages such as dural incision, cortical fistula, tumor resection, and hemostasis. Based on the identification results, it intelligently controls the projection unit to dynamically adjust the layer, content, and display mode of the projected information. This achieves context-adaptive presentation of surgical navigation information. Through AI intelligent perception of the surgical progress, it automatically switches to the most relevant navigation content, significantly reducing the need for doctors to manually operate the interface. This allows doctors to fully focus on the surgical procedure, greatly improving the smoothness of the surgical process and the efficiency of human-computer interaction.

[0060] Preferably, the data processing unit also integrates a fourth AI module for: Access a data pool containing massive amounts of surgical data; Based on real-time surgical data, including instrument location, tissue classification results, and surgical stage, comparative analysis is performed with the data pool. When the current operation is identified as conforming to a preset high-risk pattern, a predictive alarm signal is generated.

[0061] In this embodiment, the data processing unit integrates a fourth AI module. By accessing a massive surgical data pool, it intelligently compares and analyzes real-time surgical data (including instrument location, tissue classification results, and surgical stage) with historical cases in the database. When the current operation is detected to conform to a preset high-risk pattern, a predictive alert is immediately generated. This achieves proactive early warning of surgical risks. Through big data-driven intelligent analysis, it can identify potential risks beyond the doctor's personal experience (such as abnormal blood vessels, proximity of functional areas, etc.), enabling the surgical team to take preventative measures in advance, significantly reducing the probability of intraoperative complications, and providing proactive intelligent protection for surgical safety.

[0062] like Figure 2 As shown, the present invention also provides a neurosurgical navigation method based on light projection, comprising the following steps: S51: Acquire real-time image data of the surgical area through the image acquisition unit; S52: Processing image data through the data processing unit, including: Establish and update a digital twin model of the surgical area in real time, and simulate deformation to correct the projected coordinates in reverse; analyze hyperspectral data in real time to identify and classify tissues; analyze surgical videos to identify surgical stages; perform risk prediction and generate alerts; S53: The corrected navigation content is projected onto the surgical area in real time through the projection unit based on the data processing results.

[0063] In this embodiment, the surgical area data is acquired in real time by the image acquisition unit. The data processing unit then performs multi-dimensional intelligent processing, including digital twin model construction and deformation compensation, tissue spectral analysis, surgical stage identification, and risk prediction. Finally, the projection unit accurately projects the real-time deformation-corrected navigation information onto the corresponding location in the surgical field. The beneficial effect of this technical solution is that it creates an intelligent augmented reality navigation environment that can dynamically adapt to the surgical process. Through multimodal data fusion and AI analysis, it achieves real-time and accurate compensation for intraoperative navigation, visual differentiation of tissue types, and advanced warning of surgical risks, significantly improving the safety, accuracy, and operational efficiency of neurosurgery.

[0064] Example 1: Glioma Resection Surgical scenario: A patient is diagnosed with a high-grade glioma in the left temporal lobe and needs to undergo microscopic tumor resection.

[0065] System configuration and workflow: 1. Preoperative preparation: The patient's preoperative MRI (T1, T2, FLAIR, DTI) and CTA data were uploaded to the system.

[0066] The first AI module runs automatically, accurately segmenting the tumor outline, corticospinal tract (pyramidal tract), and important arteries and veins into a three-dimensional model.

[0067] Based on a safety rule base of "maximum resection and maximum protection of function", the system automatically plans an optimal surgical approach that avoids important blood vessels and pyramidal tracts, and generates projections of the tumor boundary (red) and safety boundary (yellow, extending 2mm outward).

[0068] 2. Intraoperative navigation and execution: Initial registration and digital twin creation: After the patient's head frame is fixed, the system completes the initial spatial registration using reference markers. A binocular vision camera located above the surgical field immediately captures the scene, and the initial three-dimensional surface model (digital twin) of the bone window area is quickly reconstructed using the SGM stereo matching algorithm and triangulation.

[0069] Deformation compensation and real-time projection: After the surgery begins, when tissue displacement occurs due to instrument traction or cerebrospinal fluid loss: The deformation simulation engine runs continuously. It receives tracking data from the instruments and tissue displacement data calculated using optical flow, simulates deformation based on a mass-spring model (which assigns specific viscoelastic parameters to brain tissue), and dynamically updates the digital twin model.

[0070] The system then reverse-maps the pre-planned tumor boundary. For example, if the engine predicts that a piece of tissue has moved inward by 3mm, the system will shift the red boundary originally planned to be projected there 3mm in reverse to compensate, and then use a projector to accurately project the corrected boundary onto the surface of the tissue that has actually moved.

[0071] Real-time tissue identification: When removing deep tumors, it is difficult to distinguish the boundary between the tumor and normal tissue with the naked eye. At this time, the hyperspectral imaging camera comes into play, and the second AI module analyzes the spectral characteristics of each pixel in real time to identify residual tumor tissue. The system immediately controls the projector to project these suspicious areas directly onto the surgical field as a dark red dot pattern, guiding the doctor to make precise removal.

[0072] Intelligent switching of surgical stages: The third AI module analyzes the surgical video. When it recognizes that the doctor is using bipolar electrocoagulation to stop bleeding (the stage changes to "hemostasis"), the system automatically fades the projection of the tumor boundary and instead enhances and projects the outline of nearby small blood vessels to assist the doctor in precise electrocoagulation.

[0073] Predictive alert: When surgical instruments approach a "high-risk area" that is not shown in preoperative imaging but is indicated by the database to have a 30% probability of containing a variant artery, the fourth AI module triggers an alert. The projector projects a flashing red warning circle over the area, and the workstation simultaneously issues a voice prompt, "Warning, there may be a blood vessel ahead," reminding the doctor to explore or avoid the area.

[0074] In this case, the system successfully overcame navigation errors caused by brain tissue deformation, revealed tumor remnants invisible to the naked eye through hyperspectral imaging, and warned of potential risks. Ultimately, the surgery achieved subtotal tumor resection, and the patient recovered smoothly without any postoperative neurological deficits.

[0075] Example 2: Resection of a meningioma at the base of the skull Surgical scenario: A patient has a tuberculum sellae meningioma, with the tumor encasing the optic nerve and internal carotid artery, making the surgery extremely difficult and risky.

[0076] System configuration and workflow: 1. Preoperative preparation: The system processes the patient's high-resolution MRI and CTA data, and the first AI module precisely segments key structures such as the tumor, optic nerve, internal carotid artery, and pituitary stalk.

[0077] The system-planned safe resection boundary automatically narrows near nerves and blood vessels to ensure sufficient safety distance.

[0078] 2. Intraoperative navigation and execution: High-precision registration and visualization: Due to the deep and narrow field of vision in skull base surgery, the initial registration accuracy is required to be extremely high. After establishing the initial digital twin model, the system uses the ICP algorithm (Iterative Closest Point Algorithm) to precisely register it with the preoperatively segmented optic nerve and blood vessel models, ensuring the initial accuracy of the projection.

[0079] Microscopic deformation compensation and vascular projection: When carefully separating a tumor from the optic nerve, even the slightest manipulation can cause tissue displacement.

[0080] The deformation simulation engine runs at extremely high frequencies to simulate these microscopic deformations and corrects the projected coordinates in reverse.

[0081] The projector continuously projects the outline of the internal carotid artery as a bright blue line and the optic nerve as a yellow line onto the surgical field, moving synchronously with the slightest movement of the tissue, as if giving the doctor a pair of "smart glasses" that can "see through" the tissue and annotate key structures in real time.

[0082] Tissue classification and protection of key structures: In the area where the tumor and nerve are most tightly adhered, the second AI module distinguishes between the tumor spectrum and the nerve fiber spectrum using hyperspectral data. The system enhances the projection of the nerve pathway with green dashed lines, greatly assisting doctors in maximizing the protection of optic nerve function during tumor removal.

[0083] Occlusion Handling: When a surgical instrument (such as a suction device) temporarily obscures the blue navigation line projected onto the blood vessel, the visibility assessment module is immediately activated. After detecting the occlusion, the system does not simply turn off the projection, but intelligently projects the blue line onto the side of the suction device's shaft, like a "detour" arrow, continuously reminding the doctor of the location of the obscured blood vessel below.

[0084] Risk prediction: When a doctor operates on a vulnerable branch near the internal carotid artery, the fourth AI module compares historical data and predicts the risk of bleeding. The system immediately marks the point with a bright purple flashing spot and issues an alarm, successfully preventing a possible major hemorrhage.

[0085] In this complex surgery, the real-time, dynamic, and high-precision augmented reality navigation provided by the system was like giving the doctor "X-ray vision" and "early warning capabilities," enabling him to clearly identify and operate precisely in the complex skull base structure. Ultimately, the tumor was completely removed, and all important neurovascular structures were perfectly protected. The patient's postoperative vision was even improved.

[0086] The embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A neurosurgical navigation system based on light projection, characterized in that: include: The image acquisition unit is used to acquire real-time image data of the surgical area; The data processing unit is communicatively connected to the image acquisition unit and is used to process the image data; The projection unit, which is connected in communication with the data processing unit, is used to project navigation information onto the surgical area in the form of light rays.

2. The neurosurgical navigation system based on light projection according to claim 1, characterized in that: The data processing unit is configured as follows: Based on binocular image sequences, a real-time three-dimensional surface model of the surgical area is reconstructed using stereo vision algorithms, serving as the initial digital twin model; It integrates a physics-based deformation simulation engine, which receives real-time instrument tracking data and tissue displacement data as input to simulate soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model. The navigation information based on preoperative image planning will be transformed and reverse-mapped according to the updated digital twin model to generate corrected projection commands to compensate for actual deformation.

3. The neurosurgical navigation system based on light projection according to claim 2, characterized in that: The data processing unit, when reconstructing a real-time 3D surface model of the surgical area based on a stereoscopic vision algorithm using a binocular image sequence as the initial digital twin model, specifically includes: S11: Image acquisition, the left and right views are acquired simultaneously by a binocular vision camera to form a binocular image pair; S12: Camera calibration, based on the internal and external parameters of the binocular camera obtained in advance through the checkerboard calibration method, to establish the imaging model of the camera; S13: Image preprocessing and correction. The original binocular images acquired are distorted using the camera's internal and external parameters, and stereo correction is performed to generate corrected image pairs for stereo matching. S14: Stereo matching, performing pixel matching on the corrected image pairs, calculating the corresponding pixel in the right view for each pixel in the left view using a matching algorithm, and then calculating a disparity map representing the disparity value of the pixel. S15: 3D coordinate calculation. Based on the principle of triangulation, using the disparity map and camera parameters, calculate the 3D coordinates of each pixel in the image in the binocular camera coordinate system. S16: Point cloud generation and surface reconstruction. This process generates a dense 3D point cloud from a set of pixels with 3D coordinates, and then applies filtering and surface reconstruction algorithms to the point cloud to generate a continuous meshed 3D surface model.

4. A neurosurgical navigation system based on light projection according to claim 3, characterized in that: The deformation simulation engine receives real-time instrument tracking data and tissue displacement data as input, simulates soft tissue deformation in the surgical area caused by instrument operation, gravity, and cerebrospinal fluid loss, and dynamically updates the digital twin model. Specifically, this includes: S21: Data input, continuously receiving real-time data streams from the binocular vision camera; S22: Physical modeling, which converts the reconstructed initial digital twin 3D surface model into a mechanical model with physical properties; S23: External force calculation and application. Based on the input real-time data, calculate and apply the external forces acting on the mechanical model. S24: Deformation simulation calculation, based on the applied external force and the constitutive equation of the material, calculates the new equilibrium state of the mechanical model under the action of the new external force by solving the dynamic equation and the static equilibrium equation, and generates a digital twin model to predict the deformation. S25: Model update, update the new vertex coordinates obtained from deformation simulation to the geometric data of the digital twin model, and complete the dynamic update of the digital twin model.

5. A neurosurgical navigation system based on light projection according to claim 4, characterized in that: The data processing unit, when transforming and reversing the navigation information planned based on preoperative images according to the updated digital twin model to generate corrected projection commands and compensate for actual deformation, specifically includes: S31: Establish a mapping relationship. Before any deformation occurs, establish a corresponding mapping relationship between each pre-planned navigation point in the preoperative image coordinate system and the nearest vertex on the surface of the initial digital twin model. S32: Deformation field transfer. After the deformation simulation engine calculates the deformation of the digital twin model from the initial state to the current state, it obtains the displacement vector of each vertex to obtain the overall deformation field of the model. S33: Reverse mapping. For each pre-planned navigation point, find its corresponding initial model vertex according to the mapping relationship and read the displacement of the vertex in the current deformation state. Then, perform reverse calculation, apply the displacement vector in reverse to the pre-planned navigation point, and calculate the theoretical position point of the current deformed field. S34: Coordinate system transformation, transforming the coordinates of the corrected theoretical position points from the preoperative image coordinate system to the projector coordinate system; S35: Projection command generation. The coordinates transformed to the projector coordinate system are converted into pixel coordinates on the projector image plane. Combined with the navigation information, the final digital projection command is generated and sent to the projection unit for execution.

6. A neurosurgical navigation system based on light projection according to claim 5, characterized in that: Before generating the projection command, it also includes determining whether the target projection point is occluded from the current viewpoint. If it is occluded, it is projected onto the edge of the occluder as a warning. Specifically, this includes: S41: Scene depth map construction, based on the current 3D point cloud data of the surgical area, a depth map is generated through a surface reconstruction algorithm; S42: Projection point depth acquisition. For a point to be projected, calculate the depth value in the projector coordinate system. S43: Depth comparison and occlusion judgment: Project the point onto the projector image plane to obtain its corresponding pixel coordinates; read the depth value corresponding to the pixel coordinates in the scene depth map, and make occlusion judgment by comparing the two depth values. S44: Edge alert, identify the outline of the object causing the occlusion on the projector image plane; calculate the set of edge pixels of the outline, and project the original projection content onto the edge area of ​​the occluding object's outline.

7. A neurosurgical navigation system based on light projection according to claim 6, characterized in that: The data processing unit also integrates a first AI module, used for: The preoperative MRI and CT images of patients are automatically segmented to identify key structures, including the three-dimensional contours of tumors, blood vessels, nerves and functional areas. Based on the segmentation results and a pre-defined surgical safety rule base, the system automatically calculates and recommends the optimal surgical approach and safe resection boundaries.

8. A neurosurgical navigation system based on light projection according to claim 7, characterized in that: The data processing unit also integrates a second AI module, used for: Real-time analysis of the spectral data captured by the hyperspectral imaging camera; Based on the reflectance spectral characteristics of tissues, real-time pixel-level classification and identification of surgical field tissues can be performed to distinguish tissue types including normal brain tissue, tumor tissue, blood vessels and nerves. The recognition results are converted into augmented reality (AR) projection content, and the projection unit is controlled to project different tissues in specific colors or patterns onto the corresponding positions in the surgical field.

9. A neurosurgical navigation system based on light projection according to claim 8, characterized in that: The data processing unit also integrates a third AI module, used for: Continuous analysis of the surgical video stream acquired by the image acquisition unit; The system automatically identifies the current surgical stage based on visual features in the video, including dural incision, cortical fistula, tumor resection, and hemostasis. Based on the identified surgical stage, the projection unit automatically controls the switching and adjustment of the projected information's layer, content, and display mode.

10. A neurosurgical navigation method based on light projection, characterized in that: The following steps: S51: Acquire real-time image data of the surgical area through the image acquisition unit; S52: Processing image data through the data processing unit, including: Establish and update a digital twin model of the surgical area in real time, and simulate deformation to correct the projected coordinates in reverse; analyze hyperspectral data in real time to identify and classify tissues; analyze surgical videos to identify surgical stages; perform risk prediction and generate alerts; S53: The corrected navigation content is projected onto the surgical area in real time through the projection unit based on the data processing results.

Images