An intraoperative real-time navigation system and method based on multi-modal image fusion

Abstract

The application provides an intraoperative real-time navigation system and method based on multi-modal image fusion. The system comprises: a preoperative modeling module generating a preoperative multi-modal navigation reference model with tissue deformation parameters; an intraoperative acquisition module acquiring intraoperative surface point cloud, ultrasound section image and pose; a local window module determining a local registration window; a double-constraint registration module generating surface displacement constraint and internal anatomical feature displacement constraint; a deformation compensation module solving a local deformation compensation field according to the two types of constraints and tissue deformation parameters and updating the model; a reliability evaluation module generating fusion reliability by comprehensively considering local registration residual, feature quantity, point cloud integrity, ultrasound clarity, occlusion ratio, constraint consistency and deformation amplitude; and a display control module outputting normal, degraded, hidden or re-registration instructions according to the reliability. The application improves navigation accuracy and reliability through double-constraint local deformation compensation and reliability-controlled display.

Description

Technical Field

[0001] This invention relates to the fields of medical image processing, intraoperative navigation, and image registration, and particularly to an intraoperative real-time navigation system and method based on multimodal image fusion. Background Technology

[0002] In surgeries such as neurosurgery, hepatobiliary surgery, orthopedics, urology, and tumor resection, surgeons typically need to create a three-dimensional model of the surgical area based on preoperative CT, MRI, CTA, MRA, PET-CT, or other medical images. During the operation, this model is used for navigation and visualization of lesions, blood vessels, nerves, bony structures, or soft tissue boundaries, taking into account the position of surgical instruments. Preoperative images are characterized by intact tissue structures, high spatial resolution, and clear anatomical relationships. However, because they are acquired before the surgical procedure, they cannot directly reflect tissue deformation caused by intraoperative tissue traction, resection, air inflation, bleeding, changes in body position, or instrument contact.

[0003] Current intraoperative navigation systems mostly employ rigid registration between preoperative images and the patient's surgical space, or registration based on a few landmarks, overlaying the preoperative model onto the surgical field or display terminal. This method can provide some navigational reference in scenarios with prominent bony structures and minimal soft tissue deformation. However, in soft tissue organs, near tumor boundaries, in densely vascularized areas, or in areas where surgical resection is progressing rapidly, discrepancies easily arise between the preoperative model and the actual intraoperative tissue location. If the lesion boundary, vascular pathways, or safety boundaries are still displayed according to the uncorrected preoperative model, inconsistencies between the navigation layer and the actual anatomical structure may occur.

[0004] Some approaches incorporate intraoperative surface point clouds, endoscopic images, or intraoperative ultrasound images for intraoperative correction. Intraoperative surface point clouds can reflect the geometric changes of exposed tissue surfaces, but they can only describe the surface morphology and are insufficient to constrain the boundaries of deep lesions, deep blood vessels, or soft tissues. Intraoperative ultrasound cross-sectional images can display some internal anatomical structures, but their imaging field of view is limited, image noise is high, and the cross-sectional position changes with probe posture. Relying solely on intraoperative ultrasound for overall model updates is easily affected by local image quality and feature matching.

[0005] Furthermore, intraoperative data acquisition is easily affected by factors such as instrument obstruction, blood coverage, tissue reflection, ultrasound probe compression, missing surface point clouds, and patient reference pose drift. If the navigation system continues to display the complete navigation layer even when registration quality deteriorates, it is difficult for physicians to promptly determine the reliability of the current navigation results. Existing technologies lack sufficient reliability evaluation of intraoperative fusion results, typically lacking a comprehensive assessment of local registration residuals, the number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument obstruction ratio, constraint consistency, and deformation compensation magnitude. They also lack control mechanisms to adjust the navigation display status based on reliability, such as normal display, degraded display, partial hiding, or re-registration triggering.

[0006] Therefore, there is a need for an intraoperative real-time navigation system and method that can simultaneously utilize intraoperative surface point clouds and intraoperative ultrasound cross-sectional images to perform local deformation compensation on the preoperative multimodal navigation reference model and control the display status of the navigation layer based on the fusion reliability. Summary of the Invention

[0007] In view of this, embodiments of the present invention provide an intraoperative real-time navigation system and method based on multimodal image fusion to solve the technical problems existing in the prior art.

[0008] The technical solution of this invention is implemented as follows: A real-time intraoperative navigation system and method based on multimodal image fusion, comprising a processor, a memory, an intraoperative data acquisition interface, an instrument positioning interface, and a display terminal, wherein the processor calls a program in the memory to form: The preoperative modeling module is used to register, segment, and reconstruct three-dimensional images of at least two preoperative medical images of the same surgical area to generate a preoperative multimodal navigation reference model with tissue deformation parameters. The intraoperative acquisition module is used to acquire intraoperative surface point clouds, intraoperative ultrasound cross-sectional images, patient reference pose, and instrument end-effector pose. The initial registration module is used to establish the initial transformation relationship between the preoperative image coordinate system and the patient's surgical space coordinate system; The local window module is used to determine the local registration window based on the instrument end pose, the coverage of the ultrasonic section, and the effective field of view of the surface point cloud. The dual-constraint registration module is used to match the intraoperative surface point cloud with the corresponding surface of the preoperative model within the local registration window to generate surface displacement constraints, and to match the internal anatomical features in the intraoperative ultrasound section with the corresponding internal structure of the preoperative model to generate internal anatomical feature displacement constraints. The deformation compensation module is used to solve the local deformation compensation field and update the corresponding model region based on two types of displacement constraints and tissue deformation parameters. The credibility evaluation module generates fusion credibility based on local registration residuals, number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, constraint consistency, and deformation compensation amplitude. The display control module is used to output normal display instructions, downgraded display instructions, partial hiding instructions, or partial re-registration trigger instructions according to the fusion confidence level.

[0009] Compared with the prior art, the present invention has the following advantages: This invention unifies at least two types of preoperative medical images into the same preoperative image coordinate system during the preoperative stage, and segments and models lesions, blood vessels, nerves, bony structures and soft tissues, so that the preoperative multimodal navigation reference model simultaneously includes surface structure, internal anatomical structure and tissue deformation parameters, providing a data basis for intraoperative local registration and deformation compensation.

[0010] This invention simultaneously acquires surface point clouds, intraoperative ultrasound cross-sectional images, patient reference pose, and instrument end-effector pose during surgery. Based on the instrument end-effector pose, ultrasound cross-sectional coverage, and effective field of view of the surface point cloud, a local registration window is determined, which concentrates the registration calculation on the area relevant to the current surgical operation and reduces the computational burden caused by overall re-registration of unrelated areas.

[0011] This invention generates both surface displacement constraints and internal anatomical feature displacement constraints within a local registration window. Surface displacement constraints reflect the offset between the exposed tissue surface and the preoperative model surface, while internal anatomical feature displacement constraints reflect the offset between the deep structures within the ultrasound section and the internal structures of the preoperative model. Both types of constraints participate in solving the local deformation compensation field, enabling the updated preoperative multimodal navigation reference model to take into account both surface morphology changes and internal structural changes.

[0012] This invention comprehensively evaluates factors such as local registration residuals, number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, constraint consistency, local deformation compensation magnitude, and time interval since the last effective registration to generate fusion reliability. Based on the fusion reliability, the invention controls the normal display, downgraded display, partial hiding, or partial re-registration of the navigation layer, ensuring that the navigation content output by the display terminal corresponds to the reliability of the current fusion result. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the module connection relationship of an intraoperative real-time navigation system based on multimodal image fusion according to the present invention; Figure 2 This is a flowchart illustrating the overall processing flow of a real-time intraoperative navigation method based on multimodal image fusion according to the present invention. Figure 3 This is a schematic diagram of the process for determining the local registration window and dual-constraint registration in this invention; Figure 4 This is a schematic diagram of the process for local deformation compensation and fusion reliability evaluation in this invention. Detailed Implementation

[0015] The present invention will be further described below through several exemplary embodiments. Those skilled in the art can make various modifications or substitutions to the following embodiments without departing from the technical principles and core concepts of the present invention, and the modified solutions will still fall within the protection scope of the present invention. The accompanying drawings and descriptions of the embodiments in this specification are for illustrative purposes only and are not intended to limit the protection scope of the present invention.

[0016] In this invention, a module or unit can be implemented by a processor executing program instructions in a memory, or by hardware circuitry, firmware, or a combination thereof; data interaction between modules can be accomplished through an internal device bus, data interface, shared memory area, or network communication.

[0017] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings. Example 1

[0018] See Figure 1 This embodiment provides an intraoperative real-time navigation system based on multimodal image fusion, including a processor, a memory, an intraoperative data acquisition interface, an instrument positioning interface, and a display terminal. The processor calls programs in the memory to form a preoperative modeling module, an intraoperative acquisition module, an initial registration module, a local window module, a dual-constraint registration module, a deformation compensation module, a reliability evaluation module, and a display control module.

[0019] The data processing in this embodiment is as follows: The preoperative modeling module receives at least two preoperative medical images of the same surgical area and outputs a preoperative multimodal navigation reference model with tissue deformation parameters. The intraoperative acquisition module receives data from the intraoperative data acquisition interface and the instrument positioning interface during the operation and outputs intraoperative surface point cloud, intraoperative ultrasound section image, patient reference pose, and instrument end-effector pose. The initial registration module outputs the initial transformation relationship from the preoperative image coordinate system to the patient's surgical space coordinate system based on the preoperative multimodal navigation reference model, the patient reference pose, and the initial intraoperative acquisition data.

[0020] The local window module receives the instrument end-effector pose, intraoperative ultrasound section coverage, effective field of view of the surface point cloud, and key structural positions in the preoperative multimodal navigation reference model, and outputs a local registration window. The dual-constraint registration module outputs surface displacement constraints, internal anatomical feature displacement constraints, constraint consistency results, and comprehensive local registration residuals within the local registration window. The deformation compensation module outputs a local deformation compensation field based on the two types of displacement constraints, constraint consistency results, and tissue deformation parameters, and updates the corresponding local region of the preoperative multimodal navigation reference model. The reliability evaluation module outputs fusion reliability based on the comprehensive local registration residuals, the number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, constraint consistency, local deformation compensation amplitude, and the time interval since the last effective registration. The display control module outputs normal display instructions, degraded display instructions, local hiding instructions, or local re-registration trigger instructions to the display terminal based on the fusion reliability. Example 2

[0021] See Figure 1 The preoperative modeling module includes an image preprocessing unit, a cross-modal registration unit, a structural segmentation unit, a 3D model construction unit, and a tissue attribute annotation unit.

[0022] The image preprocessing unit receives at least two types of preoperative medical images. Taking MRI and CT images as examples, MRI images provide information on the boundaries of soft tissues, lesions, and perivascular soft tissues, while CT images provide information on bony structures or high-density structures. The image preprocessing unit performs denoising, grayscale normalization, voxel resampling, and artifact suppression on each type of preoperative medical image to obtain preprocessed image data. Voxel resampling ensures uniform voxel spacing across different preoperative medical images, and grayscale normalization ensures comparable intensity ranges for subsequent registration and segmentation.

[0023] The cross-modal registration unit uses one type of preoperative medical image as a reference image to register the remaining preoperative medical images to the same preoperative image coordinate system. The unit first performs rigid registration based on anatomical landmarks or image centers, then optimizes the registration results based on mutual information, normalized cross-correlation, structural boundary overlap, or feature point distance, outputting a transformation matrix from each preoperative medical image to the preoperative image coordinate system. After cross-modal registration, lesion regions, vascular regions, neural regions, bony structures, and soft tissue regions in different images have a unified coordinate reference.

[0024] The structural segmentation unit receives preoperative medical images after cross-modal registration and segments lesion regions, vascular regions, neural regions, bony structures, and soft tissue regions. Structural segmentation can employ thresholding, region growing, graph cut, active contouring, deep learning segmentation networks, or manual correction. The structural segmentation unit outputs a 3D mask for each tissue region, where each voxel has a corresponding tissue type label.

[0025] The 3D model construction unit extracts surface meshes, centerlines, boundary contours, or volumetric data models from the 3D masks of each tissue region to generate a preoperative multimodal navigation reference model. This preoperative multimodal navigation reference model includes a surface model, an internal structure model, a set of key points, and tissue region labels. The surface model is used for matching with the intraoperative surface point cloud; the internal structure model is used for matching with internal anatomical features in the intraoperative ultrasound section; the set of key points is used for registration, reliability evaluation, and display control; and the tissue region labels are used to read corresponding tissue deformation parameters.

[0026] The tissue attribute annotation unit is configured with tissue deformation parameters according to the tissue region type. These parameters include elasticity coefficient, Poisson's ratio, deformation weight, maximum allowable deformation, deformation smoothing constraint weight, or boundary constraint weight between tissue regions. Lower deformation weights are assigned to bony structures, higher deformation weights to soft tissue regions, and deformation protection weights to vascular and neural regions. This ensures that the subsequent local deformation compensation field reflects the different responses of different tissues to external forces and tensions when updating the model.

[0027] When configuring tissue deformation parameters, the tissue attribute annotation unit first reads the tissue parameter table corresponding to the surgical site. The tissue parameter table stores parameter items according to tissue region type, including tissue region type, elasticity coefficient, Poisson's ratio, deformation weight, maximum allowable deformation, deformation smoothing constraint weight, and boundary constraint weight. The tissue parameter table is stored in memory, and the initial parameters in the tissue parameter table are formed by at least one of the following: system preset parameters, surgical site template parameters, historical registration data statistical parameters, or manually confirmed parameters.

[0028] The tissue attribute annotation unit determines the tissue region type of each model unit based on the 3D mask output by the structure segmentation unit, and reads the corresponding tissue deformation parameters from the tissue parameter table using the tissue region type as an index. When the same model unit is adjacent to two or more tissue regions, the tissue attribute annotation unit weights the tissue deformation parameters corresponding to each tissue region according to the voxel ratio or surface adjacency area ratio of the adjacent tissue regions to obtain the composite tissue deformation parameters of the model unit.

[0029] The tissue deformation parameters are written into the tissue attribute field of the preoperative multimodal navigation reference model and a corresponding relationship is established with surface mesh vertices, internal structure points, voxel nodes, or mesh control points. When the deformation compensation module extracts local model elements within the local registration window, it reads the corresponding tissue deformation parameters based on the tissue attribute field of the local model elements.

[0030] In the solution process of the local deformation compensation field, the elastic coefficient and Poisson's ratio are used for material parameter configuration in the finite element model; the deformation weight is used to adjust the degree of displacement response of the corresponding tissue region under surface displacement constraints and internal anatomical feature displacement constraints; the maximum allowable deformation is used to limit the displacement amplitude of local model elements; the deformation smoothing constraint weight is used to constrain the displacement continuity between adjacent model elements; and the boundary constraint weight is used to constrain the relative displacement at the interface of different tissue regions. The deformation weight of bony structure regions is less than that of soft tissue regions, and the maximum allowable deformation of vascular and nerve regions is less than that of the surrounding soft tissue regions.

[0031] The preoperative modeling module ultimately outputs a preoperative multimodal navigation baseline model, which includes at least: a preoperative image coordinate system, surface structure, internal anatomical structure, tissue region labels, key structural locations, and tissue deformation parameters. Example 3

[0032] See Figure 1 The intraoperative acquisition module includes a surface point cloud acquisition unit, an intraoperative ultrasound acquisition unit, a patient reference pose acquisition unit, an instrument pose acquisition unit, and a time synchronization unit.

[0033] The surface point cloud acquisition unit acquires point clouds of exposed tissue surfaces using structured light scanning equipment, depth cameras, or binocular endoscopic 3D reconstruction equipment. The surface point cloud comprises multiple 3D points, each with spatial coordinates; some points also possess color, normal vector, or confidence attributes. The surface point cloud acquisition unit performs outlier removal, sampling density equalization, and field-of-view boundary extraction on the acquisition results, outputting the intraoperative surface point cloud and the effective field of view of the surface point cloud.

[0034] The intraoperative ultrasound acquisition unit acquires intraoperative ultrasound slice images using an ultrasound probe with positioning markers. These images consist of two-dimensional grayscale images and their corresponding probe poses. Based on the ultrasound probe calibration results, the intraoperative ultrasound acquisition unit maps the pixel coordinates in the ultrasound slice to the patient's surgical space coordinate system, obtaining the coverage area of ​​the intraoperative ultrasound slice. The intraoperative ultrasound acquisition unit can also perform speckle noise suppression, grayscale normalization, and edge enhancement processing on the ultrasound slice images, making it easier to extract vascular bifurcations, lesion boundaries, or fascial interfaces.

[0035] The patient reference pose acquisition unit acquires the spatial position and orientation of the patient reference frame, bony reference points, or surgical area reference markers to form the patient reference pose. The patient reference pose is used to correct for the impact of changes in patient position or minor drifts of the reference frame on intraoperative navigation coordinates.

[0036] The instrument pose acquisition unit acquires the position and orientation of the surgical instrument tip via the instrument positioning interface, forming the instrument tip pose. The instrument tip pose includes the position coordinates and orientation vector of the instrument tip in the patient's surgical space coordinate system, and may also include the instrument type, instrument length, instrument working end radius, or instrument safety boundary.

[0037] The time synchronization unit assigns a unified timestamp to the intraoperative surface point cloud, intraoperative ultrasound section image, patient reference pose, and instrument end-effector pose. When performing local registration, the processor selects the surface point cloud, ultrasound section image, patient reference pose, and instrument end-effector pose from the same timestamp or the same time window, so that the data participating in the same local registration cycle correspond to the same intraoperative state. Example 4

[0038] See Figure 1 and Figure 2 The initial registration module is used to establish the initial transformation relationship between the preoperative image coordinate system and the patient's surgical space coordinate system.

[0039] The initial registration module first acquires calibration data. For surface point cloud acquisition devices, the initial registration module obtains the transformation matrix from the surface point cloud acquisition device coordinate system to the patient's surgical space coordinate system through calibration plate calibration or hand-eye calibration. For intraoperative ultrasound probes, the initial registration module obtains the transformation relationship between the ultrasound image coordinate system, the ultrasound probe coordinate system, and the patient's surgical space coordinate system through ultrasound probe calibration. For instrument positioning devices, the initial registration module obtains the transformation matrix from the instrument coordinate system to the patient's surgical space coordinate system through instrument geometric calibration. For display terminals, the initial registration module obtains the transformation matrix from the patient's surgical space coordinate system to the display coordinate system through display calibration.

[0040] The initial registration module then extracts preoperative registerable structures from the preoperative multimodal navigation reference model. These preoperative registerable structures include surface structures, bony landmarks, vascular bifurcation points, or anatomical boundaries. The initial registration module also extracts corresponding intraoperative registerable structures from the surface point cloud initially acquired during surgery or the patient's reference pose. Based on the matching relationship between the preoperative and intraoperative registerable structures, the initial registration module calculates the initial transformation relationship from the preoperative image coordinate system to the patient's surgical space coordinate system.

[0041] Let the point in the preoperative image coordinate system be... The corresponding point in the patient's surgical space coordinate system is The initial transformation relationship is expressed as:

[0042] in, This is the initial transformation matrix from the preoperative image coordinate system to the patient's surgical space coordinate system. The initial registration module sends this initial transformation matrix to the local window module, the dual-constraint registration module, the deformation compensation module, and the display control module. Example 5

[0043] See Figure 2 and Figure 3 The local window module is used to determine the local registration window. The local registration window is used to limit the processing area for this surface point cloud matching, internal anatomical feature matching, local deformation compensation calculation, and fusion credibility evaluation.

[0044] The local window module receives the instrument tip pose, the coverage area of ​​the intraoperative ultrasound section, the effective field of view of the surface point cloud, and the positions of key structures in the preoperative multimodal navigation reference model. The local window module first determines the window center. When the instrument tip pose is effective and the instrument tip is close to the current operating area in the preoperative planned path, the instrument tip position is used as the window center; when the intraoperative ultrasound section covers the lesion boundary, vascular bifurcation, or fascial layer interface, the center of the intraoperative ultrasound section is used as the window center; when the effective field of view of the surface point cloud covers the current operating area and the ultrasound section is temporarily unavailable, the center of the effective field of view of the surface point cloud is used as the window center.

[0045] The current operation stage is determined by at least one of the following: the preoperative planning path file, the matching result between the intraoperative instrument tip position and the preoperative planning path, the stage marker input on the display terminal, or the operation node recorded by the surgical navigation system. The local window module projects the instrument tip position onto the preoperative planning path, obtains the minimum distance between the instrument tip position and each path point on the preoperative planning path, and determines the stage to which the path point corresponding to the minimum distance belongs as the current operation stage. The current operation stage includes the surface exposure stage, lesion approach stage, critical structure proximity stage, and deep operation stage.

[0046] After determining the window center in the local window module, the window radius of the local registration window is determined based on the base radius, the intraoperative ultrasound section coverage width, the effective field of view boundary of the surface point cloud, the distance from the instrument tip to the critical structure, and the current operational stage coefficient. Let the base radius be... The intraoperative ultrasound cut width is The equivalent radius of the effective field of view of the surface point cloud is The distance from the end of the instrument to the nearest critical structure is The coefficient for the current operation stage is Then the window radius of the local registration window Determined according to the following rules:

[0047] in, Minimum window radius, The maximum window radius, Distance is affected by critical structural factors. Indicates when Time to take ,when Time to take , , , Preset weights. When the current operation stage is the surface exposure stage, Take the smaller value; when the current operation stage is the lesion approach stage or the critical structure proximity stage, Take the larger value; when the current operation stage is the deep operation stage, increase the weight corresponding to the coverage width of the intraoperative ultrasound section.

[0048] When the local registration window is adjacent to at least one critical structure in a lesion, blood vessel, or nerve, the local window module generates a directional expansion amount along the direction from the window center towards the critical structure, causing the boundary of the local registration window in the direction of the critical structure to expand outward. The directional expansion amount is determined based on the distance from the instrument tip to the critical structure, the type of critical structure, and the current stage of operation; among them, the directional expansion amount corresponding to nerve regions and blood vessel regions is greater than the directional expansion amount corresponding to ordinary soft tissue regions.

[0049] The local window module will constrain the spatial range obtained by expanding the window radius and direction with the effective field of view of the surface point cloud and the coverage of the intraoperative ultrasound section. When the number of surface point clouds in the intersection area is lower than the point cloud number threshold or the number of internal anatomical features is lower than the feature number threshold, the local window module will expand the window radius to not exceed the maximum window radius; if the threshold condition is still not met after expansion, the local window module will send a local data insufficiency flag to the credibility evaluation module.

[0050] The local registration window module determines the window radius or window boundary based on the current operation stage and the location of critical structures. If the current operation stage is close to the lesion, blood vessel, or nerve, the coverage of the local registration window in the direction of the critical structure is expanded; if the current operation stage is in the surface incision or exposure stage, the local registration window is mainly based on the effective field of view of the surface point cloud; if the current operation stage is in the deep operation stage, the local registration window is mainly based on the coverage of the intraoperative ultrasound section and the position of the instrument tip.

[0051] The local window module outputs a local registration window. Local registration window This includes the window center, window radius, window boundaries, preoperative model unit index within the window, surface point cloud index within the window, and ultrasound section coverage area within the window. Local registration window. It is sent to the dual-constraint registration module, deformation compensation module, and credibility evaluation module. Example 6

[0052] See Figure 3 The dual-constraint registration module includes a surface point cloud constraint unit, an internal ultrasonic feature constraint unit, a constraint consistency judgment unit, and a local registration residual calculation unit.

[0053] Surface point cloud constraint unit receives local registration window Intraoperative surface point cloud and preoperative multimodal navigation baseline model. Surface point cloud constraint elements are located within the local registration window of the preoperative multimodal navigation baseline model. The corresponding surface regions within the intraoperative area are mapped to the patient's surgical space coordinate system according to the initial transformation relationship, resulting in a preoperative surface prediction point set. The surface point cloud constraint unit matches the intraoperative surface point cloud with the preoperative surface prediction point set. The matching methods include nearest point matching, normal vector constraint matching, boundary curve matching, or point-to-surface distance matching based on a robust loss function.

[0054] Before generating surface displacement constraints, the surface point cloud constraint unit performs effective point screening on the intraoperative surface point cloud. Effective point screening includes removing points with confidence scores below the point cloud confidence threshold, removing points located within the instrument occlusion area, removing isolated points whose distance from neighboring points exceeds the outlier threshold, and downsampling the remaining points to ensure that the distance between adjacent sampled points is not less than the preset sampling interval.

[0055] The surface point cloud constraint unit will set the predicted points in the preoperative surface prediction point set. Candidate points in intraoperative surface point cloud When establishing a matching relationship, both distance and normal conditions are considered. The distance condition is... and The Euclidean distance between them is less than the surface distance threshold. Normal condition is Surface normal vector and The angle between the surface normal vectors is less than the normal angle threshold. When the matching relationship originates from a boundary curve, it is also determined whether the tangential angle of the boundary curve is less than the boundary tangential threshold.

[0056] For a pair of matching points that satisfy both the distance and normal conditions, the surface point cloud constraint element calculates the surface displacement vector:

[0057] in, For the first Surface displacement vectors of matching points on the surface These are the preoperative surface prediction points mapped according to the initial transformation relationship. To and Matched intraoperative surface points.

[0058] The surface point cloud constraint unit generates surface matching weights based on the matching distance, normal angle, point cloud confidence, and local point cloud density. The smaller the matching distance, the smaller the normal angle, the higher the point cloud confidence, and the more uniform the local point cloud density, the greater the surface matching weight. When the matching distance exceeds the surface distance threshold, the normal angle exceeds the normal angle threshold, or the point cloud confidence is lower than the point cloud confidence threshold, the corresponding matching point pair does not participate in the surface displacement constraint set.

[0059] The surface point cloud constraint element employs a robust loss function to suppress outliers in surface matching point pairs. For matching point pairs whose residuals exceed the outlier residual threshold, their surface matching weights are reduced or they are removed from the surface displacement constraint set. After outlier removal, the surface displacement constraint set includes the surface matching point positions, surface displacement vectors, and surface matching weights.

[0060] For preoperative surface prediction points Matching points in the intraoperative surface point cloud Surface point cloud constraint elements generate surface displacement constraints:

[0061] in, Indicates the first The surface displacement vectors of the matching points on the surface. Multiple surface displacement vectors form a set of surface displacement constraints. .

[0062] The internal ultrasound feature constraint unit receives intraoperative ultrasound section images, ultrasound section pose, and a preoperative multimodal navigation reference model. The internal ultrasound feature constraint unit extracts internal anatomical features from the intraoperative ultrasound section images, including vascular bifurcation, lesion boundaries, or fascial interfaces. Based on the ultrasound probe calibration results and ultrasound section pose, the internal ultrasound feature constraint unit maps the internal anatomical features in the ultrasound section to the patient's surgical space coordinate system, obtaining intraoperative internal feature points or feature lines. The internal ultrasound feature constraint unit locates the corresponding internal structures in the preoperative multimodal navigation reference model and maps them to the patient's surgical space coordinate system according to the initial transformation relationship, obtaining the predicted preoperative locations of the internal structures.

[0063] When the internal ultrasound feature constraint unit extracts features from the intraoperative ultrasound section image, it generates a feature type, feature location, feature direction, and feature confidence level for each internal anatomical feature. The feature confidence level is determined based on the local contrast of the ultrasound image, edge continuity, speckle noise intensity, feature region area, and feature tracking stability. The feature location of a blood vessel bifurcation is the bifurcation center point, the feature location of a lesion boundary is the boundary sampling point or boundary curve, and the feature location of the fascial layer interface is the interface sampling line.

[0064] After mapping intraoperative internal anatomical features to the patient's surgical space coordinate system, the internal ultrasound feature constraint unit searches for candidate internal structures with the same feature type in the preoperative multimodal navigation baseline model. These candidate internal structures are then mapped to the patient's surgical space coordinate system through an initial transformation relationship and matched with the intraoperative internal anatomical features. During matching, consistency of feature type, spatial distance, directional angle, and local topological relationships are simultaneously assessed. Specifically, for vessel bifurcation, the number of branches or branch angles are also assessed; for lesion boundaries, the boundary curvature or boundary normal is assessed; and for fascial interfaces, the continuity direction of the interface is assessed.

[0065] When the distance between the intraoperative internal anatomical features and the preoperative predicted location of the internal structure is less than the internal feature distance threshold. The directional angle is less than the internal feature angle threshold. Furthermore, the feature confidence level is higher than the ultrasound feature confidence level threshold. At that time, the internal ultrasound feature constraint unit confirms successful matching and generates an internal anatomical feature displacement vector:

[0066] in, For the first Displacement vectors of internal anatomical features This refers to the intraoperative internal anatomical features mapped to the patient's surgical space coordinate system. Preoperative internal structure prediction location mapped to the patient's surgical space coordinate system.

[0067] The internal ultrasonic feature constraint unit generates internal feature matching weights based on ultrasonic feature confidence, matching distance, directional angle, and local topological consistency. When the confidence level of an ultrasound feature is lower than the confidence threshold of an ultrasound feature. The matching distance is greater than the internal feature distance threshold. When local topological relationships are inconsistent, the internal ultrasound feature constraint unit will mark the corresponding matching relationship as an invalid match. Invalid matches are not included in the internal anatomical feature displacement constraint set, but their number is sent to the credibility evaluation module as a criterion for judging that there are not enough matchable features.

[0068] The internal ultrasound feature constraint unit performs out-of-point culling on the internal anatomical feature displacement constraint set. If the direction difference or amplitude difference between a certain internal anatomical feature displacement vector and other internal anatomical feature displacement vectors within the same local registration window exceeds the internal consistency threshold... If so, reduce the weight of the internal anatomical feature displacement constraint, or remove it from the set of internal anatomical feature displacement constraints.

[0069] For intraoperative internal feature points Preoperative internal structure prediction points The internal ultrasound feature constraint unit generates internal anatomical feature displacement constraints:

[0070] in, Indicates the first The displacement vectors of each internal anatomical feature. Multiple internal anatomical feature displacement vectors form a set of internal anatomical feature displacement constraints. .

[0071] The constraint consistency judgment unit receives the surface displacement constraint set. and internal anatomical feature displacement constraint set Calculate the average direction, average amplitude, and local distribution of the two types of constraints respectively. For the average vector of surface displacement constraints... and internal anatomical feature displacement constraint average vector The constraint consistency judgment unit calculates the direction difference and amplitude difference:

[0072]

[0073] When the direction difference or amplitude difference When the consistency threshold is exceeded, the constraint consistency judgment unit generates a constraint conflict flag and sends the conflict constraint location, conflict direction and conflict magnitude to the deformation compensation module and the credibility evaluation module.

[0074] The local registration residual calculation unit calculates the comprehensive local registration residual based on the surface point cloud residual, internal feature residual, and constraint consistency results. The surface point cloud residual is calculated from the point-to-surface or point-to-point distance between the intraoperative surface point cloud and the preoperative predicted surface point set; the internal feature residual is calculated from the distance between the intraoperative ultrasound internal anatomical features and the corresponding preoperative internal structures; the constraint consistency results are used to increase the residual weight in conflict areas. The comprehensive local registration residual is then sent to the credibility evaluation module. Example 7

[0075] See Figure 4The deformation compensation module receives the local registration window. Surface displacement constraint set Internal anatomical feature displacement constraint set Constraint consistency results and organizational deformation parameters.

[0076] The deformation compensation module first extracts the locally registered windows from the preoperative multimodal navigation reference model. The local model unit consists of surface mesh vertices, internal structure points, voxel nodes, or mesh control points. The deformation compensation module reads the tissue deformation parameters based on the tissue region type to which each local model unit belongs. For bony structures, the deformation compensation module reduces their displacement degrees of freedom; for soft tissue regions, it allows continuous deformation based on constraints; for vascular and nerve regions, it sets protective constraints to prevent unreasonable stretching or folding of critical structures by the deformation compensation field.

[0077] In one implementation, the deformation compensation module uses a radial basis function interpolation model to solve the local deformation compensation field. For any point within the local model element... The local deformation compensation field is represented as:

[0078] in, Point Deformation displacement, Indicates the constraint control point. Indicates the control point weights. This represents the radial basis function. Surface displacement constraints and internal anatomical feature displacement constraints are used together as control point constraint inputs.

[0079] In another implementation, the deformation compensation module uses a mesh control point deformation model to register the local registration window. The model is divided into several control grids. The displacement of the control grid nodes is solved based on the surface displacement constraints and the internal anatomical feature displacement constraints. Then, the deformation displacement of the local model elements is obtained by interpolation.

[0080] In another implementation, the deformation compensation module uses a finite element model, sets material parameters according to the type of tissue region, and uses surface displacement constraints and internal anatomical feature displacement constraints as boundary conditions or internal constraints to solve the local deformation compensation field.

[0081] When the constraint consistency judgment unit outputs a constraint conflict flag, the deformation compensation module reduces the calculation weight of the conflict constraint. If the confidence level of the surface displacement constraint is lower than that of the internal anatomical feature displacement constraint, the weight of the surface displacement constraint is reduced; if the intraoperative ultrasound image clarity is insufficient or the number of internal features is insufficient, the weight of the internal anatomical feature displacement constraint is reduced. The deformation compensation module then re-solves the local deformation compensation field based on the adjusted constraint weights.

[0082] The deformation compensation module will compensate for the local deformation field. This is applied to the corresponding local region of the preoperative multimodal navigation baseline model. For points within the local model unit... The updated location is:

[0083] The updated preoperative multimodal navigation baseline model retains the original tissue region labels and key structural attributes, and marks the updated region as the effective local region of the current registration cycle. Example 8

[0084] See Figure 4 The credibility evaluation module receives the comprehensive local registration residual, the number of matchable features, the integrity of the surface point cloud, the clarity of the ultrasound image, the instrument occlusion ratio, the constraint consistency, the local deformation compensation amplitude, and the time interval since the last effective registration.

[0085] The credibility evaluation module first normalizes each indicator. The smaller the comprehensive local registration residual, the higher the corresponding residual score; the more matching features, the higher the corresponding feature score; the higher the surface point cloud integrity, the higher the corresponding point cloud score; the higher the ultrasound image clarity, the higher the corresponding ultrasound score; the lower the instrument occlusion ratio, the higher the corresponding occlusion score; the higher the consistency between surface displacement constraints and internal anatomical feature displacement constraints, the higher the corresponding consistency score; when the local deformation compensation amplitude is within a preset reasonable range, the corresponding deformation score is higher; the shorter the time interval since the last effective registration, the higher the corresponding time score.

[0086] The credibility evaluation module normalizes each indicator according to a value range of 0 to 1. 0 indicates that the corresponding indicator does not meet the credibility display requirements, and 1 indicates that the corresponding indicator meets the credibility display requirements.

[0087] Let the comprehensive local registration residual be... The residual allowable threshold is residual score Generate as follows:

[0088] Let the number of matchable features be The threshold for the number of features is Feature scoring Generate as follows:

[0089] Let the number of effective surface point clouds within the local registration window be . The theoretical or historical effective field of view has a reference number of point cloud points. The surface point cloud completeness is Point cloud rating Generate as follows:

[0090] Let the ultrasound image clarity be... The ultrasonic resolution threshold is Ultrasound scoring Generate as follows:

[0091] Among them, ultrasound image clarity It is calculated based on the edge gradient, local contrast, and speckle noise intensity of the intraoperative ultrasound cross-sectional image.

[0092] Let the instrument obstruction ratio be . The occlusion ratio threshold is occlusion rating Generate as follows:

[0093] Let the directional difference between the average direction of surface displacement constraints and the average direction of internal anatomical feature displacement constraints be denoted as . The directional difference threshold is The difference between the average amplitude of surface displacement constraints and the average amplitude of internal anatomical feature displacement constraints is _____. The amplitude difference threshold is Consistency score Generate as follows:

[0094] Let the local deformation compensation range be The upper limit of reasonable deformation range is Deformation score Generate as follows:

[0095] Let the time interval since the last effective registration be . The effective registration time threshold is Time rating Generate as follows:

[0096] The credibility evaluation module generates a fused credibility score based on each normalized score. :

[0097] in, , , , , , , , These are the weights corresponding to the residual score, feature score, point cloud score, ultrasonic score, occlusion score, consistency score, deformation score, and temporal score, respectively, and:

[0098] The weights are configured based on the surgical site, tissue type, equipment configuration, or historical registration data. When the current local registration window is located near the boundary of a blood vessel, nerve, or lesion, the weights corresponding to the residual score, consistency score, and deformation score are increased; when the intraoperative ultrasound section covers key internal structures, the weights corresponding to the ultrasound score and feature score are increased; when the effective field of view of the surface point cloud is sufficiently covered and the ultrasound section is temporarily unavailable, the weights corresponding to the point cloud score are increased.

[0099] When the difference in direction or amplitude between surface displacement constraints and internal anatomical feature displacement constraints exceeds the consistency threshold, the credibility evaluation module reduces the consistency score. And increase the weight of the residual score. When the surface point cloud completeness is below the point cloud completeness threshold, the point cloud score is reduced. When the ultrasound image clarity is below the ultrasound clarity threshold, the ultrasound score should be reduced. When the device occlusion ratio exceeds the occlusion ratio threshold, the occlusion score is reduced. The credibility evaluation module sends indicators that are below the corresponding threshold as low credibility reasons to the display control module.

[0100] Let the normalized residual score be... Feature scoring Point cloud score Ultrasound score Obstruction score Consistency score Deformation score is Time rating , integration credibility Generate as follows:

[0101] in, , , , , , , and The weights are preset, and the sum of all weights is 1. Each weight can be configured based on the surgical site, tissue type, equipment configuration, or historical registration data.

[0102] When the directional or amplitude difference between surface displacement constraints and internal anatomical feature displacement constraints exceeds the consistency threshold, the reliability evaluation module increases the weight of the integrated local registration residual or decreases the consistency score. When the surface point cloud integrity is lower than the point cloud integrity threshold, the reliability evaluation module decreases the point cloud score. When the ultrasound image clarity is lower than the ultrasound clarity threshold, the reliability evaluation module decreases the ultrasound score. When the instrument occlusion ratio is higher than the occlusion ratio threshold, the reliability evaluation module decreases the occlusion score.

[0103] The credibility evaluation module outputs a fused credibility score. The system then sends the low-confidence cause markers to the display control module. Low-confidence cause markers include missing surface point clouds, insufficient ultrasound image clarity, excessive instrument occlusion, insufficient number of matchable features, inconsistent constraints, abnormal deformation compensation amplitude, or excessively long time since the last effective registration. Example 9

[0104] See Figure 4 The display control module receives the fusion reliability. Updated preoperative multimodal navigation baseline model and local registration window And low credibility reason markers.

[0105] The display control module will integrate credibility. Compared with normal display threshold and safety threshold Comparison, among which .

[0106] when At this time, the display control module outputs a normal display command. The display terminal will overlay the updated lesion boundaries, blood vessels, nerves, instrument positions, or safety boundaries according to the normal style. The normal style includes solid line boundaries, normal transparency, normal layer brightness, or regular prompt labels.

[0107] when At this time, the display control module outputs a degraded display command. The display terminal performs semi-transparent display, dashed boundary display, color weakening display, or prompt label display on the corresponding navigation layer. Degraded display is used to indicate that the current local registration result is for reference but its reliability is insufficient.

[0108] when At this time, the display control module outputs a partial hiding command and a partial re-registration trigger command. The display terminal hides the corresponding partial registration window. Within the navigation layer, the navigation layer of unaffected areas is retained; at the same time, it triggers the reacquisition of intraoperative surface point clouds, reacquisition of intraoperative ultrasound section images, or re-performs local registration.

[0109] The display control module only changes the display state of the navigation layer and does not directly alter the surgeon's surgical instructions. The navigation information output by the display terminal is used to assist intraoperative observation and manual judgment. Example 10

[0110] See Figure 2 This embodiment provides an intraoperative real-time navigation method based on multimodal image fusion, which is performed according to the following steps.

[0111] S1. The preoperative modeling module acquires at least two preoperative medical images of the same surgical area, and performs denoising, grayscale normalization, voxel resampling, and artifact suppression processing on the at least two preoperative medical images; using one of the preoperative medical images as a reference image, the remaining preoperative medical images are registered to the same preoperative image coordinate system; structural segmentation is performed on the lesion area, vascular area, nerve area, bony structure area, and soft tissue area; a preoperative multimodal navigation reference model is generated based on the segmentation results, and tissue deformation parameters are configured according to the tissue area type.

[0112] S2. The initial registration module performs spatial calibration on the surface point cloud acquisition device, intraoperative ultrasound probe, instrument positioning device and display terminal, and establishes the transformation relationship between the preoperative image coordinate system, the patient surgical space coordinate system, the instrument coordinate system and the display coordinate system.

[0113] S3. The initial registration module registers the preoperative multimodal navigation reference model to the patient's surgical space based on the surface structure, bony landmarks, vascular bifurcation points or anatomical boundaries in the preoperative multimodal navigation reference model, as well as the surface point cloud or patient reference pose initially acquired during the operation, and obtains the initial transformation relationship from the preoperative image coordinate system to the patient's surgical space coordinate system.

[0114] S4. The intraoperative acquisition module simultaneously acquires intraoperative surface point cloud, intraoperative ultrasound cross-sectional image, instrument end pose and patient reference pose, and configures a unified timestamp for the above data.

[0115] S5. The local registration window module determines the local registration window based on the instrument end-effector pose, the coverage area of ​​the intraoperative ultrasound section, and the effective field of view of the surface point cloud. The local registration window limits the processing area for this surface point cloud matching, internal anatomical feature matching, local deformation compensation calculation, and fusion reliability evaluation.

[0116] S6. The dual-constraint registration module matches the intraoperative surface point cloud with the corresponding surface region of the preoperative multimodal navigation reference model within the local registration window to generate surface displacement constraints. It also extracts vascular bifurcations, lesion boundaries, or fascial interfaces from the intraoperative ultrasound cross-sectional image and matches them with the corresponding internal structures in the preoperative multimodal navigation reference model to generate internal anatomical feature displacement constraints. The dual-constraint registration module compares the directional and amplitude differences between the two types of displacement constraints and calculates the comprehensive local registration residual.

[0117] S7. The deformation compensation module solves the local deformation compensation field based on surface displacement constraints, internal anatomical feature displacement constraints, and tissue deformation parameters, and updates the corresponding local region of the preoperative multimodal navigation reference model according to the local deformation compensation field. When the two types of displacement constraints are inconsistent, the deformation compensation module reduces the calculation weight of the conflicting constraints or sends the conflict results to the credibility evaluation module.

[0118] S8. The credibility evaluation module generates fusion credibility based on comprehensive local registration residuals, number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, consistency between surface displacement constraints and internal anatomical feature displacement constraints, local deformation compensation amplitude, and time interval since the last effective registration.

[0119] S9. The display control module outputs normal display instructions, downgraded display instructions, local hiding instructions, or local re-registration trigger instructions based on the fusion confidence level, and controls the display status of the corresponding navigation layer in the display terminal. When the fusion confidence level reaches the normal display threshold, the updated lesion boundary, blood vessel, nerve, instrument position, or safety boundary is overlaid and displayed; when the fusion confidence level is lower than the normal display threshold but higher than the safety threshold, the corresponding navigation layer is downgraded; when the fusion confidence level is lower than the safety threshold, the navigation layer in the corresponding local registration window is hidden, and re-acquisition of intraoperative surface point cloud, re-acquisition of intraoperative ultrasound section image, or re-execution of local registration is triggered.

[0120] The modules described above can be deployed on the local processor of the intraoperative navigation device or on an external computing unit connected to the intraoperative navigation device. Data transmission between modules is accomplished through the intraoperative data acquisition interface, instrument positioning interface, device internal bus, memory read / write, or network communication. The processor executes the above data processing steps according to the program stored in the memory, thereby realizing the intraoperative real-time navigation system and method based on multimodal image fusion of the present invention.

[0121] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various variations or substitutions within the technical scope disclosed in the present invention, and these should all be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A real-time intraoperative navigation system based on multimodal image fusion, characterized in that, It includes a processor, memory, intraoperative data acquisition interface, instrument positioning interface, and display terminal. The processor calls programs stored in memory to form: The preoperative modeling module is used to register, segment, and reconstruct three-dimensional images of at least two preoperative medical images of the same surgical area to generate a preoperative multimodal navigation reference model with tissue deformation parameters. The intraoperative acquisition module is used to acquire intraoperative surface point clouds, intraoperative ultrasound cross-sectional images, patient reference pose, and instrument end-effector pose. The initial registration module is used to establish the initial transformation relationship between the preoperative image coordinate system and the patient's surgical space coordinate system; The local window module is used to determine the local registration window based on the instrument end pose, the coverage of the ultrasonic section, and the effective field of view of the surface point cloud. The dual-constraint registration module is used to match the intraoperative surface point cloud with the corresponding surface of the preoperative multimodal navigation reference model within the local registration window to generate surface displacement constraints, and to match the internal anatomical features in the intraoperative ultrasound section image with the corresponding internal structure of the preoperative multimodal navigation reference model to generate internal anatomical feature displacement constraints. The deformation compensation module is used to solve the local deformation compensation field based on two types of displacement constraints and tissue deformation parameters, and update the corresponding local region of the preoperative multimodal navigation reference model. The credibility evaluation module is used to generate fusion credibility based on comprehensive local registration residuals, number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, constraint consistency, and deformation compensation amplitude. The display control module is used to output normal display instructions, downgraded display instructions, partial hiding instructions, or partial re-registration trigger instructions according to the fusion confidence level.

2. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The preoperative modeling module includes: The image preprocessing unit is used to perform noise reduction, grayscale normalization, voxel resampling, and artifact suppression on preoperative medical images. The cross-modal registration unit is used to register the remaining preoperative medical images to the same preoperative image coordinate system, using one of the preoperative medical images as a reference image. Structural segmentation units are used to segment lesion areas, vascular areas, nerve areas, bony structures, and soft tissue areas. A 3D model building unit is used to generate a preoperative multimodal navigation baseline model based on the segmentation results; Organization attribute annotation unit, used to configure the corresponding organizational deformation parameters according to the organizational region type.

3. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The intraoperative data acquisition module includes: The surface point cloud acquisition unit is used to acquire point clouds of exposed tissue surfaces using structured light scanning equipment, depth cameras, or binocular endoscope 3D reconstruction equipment. Intraoperative ultrasound acquisition unit, used to acquire intraoperative ultrasound cross-sectional images through an ultrasound probe with positioning marks; The patient reference pose acquisition unit is used to acquire the spatial position and / or posture of the patient reference frame, bony reference points or surgical area reference marks to form the patient reference pose; The instrument pose acquisition unit is used to acquire the position and posture of the surgical instrument end effector via the instrument positioning interface, thereby forming the instrument end effector pose. The time synchronization unit is used to configure a unified timestamp for intraoperative surface point cloud, intraoperative ultrasound cross-sectional image, patient reference pose, and instrument end-effector pose.

4. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The initial registration module obtains the transformation matrix of the surface point cloud acquisition device, intraoperative ultrasound probe, instrument positioning device, and display terminal relative to the patient's surgical space coordinate system through calibration plate calibration, probe touch calibration, reference frame positioning calibration, or hand-eye calibration. Based on the surface structure, bony landmarks, vascular bifurcation points, or anatomical boundaries in the preoperative multimodal navigation reference model, it matches them with the surface point cloud or patient reference pose initially acquired during the operation to obtain the initial transformation relationship from the preoperative image coordinate system to the patient's surgical space coordinate system.

5. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The local window module uses the instrument tip position, the center of the intraoperative ultrasound section, or the center of the effective field of view of the surface point cloud as the window center, and combines the current operation stage in the preoperative planning path with the position of at least one key structure among the lesion, blood vessel, and nerve to determine the spatial range of the local registration window; the local registration window limits the processing area for this surface point cloud matching, internal anatomical feature matching, local deformation compensation calculation, and fusion reliability evaluation.

6. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The dual-constraint registration module includes: The surface point cloud constraint unit is used to match the intraoperative surface point cloud within the local registration window with the corresponding surface region of the preoperative multimodal navigation reference model to generate surface displacement constraints. The internal ultrasound feature constraint unit is used to extract vascular bifurcation, lesion boundary or fascial layer interface from intraoperative ultrasound cross-sectional images and match them with the corresponding internal structures in the preoperative multimodal navigation reference model to generate internal anatomical feature displacement constraints. The constraint consistency judgment unit is used to compare the directional and amplitude differences between surface displacement constraints and internal anatomical feature displacement constraints. The local registration residual calculation unit is used to calculate the comprehensive local registration residual based on the surface point cloud residual, internal feature residual, and constraint consistency results.

7. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The deformation compensation module extracts local model units of the preoperative multimodal navigation reference model within the local registration window, reads the corresponding tissue deformation parameters according to the tissue region type of the local model unit, and solves the local deformation compensation field through a finite element model, a radial basis function interpolation model, or a mesh control point deformation model based on surface displacement constraints, internal anatomical feature displacement constraints, and tissue deformation parameters. When the directional difference or amplitude difference between the surface displacement constraint and the internal anatomical feature displacement constraint exceeds the consistency threshold, the calculation weight of the conflict constraint is reduced, or the conflict result is sent to the credibility evaluation module.

8. The intraoperative real-time navigation system based on multimodal image fusion according to claim 1, characterized in that: The reliability evaluation module normalizes the comprehensive local registration residual, the number of matchable features, the integrity of the surface point cloud, the clarity of the ultrasound image, the instrument occlusion ratio, the consistency between surface displacement constraints and internal anatomical feature displacement constraints, the local deformation compensation amplitude, and the time interval since the last effective registration, and generates a fusion reliability based on the normalization result. The display control module compares the fusion reliability with the normal display threshold and the safety threshold. When the fusion reliability reaches the normal display threshold, it outputs a normal display command. When the fusion reliability is lower than the normal display threshold but higher than the safety threshold, it outputs a downgraded display command. When the fusion reliability is lower than the safety threshold, it outputs a local hiding command and a local re-registration trigger command.

9. A real-time intraoperative navigation method based on multimodal image fusion, characterized in that, Includes the following steps: S1. Acquire at least two preoperative medical images of the same surgical area, perform cross-modal registration, structural segmentation and three-dimensional reconstruction on the at least two preoperative medical images, and generate a preoperative multimodal navigation reference model with tissue deformation parameters; S2. Spatial calibration of the surface point cloud acquisition device, intraoperative ultrasound probe, instrument positioning device and display terminal is performed to establish the transformation relationship between the preoperative image coordinate system, the patient surgical space coordinate system, the instrument coordinate system and the display coordinate system; S3. Based on the anatomical structures in the preoperative multimodal navigation reference model and the surface point cloud or patient reference pose initially acquired during the operation, the preoperative multimodal navigation reference model is initially registered to the patient's surgical space; S4. Intraoperatively acquire surface point cloud, intraoperative ultrasound cross-sectional image, instrument end-effector pose and patient reference pose; S5. Determine the local registration window based on the instrument end position, the coverage of intraoperative ultrasound sections, and the effective field of view of surface point clouds; S6. Within the local registration window, match the intraoperative surface point cloud with the corresponding surface region of the preoperative multimodal navigation reference model to generate surface displacement constraints, and match the internal anatomical features in the intraoperative ultrasound cross-sectional image with the corresponding internal structure in the preoperative multimodal navigation reference model to generate internal anatomical feature displacement constraints. S7. Solve the local deformation compensation field based on surface displacement constraints, internal anatomical feature displacement constraints and tissue deformation parameters, and update the corresponding local region of the preoperative multimodal navigation reference model according to the local deformation compensation field; S8. Generate fusion reliability based on comprehensive local registration residuals, number of matchable features, surface point cloud integrity, ultrasound image clarity, instrument occlusion ratio, consistency between surface displacement constraints and internal anatomical feature displacement constraints, and deformation compensation amplitude. S9. Output normal display instructions, downgraded display instructions, partial hiding instructions, or partial re-registration trigger instructions based on the fusion confidence level, and control the display status of the corresponding navigation layer in the display terminal.

10. The intraoperative real-time navigation method based on multimodal image fusion according to claim 9, characterized in that, In step S8, the surface point cloud integrity is used to characterize the acquisition quality of the surface point cloud during surgery, and the ultrasound image clarity is used to characterize the imaging quality of the ultrasound section image during surgery. When the directional difference or amplitude difference between the surface displacement constraint and the internal anatomical feature displacement constraint exceeds the consistency threshold, the weight of the comprehensive local registration residual is increased or the fusion confidence is reduced. In step S9, when the fusion confidence reaches the normal display threshold, the updated lesion boundary, blood vessel, nerve, instrument position or safety boundary is superimposed and displayed. When the fusion confidence is lower than the normal display threshold but higher than the safety threshold, the corresponding navigation layer is displayed with semi-transparent display, dashed boundary display, color weakening display or prompt label display. When the fusion confidence is lower than the safety threshold, the navigation layer in the corresponding local registration window is hidden, and the intraoperative surface point cloud is reacquired, the intraoperative ultrasound section image is reacquired, or the local registration is re-executed.

Images