A visual catheter, image fusion device, real-time navigation system based on image fusion, method, medium and program product

Abstract

The application discloses a visible catheter, an image fusion device, a real-time navigation system based on image fusion, a method, a medium and a program product. The real-time navigation system comprises a CT image acquisition device, which is used for acquiring original CT images to be fused; a visible catheter, which is used for acquiring real-time video through an imaging assembly arranged at a tip of the visible catheter and navigating and tracking the visible catheter through a navigation and tracking assembly arranged at a tail end of the visible catheter; a tracking device, which is used for acquiring real-time pose data of the visible catheter in cooperation with the navigation and tracking assembly; an image fusion device, which is used for fusing the real-time video with a DRR image to be fused, which is generated based on a rendering range of the DRR determined based on the original CT images and a tip of a surgical tool in real time according to the real-time pose data, to obtain a fused image; and a display device, which is used for outputting the fused image. The system can fuse and display a real-time visual field and preoperative planning images, thereby providing intuitive and accurate navigation guidance for surgery.

Description

Technical Field

[0001] This application relates to the field of medical navigation technology, specifically to a visual catheter, an image fusion device, a real-time navigation system based on image fusion, a method, a medium, and a program product. Background Technology

[0002] Image-guided technology is now widely used in minimally invasive surgical navigation systems such as spinal surgery and neurosurgery.

[0003] For example, patent CN101375805A discloses a method and system for computer-aided guided electronic endoscopy, which tracks the position and orientation of the endoscope probe through a spatial positioning system and fuses the spatial coordinates of the endoscope probe with preoperative CT or X-ray images for display.

[0004] In addition, existing technologies already include navigation solutions that combine endoscopic images with preoperative 3D images.

[0005] For example, patent CN115836914A discloses an endoscopic surgical navigation method and device based on video image augmented reality. It adopts a video image augmented reality method, which installs a passive tracking tool on the endoscope, combines it with a navigation system to calibrate the internal and external parameters of the endoscope, uses a pinhole imaging model to fuse preoperative CT images with endoscopic images, renders and displays virtual images in real time, reduces doctors' judgment time, and improves accuracy and safety.

[0006] However, although the above scheme achieves spatial registration between the endoscope and the three-dimensional image, as well as the combined display of real-time image and three-dimensional image, in the fused image it presents, doctors can only observe a two-dimensional virtual structure superimposed on the real-time surgical field. They cannot perceive the relative spatial position and depth relationship between the virtual structure and the actual surgical tool from the image. As a result, doctors still need to rely on their own experience to infer spatial orientation, and it is difficult to intuitively and accurately judge the positional status between the surgical tool and the target or danger area.

[0007] Specifically, the above solution still has the following shortcomings: First, existing solutions simply convert a 3D model into a 2D model and fuse it with real-time images. The fused image, used as the surgical observation field, is essentially still a 2D planar image, lacking depth information. During the procedure, doctors cannot directly perceive the spatial relationship between the virtual anatomical structure and the tip of the surgical instrument from the fused image, nor can they obtain the real-time distance between the surgical instrument and the danger zone. Consequently, they cannot make accurate spatial judgments based on the fused image.

[0008] Second, the existing solutions only disclose the method of integrating the complete 3D model with real-time images. They adopt a uniform fusion strategy for different types of anatomical structures, which cannot differentiate and highlight the key information that doctors are concerned about, nor can they meet the different observation needs for different anatomical structures.

[0009] Third, existing solutions only provide a single, fixed working mode and cannot be switched according to different surgical stages or usage scenarios, resulting in insufficient adaptability and flexibility of the navigation system in different usage scenarios.

[0010] In summary, although the existing solutions have achieved the combined display of real-time endoscopic images and preoperative 3D images, their fusion method is essentially still at the level of superimposing two-dimensional planar images. It cannot allow doctors to perceive the spatial relationship between surgical tools and virtual anatomical structures from the fused image, nor can it provide differentiated fusion display strategies for different anatomical structures, nor can it switch to an appropriate working mode according to different stages of surgery. Summary of the Invention

[0011] In view of the above-mentioned deficiencies, this application provides a visual catheter, an image fusion device, a real-time navigation system based on image fusion, a method, a medium, and a program product, which can solve the problem that the information dimensions provided by the image fusion display method in the existing surgical navigation system are single and cannot support the needs of real-time on-site judgment and multi-stage operation adaptation during surgery.

[0012] In a first aspect, this application provides a visual catheter, including a catheter body, an imaging component, and a navigation and tracking component, wherein the catheter body is a rigid structure including a tip and a tail end; The imaging component is fixedly mounted on the tip of the catheter body for acquiring real-time video; The navigation and tracking component is fixedly installed at the tail end of the duct body and is used to navigate and track the visual duct in order to obtain the real-time pose data of the visual duct in the real coordinate system. The real-time pose data is used to determine the real-time extrinsic parameters of the imaging component, which represent the pose of the imaging component in the real coordinate system. The real-time extrinsic parameters are used to dynamically determine the DRR rendering range by combining the real-time tip position of the surgical tool in the real coordinate system and the camera imaging parameters of the imaging component; the DRR rendering range is updated as the real-time tip position changes. The DRR rendering range is used to combine the target 3D volume data with annotation information in the real coordinate system to generate a DRR image to be fused that corresponds to both the camera imaging parameters and the real-time extrinsic parameters. The DRR image to be fused is used to perform fusion processing with the real-time video to obtain a fused image.

[0013] In the above technical solution, the visual catheter, through its rigid structure, ensures that the relative pose relationship between the tip imaging component and the tail navigation and tracking component is fixed and known. This establishes a precise spatial coupling benchmark for the imaging coordinate system and the tracking coordinate system from the hardware source, effectively eliminating pose drift caused by the relative motion of the components and providing a solid data alignment foundation for subsequent high-precision image fusion. Simultaneously, by using the tip of the surgical tool as the driver, the DRR rendering range and the fusion display area are precisely limited to the vicinity of the current working area of ​​the surgical tool. This allows the labeled virtual anatomical structure and the real surgical field to form a spatial correspondence in the fused image, with the surgical tool as the perceptual subject. This enables the surgeon to directly perceive the relative position between the tool and the anatomical structure from the fused image. Furthermore, since DRR image generation only requires calculations on three-dimensional volume data within a limited range, it also reduces the computational overhead per frame and improves intraoperative real-time performance.

[0014] In some embodiments, the imaging component includes a miniature camera, the forward shooting direction of the miniature camera and the upward direction of the miniature camera forming a first plane; The tracking direction of the navigation tracking component and the upward direction of the navigation tracking component form a second plane; The first plane coincides with the second plane.

[0015] In the above technical solution, the setting relationship between the imaging component and the navigation tracking component can achieve global pre-alignment of the imaging coordinate system and the tracking coordinate system in terms of attitude, thereby eliminating the angular deviation of pitch and yaw angles at the hardware level. This eliminates the need for complex software attitude correction during subsequent image fusion and registration, effectively reducing the superposition error caused by coordinate system transformation, and ensuring the accuracy and stability of precise alignment between virtual and real images in spatial perspective from the source.

[0016] Secondly, this application provides an image fusion apparatus, comprising: The CT image preprocessing module is used to preprocess the original CT images to obtain target three-dimensional volume data with annotation information in the real coordinate system. The real-time pose analysis module is used to receive the real-time pose data of the visual catheter output by the tracking device, and to analyze the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter. The real-time DRR generation module is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters; the DRR rendering range is updated as the real-time tip position changes. The real-time DRR generation module is also used to generate a DRR image to be fused that corresponds to both the camera imaging parameters and the real-time extrinsic parameters, based on the DRR rendering range and the target 3D volume data. The image fusion and overlay module is used to perform pixel-level image fusion algorithm to fuse the real-time video acquired by the imaging component with the DRR image to be fused, so as to obtain a fused image.

[0017] In the above technical solution, the image fusion device can preprocess the original CT images and add annotation information through the CT image preprocessing module, thereby providing a three-dimensional volume data foundation containing annotation information for subsequent navigation; it can also analyze the real-time pose data output by the tracking device through the real-time pose analysis module to accurately obtain the real-time extrinsic parameters of the imaging components, providing a precise spatial positioning basis for determining the subsequent DRR rendering range; and it can use the DRR real-time generation module, driven by the tip of the surgical tool, to dynamically limit the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters, so that the generated target image can be rendered more accurately. The fused DRR image always focuses on the area where the tool tip is currently located, thus providing a data foundation for the spatial correspondence between the virtual anatomical structure and the real surgical field in the fused image. Finally, the image fusion overlay module uses a pixel-level image fusion algorithm to overlay the real-time video and the DRR image to be fused at the pixel level, so that the virtual annotation information is superimposed on the corresponding position of the real surgical field with the tool tip as the sensing subject. This allows the doctor to directly perceive the spatial relationship between the surgical tool and the anatomical structure from the fused image. At the same time, since the generation of DRR images only requires calculation on the three-dimensional volume data within a limited range, it also has the additional effect of reducing the computational cost per frame.

[0018] In some implementations, the real-time DRR generation module includes a determination submodule and a construction submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The determining submodule is used to determine the axial projection distance of the tip of the surgical tool in the DRR imaging center ray direction based on the camera imaging parameters of the imaging component and the real-time extrinsic parameters. The determining submodule is further configured to determine the axial target position based on the axial projection distance; The construction submodule is used to construct a cone space with the DRR imaging ray source as the vertex, the DRR imaging center ray direction as the axis, the DRR imaging field of view as the cone angle, and the vertical plane of the axis at the target position of the axial direction as the bottom surface. The determining submodule is also used to limit the DRR rendering range based on the cone space.

[0019] In the above technical solution, the image fusion device can realize real-time dynamic cropping of the DRR rendering range according to the position of the surgical tool tip, so that the DRR image to be fused is always focused on the current area of ​​the tool tip, thereby providing a precise data basis for the spatial correspondence between the virtual anatomical structure and the real surgical field with the tool tip as the main sensing subject in the fused image.

[0020] In some embodiments, the real-time DRR generation module further includes: The retrieval submodule is used to retrieve the historical DRR image associated with the historical distance and the historical pose data when the axial projection distance is the same and the real-time pose data is the same, so that the image fusion and overlay module can use the historical DRR image as the DRR image to be fused for fusion processing. The historical distance, the historical pose data, and the historical DRR image are stored together when generating the DRR image to be fused.

[0021] In the above technical solution, the image fusion device can reuse historical DRR images and avoid screen interruptions caused by repeated rendering calculations, thereby ensuring the continuous output of fused images and enabling doctors to stably observe the virtual-real correspondence in the fused images.

[0022] In some implementations, the real-time DRR generation module includes a detection submodule and a determination submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The detection submodule is used to detect whether the surgical tool has entered the field of view of the imaging component; The determination submodule is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters when the surgical tool enters the field of view. The determining submodule is further configured to determine the complete range of the target three-dimensional volume data as the DRR rendering range when the surgical tool does not enter the field of view.

[0023] In the above technical solution, the image fusion device can adaptively switch the DRR rendering range determination strategy according to whether the surgical tool enters the field of view, so that the DRR image accurately covers the area where the tool is located when the surgical tool enters the field of view, thereby ensuring the perception of the positional relationship between the tool and the anatomical structure in the fused image.

[0024] In some implementations, the annotation information includes a first type of annotation information and a second type of annotation information. The real-time DRR generation module includes a rendering submodule. During the process of the real-time DRR generation module generating a DRR image to be fused based on the DRR rendering range and the target 3D volume data, the DRR image corresponds to both the camera imaging parameters and the real-time extrinsic parameters. The rendering submodule is used to extract partial three-dimensional volume data corresponding to the DRR rendering range from the target three-dimensional volume data; extract first type three-dimensional volume data corresponding to the first type of annotation information from the partial three-dimensional volume data; and render the first type three-dimensional volume data to generate a first initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters. The rendering submodule is further configured to extract the second type of three-dimensional volume data corresponding to the second type of annotation information from the target three-dimensional volume data; and to render the second type of three-dimensional volume data to generate a second initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters; wherein, in the second initial DRR sub-image, the area outside the projection corresponding to the second type of three-dimensional volume data is transparent; The first initial DRR sub-image and the second initial DRR sub-image together constitute the DRR image to be fused.

[0025] In the above technical solution, the image fusion device can realize the layered and differentiated presentation and independent distinction of different types of annotation information in the fused image.

[0026] In some embodiments, the image fusion device has a detection mode; when in the detection mode, the CT image preprocessing module is further configured to set position markers in the target three-dimensional volume data; The real-time DRR generation module is also used to dynamically determine the DRR rendering range based on the real-time tip position of the virtual probe tip in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters. The real-time DRR generation module includes a determination submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the virtual probe in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The determining submodule is used to determine the axial projection spacing of the position marker points in the ray direction of the DRR imaging center; The determining submodule is also used to determine the complete range of the target three-dimensional volume data as the DRR rendering range when the axial projection spacing is greater than the detection distance; The determining submodule is further configured to dynamically determine the DRR rendering range based on the detection distance, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters when the axial projection spacing is less than or equal to the detection distance. The detection distance is the distance between the tip of the virtual probe and the DRR imaging X-ray source, and the tip of the virtual probe is always located on the central X-ray of the DRR imaging.

[0027] In the above technical solution, the image fusion device can simulate the catheter advancement process with a virtual probe through the detection mode, thereby obtaining anatomical structure information at different depths in advance during the preoperative planning stage, providing doctors with a reference for path verification and risk prediction.

[0028] In some embodiments, the detection distance has a preset initial value, and the determining submodule is further used to determine the tip position of the visual conduit as a reference position when the axial projection spacing is equal to the preset initial value, and to determine the direction of the DRR imaging center ray as a reference direction. The determining submodule is further configured to update the detection distance based on the preset adjustment magnification and the projection component of the movement distance of the tip of the visual conduit in the reference direction when the axial projection spacing is less than or equal to the preset initial value; The moving distance is the distance between the real-time tip position of the visual conduit in the real coordinate system and the reference position.

[0029] In the above technical solution, the image fusion device can adaptively expand the rendering range as the depth of the visual catheter advances, so that the DRR image always covers a sufficient field of view in front of the catheter tip, thereby providing doctors with a reference for predicting the anatomical structure in the direction of tool advancement.

[0030] In some embodiments, the image fusion and overlay module includes: The distance acquisition submodule is used to acquire the tip coordinates of the visual catheter in the real coordinate system; acquire the three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target three-dimensional volume data; and determine the shortest Euclidean distance between the tip of the visual catheter and the dangerous anatomical area based on the tip coordinates and the three-dimensional point cloud data. The weight determination submodule is used to determine the fusion weights of the real-time video acquired by the imaging component and the DRR image to be fused based on the shortest Euclidean distance. The image fusion submodule is used to fuse the real-time video and the DRR image to be fused according to their respective fusion weights, using a pixel-level image fusion algorithm to obtain a fused image; The smaller the shortest Euclidean distance, the greater the fusion weight of the real-time video and the smaller the fusion weight of the DRR image to be fused.

[0031] In the above technical solution, the image fusion device can obtain the shortest Euclidean distance between the tip of the visual catheter and the dangerous anatomical area, and adaptively adjust the fusion weight of the real-time video and the DRR image to be fused according to the distance. This makes the real surgical field details of the closer the image is, the better the global observation performance of the farther the image is, thus taking into account both the clarity requirements of close-range operation and the global field of view requirements of long-range navigation.

[0032] In some embodiments, the image fusion apparatus further includes: A distance warning module is used to acquire the tip coordinates of a surgical tool in a real coordinate system, wherein the surgical tool is used in conjunction with the visual catheter; acquire 3D point cloud data of a dangerous anatomical region in the real coordinate system based on the target 3D volume data; determine the shortest Euclidean distance between the tip of the surgical tool and the dangerous anatomical region based on the tip coordinates and the 3D point cloud data; determine the warning level based on the shortest Euclidean distance and a preset multi-level warning threshold; and generate a warning prompt message based on the warning level. When the warning level is Level 1, the warning information includes a static visual prompt in the first warning area of ​​the fused image, and the static visual prompt is in a first color. When the warning level is level two, the warning information includes a dynamic visual prompt in the second warning prompt area of ​​the fused image, and the dynamic visual prompt is in a second color; When the warning level is level three, the warning information includes an audible and visual alarm signal and / or a visual warning mark; the visual warning mark includes a highlighted delineator of a third color and a distance value mark, the highlighted delineator of the third color is used to delineate the relative position of the tip of the surgical instrument and the dangerous anatomical area, and the distance value mark is used to display the value of the shortest Euclidean distance.

[0033] In the above technical solution, the image fusion device can obtain the position coordinates of the tip of the surgical tool in the real coordinate system in real time through the distance warning module, and combine it with the three-dimensional point cloud data of the dangerous anatomical area to determine the shortest Euclidean distance between the tip of the surgical tool and the dangerous area, so as to realize real-time distance measurement during the operation. Then, through the preset multi-level warning threshold, different levels of warning prompts are automatically matched according to the distance. For example, a blue static prompt bar reminds the doctor to pay attention to the operation rhythm, a yellow dynamic flashing box warns the doctor to operate with caution, and a three-level sound and light alarm, a red high-brightness circle and distance value display intuitively indicate the dangerous location and the remaining safe distance, thereby effectively reducing the risk of misoperation and ensuring surgical safety.

[0034] Thirdly, this application provides a real-time navigation system based on image fusion, including a CT image acquisition device, a visual catheter, a tracking device, a display device, and the image fusion device described in any one of the second aspects, wherein... The CT image acquisition device is used to acquire the original CT images to be fused. The visual conduit is used to acquire real-time video through an imaging component set at the tip of the visual conduit, and to navigate and track the visual conduit through a navigation and tracking component set at the tail end of the visual conduit. The tracking device is used in conjunction with the navigation tracking component to obtain the real-time pose data of the visual guide tube in the real coordinate system; The display device is used to output the fused image.

[0035] In the above technical solution, the real-time navigation system, through the coordinated operation of CT image acquisition device, visual catheter, tracking device and image fusion device, can use the tip of the surgical tool as a drive to fuse and display DRR images with annotation information and real-time video in the same frame, enabling doctors to directly perceive the spatial positional relationship between surgical tools and anatomical structures from the fused image; at the same time, since the generation of DRR images only requires calculation on the three-dimensional volume data within the dynamically determined rendering range, it can also effectively reduce the computational overhead per frame.

[0036] In some embodiments, the real-time navigation system further includes an operating device, wherein... The operating device is used to receive operating instructions input by the user; the operating instructions include at least one of the following: marking instructions for dangerous anatomical areas in the virtual three-dimensional model, path planning instructions for the movement of the visual catheter, instructions for switching modes during surgery, and warning confirmation.

[0037] In the above technical solution, the real-time navigation system can provide doctors with an intraoperative interactive entry through the operating device, thereby supporting operations such as marking dangerous anatomical areas in the virtual three-dimensional model, planning the movement path of the visual catheter, switching the intraoperative working mode, and confirming the warning information. This allows doctors to flexibly control the navigation process and parameters according to the actual needs of the surgery, thereby enhancing the operability and clinical applicability of the system and improving surgical efficiency.

[0038] Fourthly, this application provides a real-time navigation method based on image fusion, which is applied to a real-time navigation system based on image fusion. The real-time navigation system includes a CT image acquisition device, a visual catheter, a tracking device, a display device, and an image fusion device. The method includes: The CT image acquisition device acquires the original CT images to be fused; The visual catheter acquires real-time video through an imaging component at its tip and navigates and tracks the visual catheter through a navigation and tracking component at its tail. The tracking device works in conjunction with the navigation tracking component to obtain the real-time pose data of the visual duct in the real coordinate system; The image fusion device preprocesses the original CT images to obtain target 3D volume data with annotation information in the real coordinate system; it analyzes the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter; it dynamically determines the DRR rendering range based on the real-time tip position of the surgical tool tip in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters; it generates a DRR image to be fused based on the DRR rendering range and the target 3D volume data, corresponding to both the camera imaging parameters and the real-time extrinsic parameters; and it fuses the real-time video with the DRR image to be fused to obtain a fused image; wherein, the DRR rendering range is updated as the real-time tip position changes. The display device outputs the fused image.

[0039] In the above technical solution, the real-time navigation method, through the entire process of CT image acquisition, real-time video acquisition and pose tracking, dynamic determination of DRR rendering range, image fusion processing, and fused image output, can use the tip of the surgical tool as a driver to fuse the labeled DRR image and real-time video in the same frame, allowing the doctor to directly perceive the spatial positional relationship between the surgical tool and the anatomical structure from the fused image; at the same time, since the generation of DRR image only needs to perform calculations on the three-dimensional volume data within the dynamically determined rendering range, it can also reduce the computational overhead per frame.

[0040] Fifthly, this application provides a readable storage medium storing a computer program, which is executed by a plurality of processors, each of which performs a corresponding step in the image fusion-based real-time navigation method described in the fourth aspect.

[0041] Sixthly, this application provides a computer program product comprising a computer program that is run by a plurality of processors, each of the plurality of processors executing a corresponding step in the real-time navigation method based on image fusion described in the fourth aspect.

[0042] The beneficial effects of this application are as follows: By integrating a miniature camera at the tip of the visual catheter and cooperating with the hardware pre-alignment of the navigation and tracking components, high-precision spatial alignment between real-time surgical field images and virtual anatomical structures is ensured from the hardware source; by using the tip of the surgical tool as the driver, the DRR rendering range and fusion display area are dynamically limited to the vicinity of the tool's current working area, so that the virtual anatomical structure with labeled information and the real surgical field form a spatial correspondence with the tool as the perceptual subject in the same fused image, and the doctor can directly perceive the relative position between the surgical tool and the anatomical structure from the fused image; by calculating the distance between the tool tip and the dangerous anatomical area in real time, multi-level visual warnings are superimposed on the fused image, and the fusion weight of the real video and DRR image is dynamically adjusted according to the distance, so that the virtual and real ratio of the fused image changes in real time with the proximity, further enhancing the doctor's perception of the spatial distance status; in addition, by using a detection mode with a virtual probe for preoperative rehearsal and path verification, doctors can be assisted in assessing anatomical structures and optimizing surgical strategies before actual operation, further reducing the risk of misoperation. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the accompanying drawings used in the embodiments of this application will be briefly introduced below. It should be understood that the following drawings only show some embodiments of this application and should not be regarded as a limitation of the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a schematic diagram of the structure of a visual catheter provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of an image fusion device provided in an embodiment of this application; Figure 3 A schematic diagram of a target three-dimensional volume data with annotation information provided in an embodiment of this application; Figure 4 A real-time video image illustrated by a bone model is provided as an embodiment of this application; Figure 5 A schematic diagram of a fused image effect provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application; Figure 7 A schematic diagram of the system architecture of a real-time navigation system based on image fusion provided for embodiments of this application; Figure 8 This is a flowchart illustrating a real-time navigation method for image fusion provided in an embodiment of this application.

[0045] Some of the reference numerals in the specific implementation methods are as follows: 100-Visual catheter; 110-Catheter body; 120-Imaging component; 121-Miniature camera; 130-Navigation and tracking component; 131-Panel; 132-Tracking marker; 200-Image fusion device; 210-CT image preprocessing module; 220-Real-time pose analysis module; 230-Real-time DRR generation module; 231-Detection submodule; 232-Determination submodule; 233-Construction submodule; 234-Retrieval submodule; 235-Rendering submodule; 240-Image fusion and overlay module; 241-Distance acquisition submodule; 242-Weight determination submodule; 243-Image fusion submodule; 300-Electronic device; 301-Processor; 302-Memory; 303-Communication bus; 400-CT image acquisition device; 500-Tracking device; 600-Display device. Detailed Implementation

[0046] The embodiments of the technical solution of this application will now be described in detail with reference to the accompanying drawings. These embodiments are only used to more clearly illustrate the technical solution of this application and are therefore merely examples, and should not be used to limit the scope of protection of this application.

[0047] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0048] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more (including two), similarly, "multiple sets" refers to two or more sets (including two sets), and "multiple pieces" refers to two or more pieces (including two pieces), unless otherwise explicitly specified.

[0049] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0050] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0051] In existing spinal surgery navigation systems, when preoperative CT images and intraoperative real-time videos are fused and displayed, the fused image is usually just a simple two-dimensional overlay of a three-dimensional model onto the real-time surgical field. Doctors cannot perceive the spatial relationship between the virtual anatomical structure and the surgical tools from the image, making it difficult to accurately judge the positional status between the tools and the target or danger zone.

[0052] To address the aforementioned technical issues, this application provides a real-time navigation method and system based on image fusion. The aim is to dynamically limit the DRR rendering range and fusion display area to the vicinity of the tool's current working area by using the tip of the surgical tool as the driving force. This allows the virtual anatomical structure with annotation information and the real surgical field to form a spatial correspondence with the tool as the perceptual subject in the same image, thereby enabling the doctor to directly perceive the relative position between the tool and the anatomical structure from the fused image.

[0053] like Figure 1 As shown, some embodiments of this application provide a visual catheter 100, including a catheter body 110, an imaging component 120, and a navigation and tracking component 130, wherein the catheter body 110 is a rigid structure including a tip and a tail. The imaging component 120 is fixedly mounted on the tip of the catheter body 110 for acquiring real-time video; The navigation and tracking component 130 is fixedly installed at the tail end of the duct body 110 and is used to navigate and track the visual duct 100 in order to obtain the real-time pose data of the visual duct 100 in the real coordinate system. Among them, the real-time pose data is used to determine the real-time extrinsic parameters of the imaging component 120, which represent the pose of the imaging component 120 in the real coordinate system. Real-time extrinsic parameters are used to dynamically determine the DRR rendering range by combining the real-time tip position of the surgical instrument tip in the real coordinate system and the camera imaging parameters of the imaging component 120; the DRR rendering range is updated as the real-time tip position changes. DRR rendering range is used to combine the target 3D volume data with annotation information in the real coordinate system to generate a DRR image to be fused that corresponds to both camera imaging parameters and real-time extrinsic parameters. The DRR image to be fused is used to fuse with real-time video to obtain a fused image.

[0054] In this embodiment, the real-time pose data is obtained by the tracking device 500 when it performs navigation tracking on the navigation tracking component 130. The purpose of the tracking device 500 in obtaining the real-time pose data is to send it to the image fusion device so that the image fusion device 200 can analyze it and obtain the real-time extrinsic parameters of the imaging component 120 in the visual conduit 100 (i.e., the position and orientation of the imaging component 120 in the real coordinate system, specifically the three-dimensional coordinates of the optical center of the imaging component 120 in the real coordinate system, and the orientation of the three coordinate axes of the camera in the real coordinate system).

[0055] In this embodiment, the real-time pose analysis process is as follows: based on the real-time pose data of the visual conduit 100 and the pose transformation relationship between the imaging component 120 and the navigation tracking component 130 in the visual conduit 100, the real-time extrinsic parameters of the imaging component 120 are determined; at the same time, the preset camera imaging parameters of the imaging component 120 are obtained.

[0056] In this embodiment, the specific method for determining the DRR rendering range is as follows: determine the axial projection distance of the tip of the surgical tool in the current DRR imaging center ray direction; determine the axial target position based on the axial projection distance; construct a cone space with the DRR imaging ray source as the vertex, the DRR imaging center ray direction as the axis, the DRR imaging field of view as the cone angle, and the vertical plane of the axis at the axial target position as the bottom surface, and limit the cone space to the DRR rendering range.

[0057] In this embodiment, historical distance, historical pose data, and historical DRR images are stored each time a DRR image to be fused is generated. Based on this, when the axial projection distance is the same as the historical distance and the real-time pose data is the same as the historical pose data, the DRR image to be fused can directly use the stored corresponding historical DRR image.

[0058] In this embodiment, when the surgical tool enters the field of view of the imaging component 120, the DRR rendering range is dynamically determined based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters, and the real-time extrinsic parameters. When the surgical tool does not enter the field of view of the imaging component 120, the DRR rendering range is directly the complete range of the target three-dimensional volume data.

[0059] In this embodiment, the initial DRR image may include a first initial DRR sub-image and a second initial DRR sub-image; The first initial DRR sub-image is the rendering result of the first type of three-dimensional volume data corresponding to the first type of annotation information in the part of the three-dimensional volume data corresponding to the DRR rendering range in the target three-dimensional volume data; The second initial DRR sub-image is the rendering result of the second type of 3D volume data corresponding to the second type of annotation information in the target 3D volume data; in this second initial DRR sub-image, the areas outside the projection corresponding to the second type of 3D volume data are all transparent; Both the first type of annotation information and the second type of annotation information are annotation information.

[0060] In this embodiment, when a position marker is set in the target three-dimensional volume data, the DRR rendering range can be determined by the axial projection distance between the tip of the virtual probe and the position marker in the ray direction of the DRR imaging center.

[0061] In this embodiment, when the axial projection spacing is greater than the detection distance, the DRR rendering range is generated based on the complete range of the target 3D volume data; When the axial projection spacing is less than or equal to the detection distance, the DRR rendering range is dynamically determined based on the detection distance, the camera imaging parameters of the imaging component 120, and real-time extrinsic parameters.

[0062] In this embodiment, the detection distance is the distance between the tip of the virtual probe and the DRR imaging X-ray source, and the tip of the virtual probe is always located on the central X-ray of the DRR imaging.

[0063] In this embodiment, when the axial projection spacing is less than or equal to the detection distance, the detection distance will change as the tip of the visual conduit 100 moves. The detection distance is updated as follows: first, when the axial projection spacing is equal to the preset initial value, the position of the tip of the visual conduit 100 is determined as the reference position, and the direction of the DRR imaging center ray is determined as the reference direction; then, when the axial projection spacing is less than or equal to the preset initial value, the detection distance is updated according to the preset adjustment magnification and the projection component of the movement distance of the tip of the visual conduit 100 in the reference direction.

[0064] In this embodiment, the moving distance is the distance between the real-time tip position and the reference position of the tip of the visual conduit 100 in the real coordinate system.

[0065] In this embodiment, the camera imaging parameters are pre-calibrated (i.e. preset), and the target three-dimensional volume data with annotation information in the real coordinate system is obtained by the image fusion device 200 preprocessing the original CT image.

[0066] Specifically, the target three-dimensional volume data with annotation information in the real coordinate system is obtained by transforming the preoperative CT three-dimensional volume data in the CT coordinate system to the real coordinate system according to the coordinate transformation relationship between the CT coordinate system and the real coordinate system. Among them, the annotation information in the preoperative CT three-dimensional volume data in the CT coordinate system is obtained by annotating the dangerous anatomical areas in the virtual three-dimensional model according to the received annotation instructions; The coordinate transformation relationship between the CT coordinate system and the real coordinate system is determined by obtaining the first position of the registration auxiliary block in the CT coordinate system and the second position of the registration auxiliary block in the real coordinate system in the virtual 3D model, and based on the first and second positions. A virtual 3D model is a 3D model corresponding to the anatomical structure obtained by reconstructing the original CT images.

[0067] In this embodiment, the DRR image to be fused and the real-time video are specifically used for fusion processing based on a pixel-level image fusion algorithm to obtain a fused image.

[0068] In this embodiment, the fused image can be obtained by fusing the real-time video acquired by the visual conduit 100 with the DRR image to be fused based on the alpha channel weighted fusion algorithm.

[0069] In practice, the visual catheter 100 usually works together with surgical tools (such as puncture needles). Therefore, the real-time video acquired by the visual catheter 100 usually includes surgical tools, and the visual catheter 100 can also work in conjunction with surgical tools.

[0070] Specifically, when the coordinates of the tip of the surgical tool are detected, the distance between these coordinates and the 3D point cloud data of the dangerous anatomical region in the real coordinate system obtained from the target 3D volume data is confirmed, thereby obtaining the shortest Euclidean distance between the tip of the surgical tool and the dangerous anatomical region. Then, this shortest Euclidean distance guides the fusion weight of image fusion (the smaller the shortest Euclidean distance, the greater the fusion weight of the real-time video and the smaller the fusion weight of the DRR image to be fused). At the same time, this shortest Euclidean distance also guides the generation of warning levels in order to generate appropriate warning prompts (such as different levels of visual prompts, visual prompt colors, and distance values).

[0071] In the above embodiments, the visual catheter, through its rigid structure, ensures that the relative pose relationship between the tip imaging component and the tail navigation and tracking component is fixed and known. This establishes a precise spatial coupling reference for the imaging coordinate system and the tracking coordinate system from the hardware source, effectively eliminating pose drift caused by the relative motion of the components and providing a solid data alignment foundation for subsequent high-precision image fusion. Simultaneously, by using the tip of the surgical tool as the driver, the DRR rendering range and the fusion display area are precisely limited to the vicinity of the current working area of ​​the surgical tool. This allows the labeled virtual anatomical structure and the real surgical field to form a spatial correspondence in the fused image, with the surgical tool as the perceptual subject. This enables the surgeon to directly perceive the relative position between the tool and the anatomical structure from the fused image. Furthermore, since DRR image generation only requires calculations on three-dimensional volume data within a limited range, it also reduces the computational overhead per frame and improves intraoperative real-time performance.

[0072] In some embodiments, the imaging assembly 120 includes a miniature camera 121, the forward shooting direction of the miniature camera 121 and the upward direction of the miniature camera 121 forming a first plane; The tracking direction of the navigation tracking component 130 and the upward direction of the navigation tracking component 130 form a second plane; The first plane mentioned above coincides with the second plane mentioned above.

[0073] In this embodiment, the miniature camera 121 is configured with an imaging coordinate system, and the navigation and tracking component 130 is configured with a tracking coordinate system.

[0074] In this embodiment, the imaging coordinate system is a three-dimensional spatial reference system established with the optical center of the miniature camera 121 as the origin and the forward shooting direction and upward direction of the miniature camera 121 as the reference. This imaging coordinate system is used to describe the orientation of the image captured by the miniature camera 121 in three-dimensional space. Specifically, the forward shooting direction of the miniature camera 121 corresponds to the z-axis direction of the imaging coordinate system, and the upward direction of the miniature camera 121 corresponds to the y-axis direction of the imaging coordinate system.

[0075] In this embodiment, the tracking coordinate system is a three-dimensional spatial reference system established with reference to the navigation tracking component 130. This tracking coordinate system is used to describe the position and orientation of the visual guide tube 100 in real space. Specifically, the tracking coordinate system uses the tracking direction of the navigation tracking component 130 as the z-axis direction and the upward direction as the y-axis direction.

[0076] In this embodiment, the first plane is the yz plane of the imaging coordinate system, and the second plane is the yz plane of the tracking coordinate system. The overlap of the first and second planes means that the yz plane of the imaging coordinate system coincides with the yz plane of the tracking coordinate system.

[0077] Implementing this method ensures that the imaging coordinate system and the tracking coordinate system are aligned in the x-axis direction at the hardware structure level. That is, pre-alignment is achieved on the axis perpendicular to the plane formed by the shooting direction and the upward direction. This reduces the number of attitude parameters that need to be corrected in subsequent image fusion and registration, reduces the complexity of coordinate system transformation, and provides a certain hardware foundation for spatial alignment of virtual and real images.

[0078] As an optional implementation, the shooting direction of the miniature camera 121 is consistent with the tracking direction of the navigation tracking component 130, and the upward direction of the miniature camera 121 is consistent with the upward direction of the navigation tracking component 130.

[0079] In this embodiment, the shooting direction of the miniature camera 121 and the tracking direction of the navigation and tracking component 130 are the same, meaning that the z-axis direction of the imaging coordinate system and the z-axis direction of the tracking coordinate system are the same; the upward direction of the miniature camera 121 and the upward direction of the navigation and tracking component 130 are the same, meaning that the y-axis direction of the imaging coordinate system and the y-axis direction of the tracking coordinate system are the same.

[0080] In this way, there is only a translational transformation relationship between the imaging coordinate system and the tracking coordinate system in the z-axis direction, and no rotational relationship. This can effectively reduce the rotational deviation caused by coordinate system transformation from the hardware source, so that only simple attitude correction is needed to achieve accurate alignment in spatial viewpoint during subsequent virtual-real image fusion, thereby significantly improving the registration efficiency and spatial accuracy of image fusion.

[0081] For example, when the miniature camera 121 can be mounted on the 30° inclined surface at the tip of the visual conduit 100, it is only necessary to make the UP direction of the miniature camera 121 form a 60° angle with the Y-axis of the tracking coordinate system of the visual conduit 100 itself, and the positive shooting direction of the miniature camera 121 form a 60° angle with the Z-axis of the tracking coordinate system of the visual conduit 100 itself.

[0082] This design enables dual-axis locking to avoid pitch and yaw angle deviations. Furthermore, in conjunction with coordinate transformation, it can achieve full-domain attitude pre-coupling between the imaging coordinate system and the duct tracking coordinate system, so that the software does not need extensive correction in the later stage, thus reducing image overlay and registration errors from the hardware source.

[0083] In the above embodiments, the arrangement of the imaging component 120 and the navigation tracking component 130 enables global pre-alignment of the imaging coordinate system and the tracking coordinate system in terms of attitude. This eliminates the angular deviation of pitch and yaw angles at the hardware level, eliminating the need for complex software attitude correction during subsequent image fusion and registration. This effectively reduces the superposition error caused by coordinate system transformation, ensuring the accuracy and stability of precise alignment between virtual and real images in spatial perspective from the source.

[0084] In some embodiments, the imaging component 120 includes a miniature camera 121, which is used to acquire real-time video. The navigation tracking component 130 includes a panel 131 and a plurality of tracking markers 132; The panel 131 is fixedly installed at the tail end of the catheter body 110 and is used to install multiple tracking markers 132. Multiple tracking markers 132 are fixedly installed on the panel 131 for navigation and tracking of the visual conduit 100 to obtain real-time pose data of the visual conduit 100 in the real coordinate system.

[0085] In this embodiment, multiple tracking markers 132 are used in conjunction with the tracking device 500. The tracking device 500 tracks and positions the multiple tracking markers 132 to obtain the real-time pose data of the visual conduit 100 in the real coordinate system.

[0086] In this embodiment, in the optical tracking scenario, the navigation tracking component 130 may include four coplanar reflective tracking spheres (i.e., tracking markers 132), and the four coplanar reflective tracking spheres are respectively located at the four corners of the quadrilateral panel; wherein, the line connecting two diagonally arranged reflective tracking spheres on the quadrilateral panel is coplanar with the central axis of the conduit body 110 of the visible conduit 100, the line defining the z-axis of the tracking coordinate system, and the normal of the quadrilateral defining the y-axis of the tracking coordinate system.

[0087] In this embodiment, the tracking marker 132 can be a passive infrared reflective marker, which is fixedly installed on the panel 131 at the tail end of the visual conduit 100. It uses reflected infrared light to cooperate with the tracking device 500 to obtain the real-time pose data of the visual conduit 100.

[0088] Specifically, the tracking device 500 can emit infrared light to illuminate the surface of a passive infrared reflective marker. The passive infrared reflective marker reflects the infrared light, which is then received by the tracking device 500 through a camera. The tracking device 500 can then calculate the real-time pose data of the visual guide tube 100 based on the spatial positions of multiple passive infrared reflective markers.

[0089] In this embodiment, the passive infrared reflective marker has the following characteristics: passive design, no power supply required, only reflects infrared light; rigidly fixed to panel 131, which is fixed to the tail end of the conduit body 110, ensuring structural stability and reliable positioning; multiple tracking balls work together to provide spatial geometric constraints and improve positioning accuracy; and when used in conjunction with tracking device 500 (which may include an optical tracker and a bracket), it enables real-time navigation and tracking.

[0090] In this embodiment, the method uses passive infrared optical tracking as an example for navigation and tracking. Depending on actual needs, other tracking methods can also be appropriately adopted (for example, tracking using magnetic navigation, where the tracking marker 132 is an electromagnetic sensor and the tracking device is a magnetic field generator). It should be understood that this embodiment does not limit the specific type of other tracking methods.

[0091] In the above embodiments, the visual catheter 100 can directly acquire real-time visual images of the catheter tip through the miniature camera 121, providing intuitive tissue image information for the operation; through multiple tracking markers 132 on the panel 131 and in cooperation with the external tracking device, real-time tracking of the pose of the visual catheter 100 is achieved, improving the accuracy of pose data acquisition; at the same time, the panel 131 is fixed to the tail end of the catheter body 110 and the tracking markers 132 are fixed on the panel, which helps to maintain the overall structural stability and tracking accuracy of the navigation tracking component 130, thereby providing a reliable spatial positioning basis for subsequent image fusion and navigation warning.

[0092] like Figure 2 As shown, some embodiments of this application provide an image fusion apparatus 200, including: The CT image preprocessing module 210 is used to preprocess the original CT images to obtain target three-dimensional volume data with annotation information in the real coordinate system. The real-time pose analysis module 220 is used to receive the real-time pose data of the visual catheter output by the tracking device, and analyze the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter. The real-time DRR generation module 230 is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters; the DRR rendering range is updated as the real-time tip position changes. The real-time DRR generation module 230 is also used to generate a DRR image to be fused that corresponds to both camera imaging parameters and real-time extrinsic parameters, based on the DRR rendering range and the target 3D volume data. The image fusion and overlay module 240 is used to fuse the real-time video acquired by the imaging component with the DRR image to be fused using a pixel-level image fusion algorithm to obtain a fused image.

[0093] In this embodiment, the CT image preprocessing module 210 can preprocess the patient's original CT images. This preprocessing includes three-dimensional reconstruction, annotation of dangerous anatomical regions, and surgical path planning.

[0094] In this embodiment, the CT image preprocessing module 210 can also perform system registration based on the coordinate transformation relationship between the CT coordinate system and the real coordinate system of the original CT image, thereby mapping the preoperative three-dimensional volume data to the real space.

[0095] In this embodiment, the visible catheter is Figure 1 The specific structure of the visible catheter shown will not be described in detail here.

[0096] In this embodiment, the camera imaging parameters of the imaging component can be pre-calibrated using the Zhang Zhengyou calibration method on the miniature camera at the tip of the visual conduit. These camera imaging parameters include at least the intrinsic parameter matrix and distortion parameters, and may also include exposure parameters, white balance parameters, etc., depending on the specific requirements.

[0097] In this embodiment, the camera imaging parameters serve as intrinsic camera parameters, which determine the DRR imaging field of view of the subsequently constructed cone space; the real-time extrinsic parameters serve as extrinsic camera parameters, which determine the position of the DRR imaging ray source and the direction of the DRR imaging center ray in the real coordinate system. Therefore, the DRR rendering range is jointly determined by the camera imaging parameters and the real-time extrinsic parameters.

[0098] In this embodiment, the camera imaging parameters of the imaging component refer to the camera imaging parameters of the miniature camera in the imaging component. These camera imaging parameters are calibrated and determined at the factory and are generally not calibrated again during or before the operation.

[0099] In this embodiment, the real-time pose analysis module 220 can receive the real-time pose data of the visual duct output by the tracking device 500 through the communication interface, and perform analysis processing on the real-time pose data.

[0100] In this embodiment, surgical instruments refer to actual surgical instruments whose tips can be tracked in real time by a tracking device.

[0101] In this embodiment, the update frequency of the DRR rendering range and its display refresh rate can be set independently. Due to the existence of the DRR rendering range, this display refresh rate is typically much higher than that of full-screen rendering.

[0102] In this embodiment, the pixel-level image fusion algorithm may include an alpha channel weighted fusion algorithm, a frequency domain-based multi-resolution fusion algorithm (such as Laplacian pyramid fusion), or a color space-based fusion algorithm (such as HSV mapping), etc.

[0103] As an optional implementation, the image fusion and overlay module is specifically used to use an alpha channel weighted fusion algorithm to fuse the real-time video acquired by the visual conduit with the DRR image to be fused, thereby obtaining a fused image.

[0104] For example, the image fusion and overlay module 240 can use an alpha channel weighted fusion algorithm to overlay the real-time video acquired by the visual conduit with the DRR image generated from the corresponding viewpoint at the pixel level. Specifically, if the initial fusion weight of the DRR image is set to 0.5 and the initial fusion weight of the real-time image is also set to 0.5, then semi-transparent overlay and fusion of virtual and real images can be achieved.

[0105] Implementing this method, the image fusion device 200 can provide doctors with real-time intraoperative navigation functions based on the registration results and fused images, including catheter positioning, path deviation alerts, and distance warnings.

[0106] Figures 3 to 5 The diagram illustrates the effects of each processing stage in some embodiments of this application.

[0107] in, Figure 3 This is the target 3D volume data with annotation information obtained after the CT image preprocessing module performs 3D annotation on the original CT image. The annotated areas are marked with a bright color to indicate dangerous anatomical areas. Figure 4 The real-time video footage acquired by the imaging component (in this embodiment, the footage is illustrated using a bone model); Figure 5 The image fusion and overlay module is used to fuse the real-time video acquired by the imaging component with the DRR image to be fused from the corresponding viewpoint. The labeled information corresponds to the real tissue in the image.

[0108] In the above embodiments, the image fusion device can preprocess the original CT images and add annotation information through the CT image preprocessing module, thereby providing a three-dimensional volume data foundation containing annotation information for subsequent navigation; it can analyze the real-time pose data output by the tracking device through the real-time pose analysis module to accurately obtain the real-time extrinsic parameters of the imaging components, providing a precise spatial positioning basis for determining the subsequent DRR rendering range; and it can dynamically limit the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters through the real-time DRR generation module, so that the generated image fusion device can achieve the desired effect. The DRR image is always focused on the area where the tool tip is currently located, thus providing a data foundation for the spatial correspondence between the virtual anatomical structure and the real surgical field in the fused image. Finally, the real-time video and the DRR image to be fused are superimposed pixel by pixel using a pixel-level image fusion algorithm through the image fusion overlay module. This allows the virtual annotation information to be superimposed on the corresponding position of the real surgical field with the tool tip as the main sensing subject. This enables the doctor to directly perceive the spatial relationship between the surgical tool and the anatomical structure from the fused image. At the same time, since the generation of DRR images only requires calculation on the three-dimensional volume data within a limited range, it also has the additional effect of reducing the computational cost per frame.

[0109] In some embodiments, the CT image preprocessing module 210 includes: The three-dimensional reconstruction submodule 211 is used to perform three-dimensional reconstruction based on the original CT images to obtain a virtual three-dimensional model corresponding to the anatomical structure. The calibration submodule 212 is used to obtain the first position of the registration auxiliary block in the CT coordinate system and the second position of the registration auxiliary block in the real coordinate system in the virtual 3D model; and to determine the coordinate transformation relationship between the CT coordinate system and the real coordinate system based on the first position and the second position. The annotation submodule 213 is used to annotate dangerous anatomical regions in the virtual three-dimensional model according to the received annotation instructions, and obtain preoperative CT three-dimensional volume data with annotation information; wherein, dangerous anatomical regions include one or more of the lesion region, the surrounding spinal cord region, and the nerve region; The transformation submodule 214 is used to transform the preoperative CT three-dimensional volume data from the CT coordinate system to the real coordinate system according to the coordinate transformation relationship, so as to obtain the target three-dimensional volume data.

[0110] In this embodiment, the registration auxiliary block includes multiple metal balls that can be visualized under the original CT image. The metal balls are used to provide the spatial position of the metal balls in the CT coordinate system, so as to cooperate with the tracking device 500 to obtain the spatial position of the metal balls in the real coordinate system, thereby calculating the coordinate transformation relationship between the CT coordinate system and the real coordinate system, and thus realizing the spatial registration function.

[0111] In this embodiment, the CT image preprocessing module 210 can receive the acquired raw DICOM format CT images (of the patient) in advance and perform subsequent operations.

[0112] In this embodiment, the annotation submodule 213 can annotate dangerous anatomical areas such as lesion areas, surrounding spinal cord areas, and nerve areas. The annotated areas can be obtained by contour delineation or annotation to form a three-dimensional point cloud. Specifically, the annotation method can be manual annotation or AI annotation using a pre-trained image recognition model.

[0113] In this embodiment, the annotation submodule 213 can automatically map the annotation area to the global CT coordinate system to generate preoperative CT three-dimensional volume data with annotation information.

[0114] In the above embodiments, the image fusion device can accurately map preoperative planning information to the real coordinate system by performing three-dimensional reconstruction, coordinate system registration and marking of dangerous anatomical areas on the original CT images, thereby providing a complete and accurate three-dimensional volume data foundation for real-time intraoperative navigation and precise early warning, and ensuring the spatial accuracy and reliability of navigation information.

[0115] In some embodiments, the real-time pose analysis module 220 is specifically used to determine the real-time extrinsic parameters of the imaging component in the real coordinate system based on the real-time pose data of the visual conduit and the pose transformation relationship between the imaging component and the navigation tracking component in the visual conduit.

[0116] In this embodiment, in a navigation system that has completed registration, the tracking device can track and capture the pose information of multiple objects (such as visual catheters or other surgical tools equipped with locators) in the real coordinate system.

[0117] In this embodiment, the CT three-dimensional volume data corresponds to the actual anatomical structure of the human body, and its pose can also be defined in the real coordinate system.

[0118] In this embodiment, the real-time pose analysis module 220 can determine the pose of the micro camera in the real coordinate system by returning the real-time pose data of the visual guide tube in the real coordinate system based on the tracking device, and combining the pose transformation relationship of the micro camera relative to the navigation tracking component.

[0119] In this embodiment, the real-time pose analysis module 220 can combine the pose of the miniature camera and its camera imaging parameters to determine the imaging space range of the miniature camera under the camera imaging parameters.

[0120] In the above embodiments, the image fusion device can accurately calculate the real pose of the imaging component in the real coordinate system based on the real-time pose data of the visual duct obtained by the tracking device and the pre-calibrated pose transformation relationship between the imaging component and the navigation tracking component, thereby determining the real-time extrinsic parameters of the imaging component, and thus providing accurate spatial orientation parameters for the real-time generation of subsequent DRR images, ensuring that the virtual image and the real field of view correspond accurately in terms of viewpoint.

[0121] In some embodiments, the real-time DRR generation module 230 is specifically used to call the accelerated ray casting DRR algorithm with real-time extrinsic parameters and camera imaging parameters as virtual projection parameters, to render a portion of the three-dimensional volume data corresponding to the DRR rendering range in the target three-dimensional volume data, and generate an initial DRR image corresponding to both the camera imaging parameters and the real-time extrinsic parameters; and to overlay and embed annotation layers of different depths in the portion of the three-dimensional volume data into the initial DRR image to obtain an integrated DRR image to be fused with annotation information. The pixel size of the initial DRR image is the same as the pixel size of the real-time video acquired by the imaging component.

[0122] In this embodiment, partial 3D volume data refers to the portion of the target 3D volume data that corresponds to the DRR rendering range.

[0123] In this embodiment, the DRR real-time generation module 230 can use the currently calculated pose of the miniature camera as a virtual projection parameter to create a virtual camera in the spatial real coordinate system. The pose and imaging parameters of the virtual camera are consistent with the current pose and imaging parameters of the miniature camera.

[0124] In this embodiment, the real-time DRR generation module 230 can call the accelerated ray projection DRR algorithm to render and generate a DRR image from the target three-dimensional volume data that is completely corresponding to both the camera imaging parameters and the real-time extrinsic parameters, and the pixel size of the DRR image is the same as the pixel size of the real-time video captured by the miniature camera.

[0125] In this embodiment, the real-time DRR generation module 230 can overlay and embed annotation layers of different depths in the target 3D volume data into the DRR image to generate an integrated DRR image with annotation information; wherein the annotated parts are displayed in a highlight color (such as red).

[0126] In the above embodiments, the image fusion device can use camera imaging parameters and real-time extrinsic parameters as virtual projection parameters to make the shooting angle of the DRR image accurately correspond to the shooting angle of the actual imaging component; at the same time, by rendering only a portion of the three-dimensional volume data within the DRR rendering range, the computational overhead of generating a single frame of DRR is significantly reduced; in addition, by overlaying and embedding annotation layers of different depths in a portion of the three-dimensional volume data into the initial DRR image, the preoperative annotation information can be accurately presented at the corresponding depth position; finally, by setting the pixel size of the initial DRR image to be the same as the pixel size of the real-time video, a size matching basis can be provided for subsequent pixel-level fusion.

[0127] In some embodiments, the real-time DRR generation module includes a determination submodule 232 and a construction submodule 233. During the process of the real-time DRR generation module dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging components, and real-time extrinsic parameters, the real-time DRR generation module... The determination submodule 232 is used to determine the axial projection distance of the tip of the surgical tool in the DRR imaging center ray direction based on the camera imaging parameters and real-time extrinsic parameters of the imaging component. The determination submodule 232 is also used to determine the axial target position based on the axial projection distance; Construct submodule 233 to build a cone space with the DRR imaging ray source as the vertex, the DRR imaging center ray direction as the axis, the DRR imaging field of view angle as the cone angle, and the vertical plane of the axis at the target position as the bottom surface; Submodule 232 is also used to limit the DRR rendering range based on the cone space.

[0128] In this embodiment, the DRR imaging ray source, the DRR imaging center ray direction, and the DRR imaging field of view can all be determined according to the camera imaging parameters and real-time external parameters.

[0129] For example, the position of the DRR imaging ray source can be determined by the position of the camera optical center in the real-time extrinsic parameters, the direction of the DRR imaging center ray can be determined by the direction of the camera optical axis in the real-time extrinsic parameters, and the DRR imaging field of view can be calculated by the intrinsic parameter matrix in the camera imaging parameters.

[0130] In this embodiment, the axial projection distance refers to the distance between the projection point of the tip of the surgical instrument in the DRR imaging center ray direction and the DRR imaging ray source.

[0131] For example, the device can be operated according to the following steps; (1) Obtain the real-time tip position P1 of the surgical tool tip that enters the field of view of the imaging component in the CT image space (corresponding display coordinate system); (2) Calculate the axial projection distance of the real-time tip position P1 in the direction of the current DRR imaging center ray. Where S is the location of the virtual ray source. The direction vector of the central ray; (3) Determine the axial focal point position parameter H=H1+△H based on the preset redundancy amount △H; △H can be set to 2~5 mm; (4) Taking the virtual ray source S as the vertex and the central ray as the ray source S as the ray source S, the virtual ray source S is ... Construct a cone-shaped space with the axis as the axis and the vertical plane at a distance H1 from the ray source as the base. (5) Perform ray projection calculation only on CT voxels that fall within the cone space to generate DRR images.

[0132] Based on this, the device can perform real-time DRR calculations only on the three-dimensional volume data within a certain range close to the tip of the surgical instrument, and then fuse the DRR image obtained from this calculation with the real-time data. The specific fusion method is as described above and will not be repeated here.

[0133] In this embodiment, the device can update the DRR rendering range in real time as the depth of the surgical tool changes, thereby providing more targeted and flexible display information.

[0134] In the above embodiments, the image fusion device can realize real-time dynamic cropping of the DRR rendering range according to the position of the surgical tool tip, so that the DRR image to be fused is always focused on the current area of ​​the tool tip, thereby providing a precise data basis for the spatial correspondence between the virtual anatomical structure and the real surgical field with the tool tip as the main sensing subject in the fused image.

[0135] In some embodiments, the real-time DRR generation module 230 further includes: The retrieval submodule 234 is used to retrieve the historical DRR image associated with the historical distance and historical pose data when the axial projection distance is the same as the historical distance and the real-time pose data is the same as the historical pose data, so that the image fusion and overlay module 240 can use the historical DRR image as the DRR image to be fused for fusion processing. Among them, historical distance, historical pose data, and historical DRR images are stored together when generating the DRR image to be fused.

[0136] In this embodiment, the device can record historical distance, historical pose data and historical DRR images in pairs through caching, so that they can be directly called under the same conditions.

[0137] In the above embodiments, the image fusion device can reuse historical DRR images, avoid screen interruptions caused by repeated rendering calculations, thereby ensuring the continuous output of fused images and enabling doctors to stably observe the virtual-real correspondence in the fused images.

[0138] In some embodiments, the real-time DRR generation module 230 includes a detection submodule 231 and a determination submodule 232. During the process of the real-time DRR generation module 230 dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and real-time extrinsic parameters, the real-time DRR generation module 230... The detection submodule 231 is used to detect whether the surgical tool has entered the field of view of the imaging component; The determination submodule 232 is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters when the surgical tool enters the field of view. The determination submodule 232 is also used to determine the complete range of the target 3D volume data as the DRR rendering range when the surgical tool is not in the field of view.

[0139] In this embodiment, the visual conduit and the imaging component are rigidly integrated, so it should be understood that the visual conduit will not enter the field of view of the imaging component when the method is implemented.

[0140] In this embodiment, the detection submodule 231 can be used to detect whether the tip of the surgical tool enters the field of view of the imaging component.

[0141] For example, the detection submodule 231 can acquire the real-time tip position of the surgical tool in the real coordinate system, and determine the field of view of the imaging component based on the real-time extrinsic parameters of the imaging component and the camera imaging parameters. Based on this, if the real-time tip position falls within the field of view, it is determined that the surgical tool has entered the field of view; otherwise, it is determined that it has not entered.

[0142] Furthermore, if a DRR image has been generated previously, detection can also be performed based on the cone-shaped space covered by the existing DRR imaging rays to determine whether the coordinates of the surgical tool tip fall within the cone-shaped space covered by the DRR imaging rays. This embodiment does not limit this.

[0143] In this embodiment, the premise for dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, camera imaging parameters, and real-time extrinsic parameters is that the tip of the surgical tool can be located.

[0144] Therefore, it should be understood that detecting whether a surgical tool has entered the field of view of the imaging component is actually detecting whether the tip of the surgical tool has entered the field of view of the imaging component.

[0145] If we infer from the literal meaning, then after detecting that the surgical tool has entered the field of view, there should be a further step to detect whether the tip of the surgical tool has entered the field of view. During this detection process, the determination of the DRR rendering range is not executed first; it needs to wait until the tip of the surgical tool enters the field of view of the imaging component before it is executed.

[0146] In this embodiment, when surgical tools (such as pedicle screwdrivers with guides, bone drills, puncture needles, etc.) enter the field of view of the imaging component, the image fusion device 200 can automatically trigger or be selected by the operator to "follow-up DRR dynamic generation function" and then start to perform the operation of dynamically determining the DRR rendering range.

[0147] In the above embodiments, the image fusion device can adaptively switch the DRR rendering range determination strategy according to whether the surgical tool enters the field of view, so that the DRR image accurately covers the area where the tool is located when the surgical tool enters the field of view, thereby ensuring the perception of the positional relationship between the tool and the anatomical structure in the fused image.

[0148] In some embodiments, the annotation information includes a first type of annotation information and a second type of annotation information. The real-time DRR generation module 230 includes a rendering submodule 235. During the process of the real-time DRR generation module 230 generating a DRR image to be fused corresponding to both camera imaging parameters and real-time extrinsic parameters based on the DRR rendering range and target 3D volume data, the real-time DRR generation module 230... The rendering submodule 235 is used to extract partial 3D volume data corresponding to the DRR rendering range from the target 3D volume data; extract the first type of 3D volume data corresponding to the first type of annotation information from the partial 3D volume data; and render the first type of 3D volume data to generate a first initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters. The rendering submodule 235 is also used to extract the second type of three-dimensional volume data corresponding to the second type of annotation information in the target three-dimensional volume data; and to render the second type of three-dimensional volume data to generate a second initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters; wherein, the area outside the projection corresponding to the second type of three-dimensional volume data in the second initial DRR sub-image is transparent; The first initial DRR sub-image and the second initial DRR sub-image together constitute the DRR image to be fused.

[0149] In this embodiment, during surgical intervention, some anatomical information needs to be continuously displayed for global reference, while other information needs to be dynamically updated as the surgical tool moves to focus on the current working area. Therefore, the annotation information can include multiple types, such as the first type of annotation information and the second type of annotation information corresponding to the two types of information mentioned above.

[0150] For example, the 3D volume data corresponding to the first type of annotation information is extracted and rendered within the DRR rendering range to achieve dynamic updating of the work area information; The 3D volume data corresponding to the second type of annotation information is directly extracted and rendered from the target 3D volume data to achieve continuous presentation of global reference information.

[0151] For example, the first type can be an extended anatomical structure (such as the spinal cord or nerve roots) that only shows the local area falling within the pyramidal region; the second type can be a fixed target (such as a designated lesion) that is always shown and is not limited by the pyramidal region.

[0152] In this embodiment, for the first type of annotation information, the device can perform DRR operation as described above to obtain the first initial DRR sub-image.

[0153] In this embodiment, the spatial coordinates of the second type of annotation information can be obtained due to its independence. Therefore, the device can perform DRR operations only on this part of the information to obtain the second initial DRR sub-image.

[0154] The first initial DRR sub-image and the second initial DRR sub-image both have the same size as the real-time image, but the second initial DRR sub-image only contains the projection information of the second type of annotation information, while the rest is transparent.

[0155] In this embodiment, the subsequent image fusion and overlay module 240 needs to fuse the real-time image, the first initial DRR sub-image, and the second initial DRR sub-image in the real-time video to obtain a fused image.

[0156] Therefore, the device can output prompts for the second type of annotation information that needs to be displayed frequently on the screen, as well as prompts for the first type of annotation information that limits the different depths and shapes according to the position of the surgical tool tip, so that the operator can observe them.

[0157] In the above embodiments, the image fusion device can realize the layered and differentiated presentation and independent distinction of different types of annotation information in the fused image.

[0158] In some embodiments, the image fusion apparatus 200 has a follow mode and a probe mode. When the image fusion apparatus 200 is in follow mode, the modules and sub-modules included in the image fusion apparatus 200 can perform various operations described in the above embodiments. When the image fusion apparatus 200 is in probe mode, the image fusion apparatus 200 may include: The CT image preprocessing module 210 is used to preprocess the original CT images to obtain target three-dimensional volume data with annotation information in the real coordinate system. The real-time pose analysis module 220 is used to receive the real-time pose data of the visual catheter output by the tracking device, and analyze the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter. The real-time DRR generation module 230 is used to dynamically determine the DRR rendering range based on the real-time tip position of the virtual probe in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters; the DRR rendering range is updated as the real-time tip position changes. The real-time DRR generation module 230 is also used to generate a DRR image to be fused that corresponds to both camera imaging parameters and real-time extrinsic parameters, based on the DRR rendering range and the target 3D volume data. The image fusion and overlay module 240 is used to fuse the real-time video acquired by the imaging component with the DRR image to be fused using a pixel-level image fusion algorithm to obtain a fused image.

[0159] In this embodiment, the virtual probe refers to a virtual object whose tip position has a spatial transformation relationship with the real-time extrinsic parameters of the imaging component.

[0160] In this embodiment, the follow mode refers to a mode in which the actual surgical tool is the target object, and the DRR rendering range is dynamically updated according to the real-time position change of the tip of the surgical tool in the real coordinate system, so as to provide the doctor with a real-time navigation fusion image corresponding to the current working area of ​​the surgical tool during the operation. The detection mode refers to a mode that uses a virtual probe as the target object, and the DRR rendering range is dynamically updated according to the real-time position change of the virtual probe tip in the real coordinate system, so as to provide doctors with anatomical structure prediction information at different depths during the preoperative or intraoperative planning stage.

[0161] As an optional implementation, the image fusion device 200 may also include a mode switching module for switching between a follow mode and a probe mode.

[0162] In this embodiment, the follow mode and the detection mode can be switched and adjusted based on the doctor's operation, or they can be switched and adjusted adaptively based on the actual situation, such as whether or not surgical tools are detected.

[0163] In the above embodiments, the image fusion device can pre-assess anatomical information at different depths during catheter advancement using a virtual probe in probe mode, providing doctors with preoperative risk prediction and path verification capabilities. This assists doctors in optimizing surgical strategies and reducing the risk of misoperation before actual operation. Simultaneously, the probe mode (targeting the virtual probe) and the follow mode (targeting the surgical tool) work together, allowing doctors to flexibly switch working modes according to the surgical stage, balancing the dual needs of preoperative planning and real-time intraoperative navigation, thereby further enhancing the system's clinical applicability.

[0164] In some embodiments, the image fusion device 200 has a detection mode; when in the detection mode, the CT image preprocessing module 210 is further configured to set position markers in the target three-dimensional volume data; The DRR real-time generation module 230 is also used to dynamically determine the DRR rendering range based on the real-time tip position of the virtual probe tip in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters. The real-time DRR generation module 230 includes a determination submodule 232. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the virtual probe in the real-world coordinate system, the camera imaging parameters of the imaging component, and real-time extrinsic parameters, the real-time DRR generation module 230... The determination submodule 232 is used to determine the axial projection spacing of the position marker points in the direction of the central ray of the DRR imaging; The determination submodule 232 is also used to determine the complete range of the target three-dimensional volume data as the DRR rendering range when the axial projection spacing is greater than the detection distance; The determination submodule 232 is also used to dynamically determine the DRR rendering range based on the detection distance, the camera imaging parameters of the imaging component, and the real-time external parameters when the axial projection spacing is less than or equal to the detection distance. The detection distance is the distance between the tip of the virtual probe and the DRR imaging X-ray source, and the tip of the virtual probe is always located on the central X-ray of the DRR imaging.

[0165] In this embodiment, the axial projection spacing refers to the distance between the projection point of the position marker on the DRR imaging center ray direction and the DRR imaging ray source.

[0166] For example, the aforementioned location marker can generally be set at a position slightly above the aforementioned marking information, while the detection distance has a preset initial value l.

[0167] Based on this, it is determined that submodule 232 can monitor the axial projection spacing in real time, and when the axial projection spacing is greater than the preset initial value, the complete range of the target three-dimensional volume data is determined as the DRR rendering range; When the axial projection spacing is less than or equal to the preset initial value, the DRR rendering range is dynamically determined based on the detection distance, the camera imaging parameters of the imaging component, and the real-time external parameters.

[0168] It should be understood that the determining submodule 232 actually determines whether the tip of the virtual probe touches the projection point of the aforementioned position mark in the direction of the camera's central ray. If it does not touch, it means the distance between the tip of the visual guide and the position mark in the direction of the camera's central ray is greater than a preset initial value; if it touches, it means the distance between the tip of the visual guide and the position mark in the direction of the camera's central ray is equal to the preset initial value; if it crosses, it means the distance between the tip of the visual guide and the position mark in the direction of the camera's central ray is less than the preset initial value.

[0169] Based on this, the device can render at a basic DRR rendering range at a distance to obtain a basic field of view; at the same time, as the conduit moves closer, the detection distance will begin to increase adaptively, thereby expanding the DRR rendering range in sync to obtain a wider field of view than the basic field of view.

[0170] In the above embodiments, the image fusion device can simulate the catheter advancement process with a virtual probe through a detection mode, thereby obtaining anatomical structure information at different depths in advance during the preoperative planning stage, providing doctors with a reference for path verification and risk prediction.

[0171] In some embodiments, the detection distance has a preset initial value. The determining submodule is further used to determine the tip position of the visible duct as the reference position when the axial projection spacing is equal to the preset initial value, and to determine the direction of the DRR imaging center ray as the reference direction. The determination submodule is specifically used to update the detection distance based on the projection component of the moving distance of the tip of the visual conduit in the reference direction when the axial projection spacing is less than or equal to the preset initial value. The movement distance is the distance between the real-time tip position and the reference position of the visual catheter tip in the real coordinate system.

[0172] In this embodiment, the formula used to update the detection distance can be: H = l + nd; Specifically, when the axial projection spacing is less than or equal to the preset initial value l, this formula is used to update the detection distance; The detection range can be H; The default initial value can be l; The preset adjustment ratio can be n (n is a constant greater than 1, such as 3); The projection component of the travel distance in the reference direction can be d.

[0173] In this embodiment, the reference position is the location of the catheter tip when the axial projection distance is equal to a preset initial value. Subsequently, as the visible catheter advances, the distance between its real-time tip position and the reference position in three-dimensional space is the travel distance.

[0174] In this embodiment, the visual guide tube may not advance entirely along the reference direction (i.e., the direction of the central ray of DRR imaging). Therefore, since the travel distance is a straight-line distance in three-dimensional space, and only the component along the reference direction contributes to updating the detection distance, it is necessary to project the travel distance onto the reference direction and update the detection distance using the projected component combined with a preset adjustment magnification.

[0175] In this embodiment, when the tip of the visual catheter enters the aforementioned range, as the visual catheter advances, richer anatomical information about the front of the visual catheter can be obtained with a small movement.

[0176] In the above embodiments, the image fusion device can adaptively expand the rendering range as the depth of the visual catheter advances, so that the DRR image always covers a sufficient field of view in front of the catheter tip, thereby providing doctors with a reference for predicting the anatomical structure in the direction of tool advancement.

[0177] In some embodiments, the image fusion and overlay module 240 includes: The distance acquisition submodule 241 is used to acquire the tip coordinates of the visual catheter in the real coordinate system; acquire the three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target three-dimensional volume data; and determine the shortest Euclidean distance between the tip of the visual catheter and the dangerous anatomical area based on the tip coordinates and the three-dimensional point cloud data. The weight determination submodule 242 is used to determine the fusion weights of the real-time video acquired by the imaging component and the DRR image to be fused based on the shortest Euclidean distance. The image fusion submodule 243 is used to fuse the real-time video and the DRR image to be fused according to their respective fusion weights, using a pixel-level image fusion algorithm to obtain a fused image. Among them, the smaller the shortest Euclidean distance, the greater the fusion weight of the real-time video and the smaller the fusion weight of the DRR image to be fused.

[0178] In this embodiment, the distance acquisition submodule 241 can perform nearest point distance calculation with the three-dimensional point cloud data of the marked dangerous anatomical area based on the real-time spatial coordinates of the tip of the visual catheter in the real coordinate system output by the navigation system, and calculate the shortest Euclidean distance between the tip of the visual catheter and the marked area in real time.

[0179] In this embodiment, the image fusion and overlay module 240 is equipped with a dynamic weight adjustment mechanism.

[0180] For example, when the distance between the tip of the visual catheter 100 and the marked area is less than 5mm, the image fusion and overlay module 240 can automatically increase the fusion weight of the real-time video to 0.8 and reduce the fusion weight of the DRR image to 0.2 to ensure that the doctor can clearly observe the surgical field tissue. When the distance between the tip of the visual catheter 100 and the marked area is greater than 20mm, the image fusion and overlay module 240 automatically increases the fusion weight of the DRR image to 0.7 and reduces the fusion weight of the real-time video to 0.3, so that doctors can observe the overall anatomical positional relationship.

[0181] In this embodiment, since the pixel size of the DRR image is the same as that of the real-time video, the two can be superimposed at the pixel level, and the marked area in the DRR image is accurately superimposed onto the corresponding position in the real-time video.

[0182] Based on this, in the final fused images, doctors can simultaneously observe real tissues and pre-planned anatomical structures (including markings of dangerous areas) in the same frame, without having to switch between multiple frames for comparison, reducing visual burden and improving the intuitiveness and precision of surgical operations.

[0183] In the above embodiments, the image fusion device can acquire the coordinates of the tip of the visual catheter in the real coordinate system in real time through the distance acquisition submodule, and combine it with the three-dimensional point cloud data of the dangerous anatomical area to accurately determine the shortest Euclidean distance between the tip of the visual catheter and the dangerous area, providing a reliable data basis for dynamic weight adjustment; the weight determination submodule adaptively adjusts the fusion weights of the real-time video and the DRR image to be fused according to the shortest Euclidean distance, so that the fusion weights change dynamically in real time with the risk level, realizing intelligent switching of image fusion strategy according to actual clinical needs; the image fusion submodule performs pixel-level fusion processing on the real-time video and the DRR image to be fused according to the adjusted fusion weights. When the tip of the visual catheter approaches the dangerous area, the display ratio of the real-time video is automatically increased to highlight the details of the real surgical field. At a safe distance, the display ratio of the DRR image is appropriately increased to facilitate overall anatomical observation, thereby taking into account the local clarity requirements during close operation and the global field of view requirements during long-distance navigation, effectively improving the clinical adaptability and safety of intraoperative image fusion.

[0184] In some embodiments, the image fusion apparatus 200 further includes: The distance warning module 260 is used to acquire the tip coordinates of the surgical tool in the real coordinate system. The surgical tool is used in conjunction with the visual catheter. It acquires three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target three-dimensional volume data. Based on the tip coordinates and the three-dimensional point cloud data, it determines the shortest Euclidean distance between the tip of the surgical tool and the dangerous anatomical area. Based on the shortest Euclidean distance and the preset multi-level warning thresholds, it determines the warning level. It generates warning prompt information based on the warning level. When the warning level is Level 1, the warning information includes static visual prompts in the first warning area of ​​the fused image, and the static visual prompts are in the first color. When the warning level is level 2, the warning information includes a dynamic visual cue in the second warning area of ​​the fused image, and the dynamic visual cue is in the second color; When the warning level is level three, the warning information includes audible and visual alarm signals and / or visual warning markers; the visual warning markers include a highlighted circle in a third color and a distance value marker. The highlighted circle in the third color is used to circle the relative position of the tip of the surgical instrument and the dangerous anatomical area, and the distance value marker is used to display the value of the shortest Euclidean distance.

[0185] In this embodiment, the distance warning module 260 can perform nearest point distance calculation with the three-dimensional point cloud data of the marked dangerous anatomical area based on the real-time spatial coordinates of the tip of the surgical tool (such as a puncture needle) in the real coordinate system output by the navigation system, and calculate the shortest Euclidean distance between the tip of the surgical tool and the marked area in real time.

[0186] In this embodiment, although both the distance warning module 260 and the distance acquisition submodule 241 can acquire the shortest Euclidean distance between the tip of the surgical instrument and the dangerous anatomical area, it should be understood that they are the same application of the same function in different modules, rather than mutual calls between modules. That is, although the distance warning module 260 and the distance acquisition submodule 241 have the same function, they belong to different modules, and each of these different modules works for its own purpose.

[0187] In this embodiment, three warning thresholds are set: a level one warning is triggered when the shortest Euclidean distance is in the range of 5mm to 10mm; a level two warning is triggered when the shortest Euclidean distance is in the range of 2mm to 5mm; and a level three warning is triggered when the shortest Euclidean distance is less than 2mm.

[0188] In this embodiment, when a Level 1 warning is triggered, a blue static warning bar is displayed in the periphery of the fused image (i.e., the first warning warning area).

[0189] In this embodiment, when a level 2 warning is triggered, a yellow dynamic flashing frame is displayed in the four edges / four inner frames of the fused image (i.e., the second warning prompt area, where the second warning prompt area and the first warning prompt area may be the same).

[0190] In this embodiment, when a Level 3 warning is triggered, the real-time navigation system simultaneously issues an audible and visual alarm signal and highlights the relative position of the surgical tool tip and the dangerous anatomical area in red on the fused image.

[0191] This implementation method allows doctors to perform small-scale, shallow-depth operations even at low-risk levels by setting up a multi-level early warning mechanism. As the catheter approaches the danger zone, the warning level gradually increases to remind doctors to adjust the operation direction in a timely manner and avoid damaging nerve tissue.

[0192] In the above embodiments, the image fusion device can obtain the position coordinates of the surgical tool tip in the real coordinate system in real time through the distance warning module, and calculate the shortest Euclidean distance between the surgical tool tip and the danger area by combining the three-dimensional point cloud data of the dangerous anatomical area, so as to realize real-time distance measurement during surgery. Then, through the preset multi-level warning threshold, different levels of warning prompts are automatically matched according to the distance. For example, a blue static prompt bar reminds the doctor to pay attention to the operation rhythm, a yellow dynamic flashing box warns the doctor to operate with caution, and a three-level sound and light alarm, a red high-brightness circle and distance value display intuitively indicate the dangerous location and the remaining safe distance, thereby effectively reducing the risk of misoperation and ensuring surgical safety.

[0193] like Figure 6 As shown, this application provides an electronic device 300, which serves as the physical device of an image fusion apparatus 200. The electronic device 300 includes a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanism (not shown). The memory 302 stores a computer program that can be executed by the processor 301. When the electronic device is running, the processor 301 executes the computer program to realize the functions of each module in the image fusion apparatus.

[0194] like Figure 7 As shown, some embodiments of this application provide a real-time navigation system based on image fusion, including a CT image acquisition device 400, a visual catheter 100, a tracking device 500, a display device 600, and... Figure 2 The image fusion device 200 shown, wherein, CT image acquisition device 400, used to acquire raw CT images to be fused; The visual conduit 100 is used to acquire real-time video through an imaging component set at the tip of the visual conduit, and to navigate and track the visual conduit 100 through a navigation and tracking component set at the tail end of the visual conduit. The tracking device 500 is used in conjunction with the navigation tracking component to acquire real-time pose data of the visual guide tube 100 in the real coordinate system; The image fusion device 200 is used to preprocess the original CT images to obtain target 3D volume data with annotation information in the real coordinate system; analyze the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter 100; dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool tip in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters; generate a DRR image to be fused corresponding to both the camera imaging parameters and the real-time extrinsic parameters based on the DRR rendering range and the target 3D volume data; and fuse the real-time video with the DRR image to be fused to obtain a fused image; wherein, the DRR rendering range is updated as the real-time tip position changes. Display device 600 is used to output fused images.

[0195] In this embodiment, the visual catheter 100 is Figure 1 The specific structure of the visible catheter shown will not be described in detail here.

[0196] In this embodiment, the image fusion device 200 is Figure 2 The image fusion device shown is an image fusion apparatus 200. Specifically, the image fusion apparatus 200 includes: a CT image preprocessing module 210; a real-time pose analysis module 220; a real-time DRR generation module 230; a detection submodule 231; a determination submodule 232; a construction submodule 233; a retrieval submodule 234; a rendering submodule 235; an image fusion and overlay module 240; a distance acquisition submodule 241; a weight determination submodule 242; and an image fusion submodule 243. Therefore, its specific functions and applications will not be elaborated upon here.

[0197] In this embodiment, the CT image acquisition device 400 is capable of acquiring the patient's original CT images. These original CT images can be used for surgical planning, registration, and data annotation of various tissues and organs.

[0198] In this embodiment, the image fusion device 200 and the display device 600 can be mounted on a trolley so that the trolley can carry and move these devices.

[0199] In this embodiment, the display device 600 is specifically used to output the fused images and interactive interface during the navigation process to the doctor. The display device 600 can be a common LCD display, AR glasses, or any device that can be used to display the fused images and interactive interface during the navigation process.

[0200] As an optional implementation, the real-time navigation system may also include a locator installed near the patient's lesion to provide a spatial positioning reference for the navigation system, thereby establishing a coordinate system for the surgical area (i.e., the real coordinate system in this application), aligning the preoperative CT image space with the actual surgical space during the registration process, and simultaneously tracking slight patient displacements in real time and feeding them back to the navigation system to ensure the accuracy of the tracking path of the visual catheter 100.

[0201] The visual catheter 100 is directed towards the area where the locator is located. The locator is used not only to synchronize the coordinate system, but also to provide directional guidance for the advancement of the catheter.

[0202] For example, the locator can be a reference frame, which is fixed near the patient's lesion during surgery to cooperate with the tracking device to complete spatial positioning and real-time tracking functions.

[0203] As an alternative implementation, the real-time navigation system may also include a guide and a calibration device.

[0204] The guide can be a rigid sleeve or clamp with several passive infrared reflection tracking balls installed on it, which are used to physically bind to ordinary surgical tools (such as puncture needles) so that the surgical tools can be identified by the tracking device. The calibration device can be a positioning structure (such as a conical hole or groove) adapted to the guide and surgical tool assembly, used to record the spatial offset of the surgical tool tip relative to the guide and inform the real-time navigation system (specifically, the image fusion device 200) of the offset, so that the real-time navigation system can calculate the precise position of the surgical tool tip in real time by tracking the guide.

[0205] For example, in use, the guide can be physically bound to the puncture needle first; then the bound assembly can be inserted into the known structure of the calibration device; at this time, due to the special design of the calibration device, it can directly determine the spatial offset of the puncture needle tip relative to the guide and inform the real-time navigation system to complete the registration; thereafter, the calibration device is removed, and the real-time navigation system can calculate the precise position of the puncture needle tip in real time by tracking the position of the guide in real time and combining the spatial offset, so as to achieve real-time tracking of the puncture needle.

[0206] Based on the above implementation methods, this real-time navigation system can be widely used in various spinal surgical procedures that require precise operation, such as spinal endoscopic surgery, spinal puncture surgery, and spinal internal fixation implantation.

[0207] In the above embodiments, the real-time navigation system can achieve the coordinated operation of the CT image acquisition device, visual catheter, tracking device and image fusion device. Driven by the tip of the surgical tool, it can fuse and display the labeled DRR image and real-time video in the same frame, allowing doctors to directly perceive the spatial relationship between the surgical tool and the anatomical structure from the fused image. At the same time, since the generation of DRR image only requires calculation on the three-dimensional volume data within the dynamically determined rendering range, it can also effectively reduce the computational overhead per frame.

[0208] In some embodiments, the real-time navigation system further includes an operating device, wherein... An operating device is used to receive operating instructions input by the user; the operating instructions include at least one of the following: marking instructions for dangerous anatomical areas in the virtual three-dimensional model, path planning instructions for the movement of the visual catheter, instructions for switching modes during the operation, and warning confirmation.

[0209] In this embodiment, the annotation instruction refers to the operation instruction that the doctor uses the operating device to delineate or mark dangerous anatomical areas such as lesions, spinal cord, and nerves on the virtual three-dimensional model. After receiving the instruction, the annotation submodule 213 performs annotation processing on the virtual three-dimensional model accordingly.

[0210] In this embodiment, the path planning instruction refers to the operation instruction given by the doctor through the operating device to set the trajectory of the visual catheter from the entry point to the lesion. After receiving the instruction, the system generates surgical path planning data for intraoperative navigation guidance.

[0211] In this embodiment, the intraoperative mode switching command refers to the operation command given by the doctor to switch the system's working mode during surgery via the operating device. For example, switching from navigation mode to warning mode, switching from fusion display mode to single image mode, adjusting the dynamic weight switch, etc.

[0212] In this embodiment, the warning confirmation command refers to the operation command by which the doctor confirms or cancels the warning prompt issued by the system through the operating device. For example, after a warning is triggered, the doctor views the warning information and clicks to confirm, and the system records that the warning has been responded to by the doctor.

[0213] In the above embodiments, the real-time navigation system can provide doctors with an intraoperative interactive entry through the operating device, thereby supporting operations such as marking dangerous anatomical areas in the virtual three-dimensional model, planning the movement path of the visual catheter, switching the intraoperative working mode, and confirming the warning information. This allows doctors to flexibly control the navigation process and parameters according to the actual needs of the surgery, thereby enhancing the operability and clinical applicability of the system and improving surgical efficiency.

[0214] In some embodiments, the image fusion-based real-time navigation system may be implemented according to the following process when put into use.

[0215] (1) Pre-calibration of the camera The visual conduit 100 is placed in front of the custom checkerboard calibration plate, and the miniature camera 121 is run to acquire calibration images. The camera imaging parameters of the miniature camera 121 are obtained by using the Zhang Zhengyou calibration method through the camera calibration module.

[0216] (2) CT image acquisition and preprocessing The patient lies prone on the operating table, the reference frame is fixed near the lesion, and the registration aid is placed in the lesion area. The CT image acquisition device 400 acquires CT image data of the lesion area containing the registration aid. The CT image data is imported into the three-dimensional reconstruction submodule 211 for three-dimensional reconstruction, forming a virtual three-dimensional model corresponding to the real patient's anatomical structure. The dangerous anatomical areas such as the spinal cord, bilateral nerve roots, and adjacent major blood vessels are three-dimensionally annotated, generating a CT volume data file with annotation information.

[0217] (3) Navigation system registration The positions of multiple metal balls in the virtual space of the registration auxiliary block in the virtual 3D model are manually selected. The positions of the multiple metal balls in the real space are obtained through the tracking device. The coordinate transformation relationship between the virtual space and the real space is calculated by the registration algorithm through the calibration submodule 212 to realize the system registration.

[0218] (4) Visual catheter placement and intraoperative tracking The visual catheter 100 is inserted into the patient's body along the planned path. The imaging component 120 acquires images of the tissue around the catheter tip in real time, while the tracking device 500 acquires the spatial pose data of the visual catheter 100 in real time and transmits it to the image fusion device 200.

[0219] (5) Real-time camera pose analysis The real-time pose analysis module 220 calculates the current pose of the micro-camera 121 in the real coordinate system (i.e., the real-time extrinsic parameters of the micro-camera 121) based on the real-time pose data of the visual guide tube 100 output by the tracking device 500 and the pose transformation relationship of the micro-camera 121 relative to the navigation tracking component 130.

[0220] (6) DRR image generation The real-time DRR generation module 230 uses the current pose and camera imaging parameters of the miniature camera 121 as virtual projection parameters, calls the accelerated ray projection DRR algorithm, renders the part of the three-dimensional volume data corresponding to the DRR rendering range in the target three-dimensional volume data, generates a DRR image corresponding to the camera imaging parameters and real-time extrinsic parameters, and overlays and embeds annotation layers of different depths into the DRR image to obtain a DRR image to be fused with annotation information.

[0221] (7) Image fusion The image fusion and overlay module 240 uses an alpha channel weighted fusion algorithm to overlay the real-time video acquired by the visual conduit 100 with the DRR image at the pixel level to obtain a fused image; the initial fusion weights are each set to 0.5, and the weights are dynamically adjusted according to the distance between the tip of the visual conduit 100 and the labeled area.

[0222] (8) Distance warning The distance warning module 260 calculates the shortest Euclidean distance between the tip of the surgical tool and the nearest marked area in real time, triggers corresponding level warning prompts according to preset multi-level warning thresholds, and displays the warning information in the fused image.

[0223] (9) Export of surgical records After the surgery is completed, the system automatically exports the fused images and early warning records of the entire surgical process.

[0224] like Figure 8 As shown, some embodiments of this application provide a real-time navigation method based on image fusion. It should be understood that this method is similar to... Figure 7 The real-time navigation system shown can be executed by the real-time navigation system. The specific functions and effects of the method can be found in the description of the system embodiment above. To avoid repetition, detailed descriptions are omitted here.

[0225] In this embodiment, the image fusion-based real-time navigation method is applied to an image fusion-based real-time navigation system. The real-time navigation system includes a CT image acquisition device, a visual catheter, a tracking device, a display device, and an image fusion device. The method includes: S810, CT image acquisition device acquires the original CT images to be fused.

[0226] S820: The visual catheter acquires real-time video through an imaging component at the tip of the visual catheter, and navigates and tracks the visual catheter through a navigation and tracking component at the tail end of the visual catheter.

[0227] In this embodiment, the visual catheter includes a catheter body, an imaging component, and a navigation and tracking component, wherein the catheter body is a rigid structure including a tip and a tail. The imaging component is fixedly mounted at the tip of the catheter body to acquire real-time video; The navigation and tracking component is fixedly installed at the tail end of the duct body and is used to navigate and track the visible duct so that the tracking device can obtain real-time pose data of the visible duct in the real coordinate system.

[0228] In this embodiment, the imaging component includes a miniature camera, the front shooting direction of the miniature camera and the upward direction of the miniature camera form a first plane; the tracking direction of the navigation and tracking component and the upward direction of the navigation and tracking component form a second plane; the first plane and the second plane coincide.

[0229] The S830, tracking device, and navigation tracking components work together to acquire real-time pose data of the visual duct in the real coordinate system.

[0230] The S840 image fusion device preprocesses the original CT images to obtain target 3D volume data with annotation information in the real coordinate system; it analyzes the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter; it dynamically determines the DRR rendering range based on the real-time tip position of the surgical tool tip in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters; it generates a DRR image to be fused based on the DRR rendering range and the target 3D volume data, corresponding to both the camera imaging parameters and the real-time extrinsic parameters; and it fuses the real-time video with the DRR image to be fused to obtain a fused image; wherein, the DRR rendering range is updated as the real-time tip position changes.

[0231] As an optional implementation, the DRR rendering range is dynamically determined based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and real-time extrinsic parameters, including: Based on the camera imaging parameters and real-time extrinsic parameters of the imaging component, determine the axial projection distance of the tip of the surgical instrument in the direction of the central ray of the DRR imaging. Determine the axial target position based on the axial projection distance; A cone space is constructed with the DRR imaging ray source as the vertex, the DRR imaging center ray direction as the axis, the DRR imaging field of view angle as the cone angle, and the perpendicular plane of the axis at the target position along the axis as the base. The DRR rendering range is limited by the cone space.

[0232] As an optional implementation, the method further includes: When the axial projection distance is the same as the historical distance and the real-time pose data is the same as the historical pose data, the image fusion device retrieves the historical DRR image that is associated with the historical distance and the historical pose data, so that the image fusion and overlay module can use the historical DRR image as the DRR image to be fused for fusion processing. Among them, historical distance, historical pose data, and historical DRR images are stored together when generating the DRR image to be fused.

[0233] As an optional implementation, the DRR rendering range is dynamically determined based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and real-time extrinsic parameters, including: Detect whether surgical instruments have entered the field of view of the imaging component; When the surgical tool enters the field of view, the DRR rendering range is dynamically determined based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters. When the surgical tools are not within the field of view, the complete range of the target 3D volume data is defined as the DRR rendering range.

[0234] As an optional implementation, the annotation information includes a first type of annotation information and a second type of annotation information. Based on the DRR rendering range and the target 3D volume data, a DRR image to be fused, corresponding to both camera imaging parameters and real-time extrinsic parameters, is generated, including: Extract partial 3D volume data corresponding to the DRR rendering range from the target 3D volume data; extract first type 3D volume data corresponding to the first type of annotation information from the partial 3D volume data; and render the first type 3D volume data to generate a first initial DRR sub-image corresponding to both camera imaging parameters and real-time extrinsic parameters. Extract the second type of 3D volume data corresponding to the second type of annotation information from the target 3D volume data; and render the second type of 3D volume data to generate a second initial DRR sub-image corresponding to both camera imaging parameters and real-time extrinsic parameters; wherein, in the second initial DRR sub-image, the area outside the projection corresponding to the second type of 3D volume data is transparent; The first initial DRR sub-image and the second initial DRR sub-image together constitute the DRR image to be fused.

[0235] As an optional implementation, when the image fusion-based real-time navigation method is in the detection mode, the method further includes: The image fusion device sets position markers in the target's three-dimensional volume data; The image fusion device determines the axial projection spacing of the location markers in the direction of the central ray of the DRR imaging; When the axial projection spacing is greater than the detection distance, the image fusion device determines the complete range of the target's three-dimensional volume data as the DRR rendering range; When the axial projection spacing is less than or equal to the detection distance, the image fusion device dynamically determines the DRR rendering range based on the detection distance, the camera imaging parameters of the imaging components, and real-time external parameters. The detection distance is the distance between the tip of the virtual probe and the DRR imaging X-ray source, and the tip of the virtual probe is always located on the central X-ray of the DRR imaging.

[0236] As a further optional implementation, the detection distance has a preset initial value, and the method further includes: When the axial projection spacing is equal to the preset initial value, the image fusion device determines the tip position of the visible conduit as the reference position and the direction of the DRR imaging center ray as the reference direction. When the axial projection spacing is less than or equal to the preset initial value, the image fusion device updates the detection distance according to the preset adjustment magnification and the projection component of the moving distance of the tip of the visual guide in the reference direction. The movement distance is the distance between the real-time tip position and the reference position of the visual catheter tip in the real coordinate system.

[0237] As an optional implementation, the method further includes: The image fusion device acquires the tip coordinates of the visual conduit in the real coordinate system; The image fusion device acquires three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target's three-dimensional volume data; The image fusion device determines the shortest Euclidean distance between the tip of the visual catheter and the dangerous anatomical area based on the tip coordinates and three-dimensional point cloud data. The image fusion device determines the fusion weights of the real-time video acquired by the imaging components and the DRR image to be fused based on the shortest Euclidean distance. The image fusion device uses a pixel-level image fusion algorithm to fuse the real-time video and the DRR image to be fused according to their respective fusion weights, and obtains a fused image. Among them, the smaller the shortest Euclidean distance, the greater the fusion weight of the real-time video and the smaller the fusion weight of the DRR image to be fused.

[0238] As an optional implementation, the method further includes: The image fusion device acquires the tip coordinates of the surgical instrument in the real coordinate system, and the surgical instrument is used in conjunction with the visual catheter; The image fusion device acquires three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target's three-dimensional volume data; The image fusion device determines the shortest Euclidean distance between the tip of the surgical instrument and the dangerous anatomical area based on the tip coordinates and three-dimensional point cloud data. The image fusion device determines the warning level based on the shortest Euclidean distance and preset multi-level warning thresholds; and generates warning prompt information based on the warning level. When the warning level is Level 1, the warning information includes static visual prompts in the first warning area of ​​the fused image, and the static visual prompts are in the first color. When the warning level is level 2, the warning information includes a dynamic visual cue in the second warning area of ​​the fused image, and the dynamic visual cue is in the second color; When the warning level is level three, the warning information includes audible and visual alarm signals and / or visual warning markers; the visual warning markers include a highlighted circle in a third color and a distance value marker. The highlighted circle in the third color is used to circle the relative position of the tip of the surgical instrument and the dangerous anatomical area, and the distance value marker is used to display the value of the shortest Euclidean distance.

[0239] S850, display device outputs fused image.

[0240] In this embodiment, the fused image is a combination of real-time video (real tissue image), DRR image (semi-transparent bone outline and third color highlighted area), and warning prompts (first color prompt bar / second color flashing box / third color circle and distance value).

[0241] In this embodiment, the method can achieve seamless integration of real field of view and virtual anatomical information in the same frame.

[0242] For example, the main image of the fused image is the real-time intraoperative video captured by the miniature camera 121 at the tip of the visual catheter 100, that is, the real tissue image (such as muscles, bones, nerves, etc.) seen at the catheter tip. This image serves as a background image and is continuously displayed after the catheter enters the body without the need to switch images; the doctor sees only the tissue currently being viewed by the catheter tip.

[0243] Then, a DRR image perfectly corresponding to the current viewpoint is overlaid on the live video. This DRR image is derived from preoperative CT 3D volumetric data and includes the 3D structure of dangerous anatomical areas such as bones, lesion regions, surrounding spinal cord, and nerves. It is semi-transparently overlaid on the corresponding locations in the live video. Among them, the dangerous anatomical areas marked preoperatively are displayed in a highlighted color (such as the third color), allowing doctors to intuitively see which locations in the real-time tissue image correspond to the preoperatively planned lesions or dangerous areas.

[0244] Therefore, doctors can simultaneously observe the following in the fused images: Actual intraoperative view (the actual tissue directly in front of the catheter tip); Virtual anatomical structures (semi-transparent outlines of bones, lesions, spinal cord, and nerves). Dangerous areas are marked (highlighted with a third color).

[0245] In the above embodiments, the real-time navigation method, through the entire process of CT image acquisition, real-time video acquisition and pose tracking of the visual catheter, pose acquisition of the tracking device, image fusion processing, and output of the fused image, can fuse DRR images with labeled information and real-time video in the same frame, driven by the tip of the surgical tool. This allows the doctor to directly perceive the spatial relationship between the surgical tool and the anatomical structure from the fused image. At the same time, since the generation of DRR images only requires calculation on the three-dimensional volume data within a dynamically determined rendering range, it can also reduce the computational overhead per frame.

[0246] This application provides a computer-readable storage medium storing a computer program, which is executed by a plurality of processors, each of which executes the corresponding method steps in any of the aforementioned optional implementations.

[0247] The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0248] This application provides a computer program product, which includes a computer program that is run by a plurality of processors, each of which executes the corresponding method steps in any of the aforementioned optional implementations.

[0249] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features therein. These modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application, and they should all be covered within the scope of the claims and specification of this application. In particular, as long as there is no conflict, the various technical features mentioned in the embodiments can be combined in any way. This application is not limited to the specific embodiments disclosed herein, but includes all technical solutions falling within the scope of the claims.

Claims

1. A visual catheter, characterized in that, It includes a catheter body, an imaging component, and a navigation and tracking component, wherein the catheter body is a rigid structure including a tip and a tail. The imaging component is fixedly mounted on the tip of the catheter body for acquiring real-time video; The navigation and tracking component is fixedly installed at the tail end of the duct body and is used to navigate and track the visual duct in order to obtain the real-time pose data of the visual duct in the real coordinate system. The real-time pose data is used to determine the real-time extrinsic parameters of the imaging component, and the real-time extrinsic parameters represent the pose of the imaging component in the real coordinate system. The real-time extrinsic parameters are used to dynamically determine the DRR rendering range by combining the real-time tip position of the surgical tool in the real coordinate system and the camera imaging parameters of the imaging component; the DRR rendering range is updated as the real-time tip position changes. The DRR rendering range is used to combine the target 3D volume data with annotation information in the real coordinate system to generate a DRR image to be fused that corresponds to both the camera imaging parameters and the real-time extrinsic parameters. The DRR image to be fused is used to perform fusion processing with the real-time video to obtain a fused image.

2. The visual catheter according to claim 1, characterized in that, The imaging component includes a miniature camera, and the forward shooting direction of the miniature camera and the upward direction of the miniature camera form a first plane; The tracking direction of the navigation tracking component and the upward direction of the navigation tracking component form a second plane; The first plane coincides with the second plane.

3. An image fusion device, characterized in that, include: The CT image preprocessing module is used to preprocess the original CT images to obtain target three-dimensional volume data with annotation information in the real coordinate system. The real-time pose analysis module is used to receive the real-time pose data of the visual catheter output by the tracking device, and to analyze the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter. The real-time DRR generation module is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters; the DRR rendering range is updated as the real-time tip position changes. The real-time DRR generation module is also used to generate a DRR image to be fused that corresponds to both the camera imaging parameters and the real-time extrinsic parameters, based on the DRR rendering range and the target 3D volume data. The image fusion and overlay module is used to perform pixel-level image fusion algorithm to fuse the real-time video acquired by the imaging component with the DRR image to be fused to obtain a fused image.

4. The image fusion apparatus according to claim 3, characterized in that, The real-time DRR generation module includes a determination submodule and a construction submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The determining submodule is used to determine the axial projection distance of the tip of the surgical tool in the DRR imaging center ray direction based on the camera imaging parameters of the imaging component and the real-time extrinsic parameters. The determining submodule is further configured to determine the axial target position based on the axial projection distance; The construction submodule is used to construct a cone space with the DRR imaging ray source as the vertex, the DRR imaging center ray direction as the axis, the DRR imaging field of view as the cone angle, and the vertical plane of the axis at the target position of the axial direction as the bottom surface. The determining submodule is also used to limit the DRR rendering range based on the cone space.

5. The image fusion apparatus according to claim 4, characterized in that, The real-time DRR generation module also includes: The retrieval submodule is used to retrieve the historical DRR image associated with the historical distance and the historical pose data when the axial projection distance is the same and the real-time pose data is the same, so that the image fusion and overlay module can use the historical DRR image as the DRR image to be fused for fusion processing. The historical distance, the historical pose data, and the historical DRR image are stored together when generating the DRR image to be fused.

6. The image fusion apparatus according to claim 3, characterized in that, The real-time DRR generation module includes a detection submodule and a determination submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the surgical tool in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The detection submodule is used to detect whether the surgical tool has entered the field of view of the imaging component; The determination submodule is used to dynamically determine the DRR rendering range based on the real-time tip position of the surgical tool in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters when the surgical tool enters the field of view. The determining submodule is further configured to determine the complete range of the target three-dimensional volume data as the DRR rendering range when the surgical tool does not enter the field of view.

7. The image fusion apparatus according to claim 6, characterized in that, The annotation information includes a first type of annotation information and a second type of annotation information. The real-time DRR generation module includes a rendering submodule. During the process of the real-time DRR generation module generating a DRR image to be fused based on the DRR rendering range and the target 3D volume data, the DRR image corresponds to both the camera imaging parameters and the real-time extrinsic parameters. The rendering submodule is used to extract a portion of the three-dimensional volume data corresponding to the DRR rendering range from the target three-dimensional volume data. Extract the first type of three-dimensional volume data corresponding to the first type of annotation information from the partial three-dimensional volume data; and render the first type of three-dimensional volume data to generate a first initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters; The rendering submodule is further configured to extract the second type of three-dimensional volume data corresponding to the second type of annotation information from the target three-dimensional volume data; and to render the second type of three-dimensional volume data to generate a second initial DRR sub-image corresponding to both the camera imaging parameters and the real-time extrinsic parameters; wherein, in the second initial DRR sub-image, the area outside the projection corresponding to the second type of three-dimensional volume data is transparent; The first initial DRR sub-image and the second initial DRR sub-image together constitute the DRR image to be fused.

8. The image fusion apparatus according to claim 3, characterized in that, The image fusion device has a detection mode; when in the detection mode, the CT image preprocessing module is also used to set position markers in the target three-dimensional volume data; The real-time DRR generation module is also used to dynamically determine the DRR rendering range based on the real-time tip position of the virtual probe tip in the real coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters. The real-time DRR generation module includes a determination submodule. During the process of dynamically determining the DRR rendering range based on the real-time tip position of the virtual probe in the real-world coordinate system, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters, the real-time DRR generation module... The determining submodule is used to determine the axial projection spacing of the position marker points in the ray direction of the DRR imaging center; The determining submodule is also used to determine the complete range of the target three-dimensional volume data as the DRR rendering range when the axial projection spacing is greater than the detection distance; The determining submodule is further configured to dynamically determine the DRR rendering range based on the detection distance, the camera imaging parameters of the imaging component, and the real-time extrinsic parameters when the axial projection spacing is less than or equal to the detection distance. The detection distance is the distance between the tip of the virtual probe and the DRR imaging X-ray source, and the tip of the virtual probe is always located on the central X-ray of the DRR imaging.

9. The image fusion apparatus according to claim 8, characterized in that, The detection distance has a preset initial value. The determining submodule is specifically used to determine the tip position of the visual conduit as the reference position when the axial projection spacing is equal to the preset initial value, and to determine the direction of the DRR imaging center ray as the reference direction. The determining submodule is further configured to update the detection distance based on the preset adjustment magnification and the projection component of the movement distance of the tip of the visual conduit in the reference direction when the axial projection spacing is less than or equal to the preset initial value; The moving distance is the distance between the real-time tip position of the visual conduit in the real coordinate system and the reference position.

10. The image fusion apparatus according to any one of claims 3-9, characterized in that, The image fusion and overlay module includes: The distance acquisition submodule is used to acquire the tip coordinates of the visual catheter in the real coordinate system; acquire the three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target three-dimensional volume data; and determine the shortest Euclidean distance between the tip of the visual catheter and the dangerous anatomical area based on the tip coordinates and the three-dimensional point cloud data. The weight determination submodule is used to determine the fusion weights of the real-time video acquired by the imaging component and the DRR image to be fused based on the shortest Euclidean distance. The image fusion submodule is used to fuse the real-time video and the DRR image to be fused according to their respective fusion weights, using a pixel-level image fusion algorithm to obtain a fused image; The smaller the shortest Euclidean distance, the greater the fusion weight of the real-time video and the smaller the fusion weight of the DRR image to be fused.

11. The image fusion apparatus according to any one of claims 3-9, characterized in that, The image fusion device also includes: The distance warning module is used to acquire the tip coordinates of the surgical tool in the real coordinate system; acquire three-dimensional point cloud data of the dangerous anatomical area in the real coordinate system based on the target three-dimensional volume data; determine the shortest Euclidean distance between the tip of the surgical tool and the dangerous anatomical area based on the tip coordinates and the three-dimensional point cloud data; determine the warning level based on the shortest Euclidean distance and the preset multi-level warning threshold; and generate a warning prompt message based on the warning level. Wherein, when the warning level is Level 1, the warning information includes a static visual prompt in the first warning prompt area of ​​the fused image, and the static visual prompt is in a first color; When the warning level is level two, the warning information includes a dynamic visual prompt in the second warning prompt area of ​​the fused image, and the dynamic visual prompt is in a second color; When the warning level is level three, the warning information includes an audible and visual alarm signal and / or a visual warning mark; the visual warning mark includes a highlighted delineator of a third color and a distance value mark, the highlighted delineator of the third color is used to delineate the relative position of the tip of the surgical instrument and the dangerous anatomical area, and the distance value mark is used to display the value of the shortest Euclidean distance.

12. A real-time navigation system based on image fusion, characterized in that, Includes a CT image acquisition device, a visual catheter, a tracking device, a display device, and the image fusion device according to any one of claims 3-11, wherein, The CT image acquisition device is used to acquire the original CT images to be fused. The visual conduit is used to acquire real-time video through an imaging component set at the tip of the visual conduit, and to navigate and track the visual conduit through a navigation and tracking component set at the tail end of the visual conduit. The tracking device is used in conjunction with the navigation tracking component to obtain the real-time pose data of the visual guide tube in the real coordinate system; The display device is used to output the fused image.

13. The real-time navigation system based on image fusion according to claim 12, characterized in that, The real-time navigation system also includes an operating device, wherein... The operating device is used to receive operating instructions input by the user; the operating instructions include at least one of the following: marking instructions for dangerous anatomical areas in the virtual three-dimensional model, path planning instructions for the movement of the visual catheter, instructions for switching modes during surgery, and warning confirmation.

14. A real-time navigation method based on image fusion, characterized in that, The image fusion-based real-time navigation method is applied to an image fusion-based real-time navigation system, which includes a CT image acquisition device, a visual catheter, a tracking device, a display device, and an image fusion device. The method includes: The CT image acquisition device acquires the original CT images to be fused; The visual catheter acquires real-time video through an imaging component at its tip and navigates and tracks the visual catheter through a navigation and tracking component at its tail. The tracking device works in conjunction with the navigation tracking component to obtain the real-time pose data of the visual duct in the real coordinate system; The image fusion device preprocesses the original CT images to obtain target 3D volume data with annotation information in the real coordinate system; it analyzes the real-time pose data to obtain the real-time extrinsic parameters of the imaging components in the visual catheter; it dynamically determines the DRR rendering range based on the real-time tip position of the surgical tool tip in the real coordinate system, the camera imaging parameters of the imaging components, and the real-time extrinsic parameters; it generates a DRR image to be fused based on the DRR rendering range and the target 3D volume data, corresponding to both the camera imaging parameters and the real-time extrinsic parameters; and it fuses the real-time video with the DRR image to be fused to obtain a fused image; wherein, the DRR rendering range is updated as the real-time tip position changes. The display device outputs the fused image.

15. A readable storage medium, characterized in that, The readable storage medium stores a computer program that is executed by a plurality of processors, each of which performs a corresponding step in the real-time navigation method based on image fusion as described in claim 14.

16. A computer program product, characterized in that, The computer program product includes a computer program that is run by a plurality of processors, each of which executes a corresponding step in the image fusion-based real-time navigation method of claim 14.

Images