Image fusion method and device applied to vascular interventional operation

Abstract

The invention provides an image fusion method and device applied to a vascular interventional operation, and the method comprises the steps: obtaining a preoperative three-dimensional image and an intra-operative two-dimensional image, the preoperative three-dimensional image at least comprises a blood vessel image, and the intra-operative two-dimensional image at least comprises a vascular interventional operation instrument image; projecting the preoperative three-dimensional image according to the scanning angle of the intraoperative two-dimensional image to obtain a preoperative two-dimensional projection image; registering the preoperative two-dimensional projection image and the intraoperative two-dimensional image to obtain image transformation relation data; processing the preoperative two-dimensional projection image by using the transformation relation data to obtain a registered two-dimensional image; and fusing the registered two-dimensional image and the intraoperative two-dimensional image to obtain a fused two-dimensional image which at least comprises a blood vessel image and a blood vessel interventional surgical instrument image.

Description

Technical Field

[0001] This invention relates to the field of medical image processing, and specifically to an image fusion method and device for use in vascular interventional surgery. Background Technology

[0002] In current vascular interventional surgeries, surgeons need the aid of DSA (Digital Subtraction Angiography) images to determine the real-time position of surgical instruments such as catheters and guidewires within the body. DSA images are intraoperative images, two-dimensional projections of X-rays penetrating the body. Two X-ray images taken before and after contrast agent injection are processed through subtraction, enhancement, and re-imaging to obtain a clear, pure vascular image. DSA images are affected by vessel diameter and blood flow velocity; at lower contrast agent doses, the visualization of the vessels is not clear and complete, and only a local image showing the direction of blood flow at the angiography point is displayed.

[0003] CT images can provide a three-dimensional spatial anatomical structure of blood vessels. Compared to DSA, CTA can display blood vessel images and three-dimensional structures more clearly. However, CT images are preoperative images and cannot reflect the real-time position of surgical instruments such as catheters and guidewires inside the body during surgery.

[0004] Therefore, doctors can only imagine the three-dimensional spatial direction and shape of blood vessels based on the real-time position of surgical instruments such as catheters and guidewires in DSA images, combined with the characteristics of the surrounding tissue structures in the corresponding field of view of CT images, and conceive the three-dimensional spatial delivery path of surgical instruments in their minds. Summary of the Invention

[0005] In view of this, the present invention provides an image fusion method for vascular interventional surgery, comprising: acquiring a preoperative three-dimensional image and an intraoperative two-dimensional image, wherein the preoperative three-dimensional image includes at least a vascular image, and the intraoperative two-dimensional image includes at least an image of vascular interventional surgical instruments; projecting the preoperative three-dimensional image according to the scanning angle of the intraoperative two-dimensional image to obtain a preoperative two-dimensional projection image; registering the preoperative two-dimensional projection image and the intraoperative two-dimensional image to obtain image transformation relationship data; processing the preoperative two-dimensional projection image using the transformation relationship data to obtain a registered two-dimensional image; and fusing the registered two-dimensional image with the intraoperative two-dimensional image to obtain a fused two-dimensional image, wherein the fused two-dimensional image includes at least a vascular image and an image of vascular interventional surgical instruments.

[0006] Optionally, after obtaining the fused two-dimensional image, the method further includes: acquiring the position information of the tip of the vascular interventional surgical instrument in the positioning coordinate system; and updating the position of the vascular interventional surgical instrument image in the fused two-dimensional image using the position information according to the mapping relationship between the positioning coordinate system and the navigation coordinate system.

[0007] Optionally, registering the preoperative two-dimensional projection image and the intraoperative two-dimensional image further includes: acquiring a first intraoperative two-dimensional image without contrast agent injection and a second intraoperative two-dimensional image with contrast agent injection; obtaining a two-dimensional vascular image based on the first and second intraoperative two-dimensional images; removing the background from the preoperative two-dimensional projection image and retaining only the vascular image; constructing a Gaussian difference-of-scale spatial image of the preoperative two-dimensional projection image and the two-dimensional vascular image after background removal using preset scale parameters; and scanning extreme points in the Gaussian difference-of-scale spatial image to retain images with higher contrast. The extreme points of the degree threshold are used as vascular feature points; a feature region of a preset size is taken centered on each vascular feature point, and the feature region is divided into multiple sub-blocks. Seed points are determined based on the orientation gradient histogram of the sub-blocks, and the descriptors of the corresponding vascular feature points are determined using the orientation values ​​of the seed points; the vascular feature points that match each other are determined using the descriptors of the vascular feature points in the preoperative two-dimensional projection image and the two-dimensional vascular image; multiple sets of matching points are randomly selected from the matching vascular feature points, and the affine matrix is ​​calculated using the descriptors and the binary image as the image transformation relationship data.

[0008] Optionally, registering the preoperative two-dimensional projection image and the intraoperative two-dimensional image further includes: segmenting the bone image in the preoperative two-dimensional projection image and the intraoperative two-dimensional image to obtain a preoperative two-dimensional bone image and an intraoperative two-dimensional bone image; constructing a Gaussian difference-of-scale spatial image of the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image using preset scale parameters; scanning extreme points in the Gaussian difference-of-scale spatial image and retaining extreme points with contrast higher than a contrast threshold as bone feature points; taking a feature region of a preset size centered on each bone feature point, dividing the feature region into multiple sub-blocks, determining seed points based on the orientation gradient histogram of the sub-blocks, and determining the descriptor of the corresponding bone feature point using the orientation value of the seed point; determining mutually matching bone feature points using the descriptors of the bone feature points in the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image; randomly selecting multiple sets of matching point pairs among the mutually matching bone feature points, and calculating an affine matrix using the descriptor and the binary image as the image transformation relationship data.

[0009] Optionally, fusing the registered two-dimensional image with the intraoperative two-dimensional image further includes: calculating image similarity based on the number of matching points in the group, the affine matrix, and the binary image, and determining whether the similarity is higher than a threshold; if the similarity is higher than the threshold, then fusing the registered two-dimensional image with the intraoperative two-dimensional image.

[0010] Optionally, before projecting the preoperative three-dimensional image based on the scanning angle of the intraoperative two-dimensional image, the method further includes: acquiring an intraoperative scene image captured by a multi-view camera device, including an intraoperative imaging device and an operating table; constructing a three-dimensional coordinate system based on the fixed axis of the operating table; extracting structural features of the intraoperative imaging device from the intraoperative scene image; and determining the scanning angle based on the position of the structural features in the three-dimensional coordinate system.

[0011] Optionally, the preoperative three-dimensional image is a preoperative CTA image, and the intraoperative two-dimensional image is an intraoperative DSA image.

[0012] Optionally, the vascular interventional surgical instrument imaging includes catheter imaging and / or guidewire imaging.

[0013] Accordingly, the present invention provides an image fusion device, characterized in that it includes: a processor and a memory connected to the processor; wherein the memory stores instructions that can be executed by the processor, and the instructions are executed by the processor to cause the processor to perform the above-described image fusion method.

[0014] The present invention also provides a vascular interventional surgery system, comprising: the above-mentioned image fusion device, and a positioning system for vascular interventional surgical instruments; wherein the positioning system includes a position sensor and a position acquisition device, the position sensor being disposed on the vascular interventional surgical instrument, and the position acquisition device being used to determine the position information of the front end of the vascular interventional surgical instrument based on the position sensor.

[0015] According to the image fusion method and device provided by the present invention, a projected image is obtained by projecting a preoperative three-dimensional image according to the scanning angle of the intraoperative two-dimensional image. This projected image can clearly display blood vessels and accurately reflect the projection of the three-dimensional structure on the two-dimensional plane, making the vascular structure more accurate, while also displaying other tissues. After registering the projected image with the intraoperative two-dimensional image to obtain a registered image with a sufficiently high similarity, the registered image is fused with the intraoperative two-dimensional image. This allows for the clear display of blood vessels and surrounding tissues while displaying the vascular interventional surgical instruments. This fusion result can enhance the surgeon's spatial awareness of tissue structure and more accurately determine the positional relationship between blood vessels and interventional instruments. Applying this solution to interventional surgery planning and navigation can improve the efficiency of interventional surgery. Attached Figure Description

[0016] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0017] Figure 1 This is a flowchart of the image fusion method in an embodiment of the present invention. Detailed Implementation

[0018] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0019] In the description of this invention, it should be noted that the terms "center," "upper," "lower," "left," "right," "vertical," "horizontal," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0020] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can also refer to the internal connection of two components; and they can refer to a wireless connection or a wired connection. Those skilled in the art can understand the specific meaning of the above terms in this invention based on the specific circumstances.

[0021] Furthermore, the technical features involved in the different embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0022] This invention provides an image fusion method for vascular interventional surgery, which can be executed by electronic devices such as computers or servers. Figure 1 The following operations are shown:

[0023] S1. Acquire preoperative 3D images and intraoperative 2D images. The preoperative 3D images must contain at least vascular images, and the intraoperative 2D images must contain at least images of the interventional vascular instruments. The preoperative 3D images can be images scanned using medical imaging equipment, such as CT angiography (CTA) images obtained with the aid of contrast agents, or MRA (magnetic resonance angiography) images obtained using MRI. Both types of images are three-dimensional volumetric data, clearly showing the three-dimensional vascular structure of the surgical subject. Since these images are scanned for preoperative diagnosis or surgical planning, they do not contain images of the interventional vascular instruments.

[0024] Intraoperative two-dimensional images refer to images obtained using medical imaging equipment in the operating room after vascular interventional surgical instruments have entered the blood vessels of the surgical patient during the procedure. These images are typically X-ray-based scans. While they can display the image of the vascular interventional surgical instruments, they cannot clearly or stably display the image of the blood vessels. Interventional surgical instruments can be guidewires, catheters, or any suitable device.

[0025] S2, the preoperative three-dimensional image is projected based on the scanning angle of the intraoperative two-dimensional image to obtain the preoperative two-dimensional projection image. Regarding the scanning angle, when performing interventional surgery, the surgeon can set the scanning angle of the medical imaging equipment according to the actual situation when performing X-ray scans on the surgical object. The device implementing this method can read this angle data in the medical imaging equipment.

[0026] If data cannot be read directly due to issues such as data interface and permissions, the angle data can be obtained through manual input or with the help of a visual tracking system. Alternatively, the projection image with the highest similarity to the intraoperative two-dimensional image can be found by traversing the projection angles. The projection angle corresponding to this projection image can be regarded as the scanning angle of the intraoperative two-dimensional image.

[0027] Preoperative 3D images can be viewed as a 3D model. Given the projection angle, calculations can be performed on the 3D data based on that angle to obtain a 2D image with a high degree of similarity to the intraoperative 2D image. Specifically, the DRR (Digitally Reconstructed Radiograph) algorithm can be used to generate a 2D image from the 3D volume data (multiple cross-sectional image data) through mathematical simulation algorithms.

[0028] To obtain the projected image, a 3D coordinate system needs to be constructed with the bed orientation and its vertical plane. The deviation angle theta (including three directions) between the actual exposure angle of the intraoperative DSA equipment and the exposure angle of the preoperative CTA equipment is calculated. The CTA data is rotated around the center point by theta, thereby calculating the planar 2D projection of the CTA at that angle.

[0029] S3: Register the preoperative 2D projection image and the intraoperative 2D image to obtain image transformation relationship data. For ease of description, the preoperative 3D image is denoted as CTA, the intraoperative 2D image as DSA, and the preoperative 2D projection image as DRR. Although the scanning angle of DSA is used in step S2, the DRR obtained directly is not completely equal to DSA because the condition of the surgical subject is different during and before surgery. In other words, the similarity between the two images is insufficient, and they cannot be directly fused.

[0030] The so-called registration in this application refers to extracting corresponding feature points (or key points) from DSA and DRR, performing registration operations using the feature information of these feature points to obtain image transformation relationship data T, and using T to process DRR to obtain a result that is sufficiently close to DSA. The registration process is the process of obtaining T.

[0031] There are various types of feature points, selection methods, and registration calculation methods. Feature points must be recognizable in both the DRR and DSA images. Specifically, they can be bones, blood vessels, or other radiopaque human tissues or organs. It should be noted that a two-dimensional image DSA can be understood as an X-ray image (image1) with contrast agent injected, or an X-ray image (image2) without contrast agent injected. The former can visualize blood vessels and bones, while the latter can visualize bones. A two-dimensional image DSA can also be understood as image2 - image1 = image3, which only includes vascular images.

[0032] S4. The preoperative two-dimensional projection image is processed using the transformation relationship data to obtain a registered two-dimensional image. The image transformation relationship data T obtained in step S3 is used to transform the DRR, and the resulting registered two-dimensional image is denoted as DRR'. DRR' has a higher similarity to DSA.

[0033] S5, the registered 2D image is fused with the intraoperative 2D image to obtain a fused 2D image, which includes at least vascular images and vascular interventional surgical instrument images. Since DRR' comes from CTA, it necessarily contains vascular images, while DSA itself contains vascular interventional surgical instrument images. Therefore, the superposition result of the two contains both of these key images. In a preferred embodiment, the similarity between DRR' and DSA can be judged first. If it is higher than a threshold, fusion is performed; otherwise, the surgeon can be prompted to change the angle of scanning the intraoperative 2D image and repeat steps S1-S5.

[0034] In this embodiment, the so-called overlay preferably involves adding all the contents of the DRR' into the CTA. The physician can scan the CTA in real time while the catheter or guidewire is moving. Without changing the scanning position and angle, the movement of the catheter or guidewire in the blood vessel can be observed in real time in the overlay results without the need for contrast agents.

[0035] According to the image fusion method provided in this embodiment of the invention, a preoperative three-dimensional image is projected according to the scanning angle of the intraoperative two-dimensional image to obtain a projected image. This projected image can clearly display blood vessels and accurately reflect the projection of the three-dimensional structure on the two-dimensional plane, making the vascular structure more accurate, while also displaying other tissues. After registering the projected image with the intraoperative two-dimensional image to obtain a registered image with sufficiently high similarity, the registered image is fused with the intraoperative two-dimensional image. This allows for the clear display of blood vessels and surrounding tissues while displaying the vascular interventional surgical instruments. This fusion result can enhance the surgeon's spatial awareness of tissue structure and more accurately determine the positional relationship between blood vessels and interventional instruments. Applying this solution to interventional surgery planning and navigation can improve the efficiency of interventional surgery.

[0036] In one embodiment, the movement of the interventional surgical instrument in an actual blood vessel can also be simulated by image simulation, displaying the movement trajectory of the interventional surgical instrument fed back by the positioning device onto a fused two-dimensional image. In this embodiment, a positioning system for the interventional surgical instrument is required to locate and track the tip of the instrument. The positioning system includes a position sensor and a position acquisition device. The position sensor is mounted on the interventional surgical instrument, and the position acquisition device is used to determine the position information of the tip of the interventional surgical instrument based on the position sensor.

[0037] Specifically, the position sensor and position acquisition device can be devices that locate based on electromagnetic signals. The position sensor specifically comprises an electromagnetic sensor and a magnetic field generating device. The electromagnetic sensor is located at the front end of the guidewire / conduit, generating electromagnetic induction in the magnetic field, and the electromagnetic induction signal indicates its spatial position. Alternatively, the position sensor can be a device that locates based on optical signals. The position sensor specifically comprises a positioning optical fiber and an optical signal transceiver. The front end of the optical fiber is located at the front end of the guidewire / conduit, or integrated into it. The optical signal reflected and refracted by the optical fiber indicates its spatial position. Using the above positioning system, the following operation can be performed after step S5:

[0038] S6, Obtain the position information of the tip of the vascular interventional surgical instrument in the positioning coordinate system;

[0039] S7, based on the mapping relationship between the positioning coordinate system and the navigation coordinate system, uses the location information to update the position of the vascular interventional surgical instrument image in the fused two-dimensional image.

[0040] Specifically, the orientation of the positioning coordinate system is corrected according to the orientation of the bed before the operation, and the mapping between the positioning coordinate system and the navigation coordinate system is established during the operation. The navigation coordinate system is the coordinate system that integrates two-dimensional images.

[0041] After obtaining the fused two-dimensional image in step S5, the initial position O(x0,y0) of the guidewire / catheter is confirmed. According to the mapping relationship G between the positioning coordinate system and the navigation coordinate system, when the guidewire moves L1 length along the direction theta1, the coordinate represented by the guidewire in the corresponding navigation coordinate system moves L2 length along theta2.

[0042] S(theta2, L2) = G*S1(theta1, L1), where S represents the displacement of the guidewire / catheter image in the fused two-dimensional image in the navigation coordinate system, and S1 represents the actual displacement of the guidewire / catheter instrument in the positioning coordinate system.

[0043] More specifically, in one embodiment, the guidewire as a whole or near the front end is a hollow structure, and a slender electromagnetic sensor is disposed in the hollow structure of the guidewire. The distance between the electromagnetic sensor and the front end of the guidewire is a fixed and known parameter. Based on the position and orientation information of the electromagnetic sensor and the known parameter, the position information of the front end of the guidewire can be calculated, thereby allowing the position of the guidewire to be displayed and updated at least in the fused two-dimensional image.

[0044] In an optional embodiment, the position of the catheter can also be displayed and updated simultaneously. In this embodiment, no electromagnetic sensor is installed in the catheter. The vascular interventional surgical instrument can calculate and report the distance difference between the tip of the guidewire and the tip of the catheter based on the amount of delivery to the guidewire and the catheter. Knowing the position information and orientation information of the tip of the guidewire and the distance difference, the position of the tip of the catheter can be calculated, and the position of the guidewire can be displayed and updated in the fused two-dimensional image based on this.

[0045] According to the above embodiments, after obtaining the fused two-dimensional image, the guidewire / catheter in the image is moved accordingly according to the movement of the actual instrument through a positioning system and image simulation, thereby reducing the X-ray scanning operation during the operation and reducing the radiation dose.

[0046] Regarding the scanning angle in step S2 above, in cases where the scanning angle cannot be directly read from the intraoperative medical imaging equipment (DSA equipment), in an optional embodiment, the scanning angle of the equipment can be obtained in the following way:

[0047] S21, Acquire intraoperative scene images captured by multi-camera equipment, including intraoperative imaging equipment and operating table;

[0048] S22, a three-dimensional coordinate system is constructed based on the fixed axis of the operating table. The structural features of the intraoperative imaging equipment are extracted from the intraoperative scene image, and the scanning angle is determined according to the position of the structural features in the three-dimensional coordinate system.

[0049] Specifically, if it is not possible to obtain the tilt angle data of the DSA device in real time during the operation, an optical positioning device can be used for assistance. Specifically, it can be a multi-camera device. A three-dimensional coordinate system is constructed with the operating table as the fixed axis, the fixed features of the DSA device are extracted, and the position of the C-arm or the table of the DSA device is tracked to calculate the offset angle in three directions during exposure.

[0050] Regarding the image registration in step S3 above, in one embodiment, blood vessel images in the image are used as feature points for registration, and step S3 further includes the following operations:

[0051] S31A: Acquire a first intraoperative two-dimensional image without contrast agent injection and a second intraoperative two-dimensional image with contrast agent injection. In this embodiment, contrast agent is needed to acquire DSA images intraoperatively. Specifically, an X-ray image image1 is obtained by scanning without contrast agent, and then, while maintaining the current scanning position, contrast agent is injected into the blood vessel, and another X-ray image image2 is obtained by scanning. The only difference between the two images is that image1 has no blood vessel and image2 has a blood vessel.

[0052] S32A: A two-dimensional vascular image is obtained based on the first intraoperative two-dimensional image and the second intraoperative two-dimensional image. The two-dimensional vascular image image3 is obtained by subtracting the two images, which contains only vascular images.

[0053] S33A involves registering the preoperative two-dimensional projection image and the two-dimensional vascular image. Specifically, this refers to performing registration operations on the two images, image3 and DRR, calculating the affine matrix T so that DRR can be transformed into image3 through T. In this embodiment, selecting blood vessels as the basis for registration can improve the accuracy of the registration results.

[0054] As described above, the DRR comes from preoperative CTA, which contains other tissues besides blood vessels, such as bone. Therefore, the DRR needs to be processed before calculation. Specifically, step S33A may include the following operations:

[0055] S33A1 removes the background from preoperative 2D projection images, retaining only the vascular images. Since the vascular images in the DRR are sufficiently clear and the vascular features are sufficiently obvious, it is relatively easy to extract the vessels (remove the background) from the images using machine vision algorithms. Additionally, considering that CT and DSA devices may have different resolutions, the DRR and image3 images need to be adjusted to the same resolution before registration, which can be done using algorithms such as linear interpolation.

[0056] S33A2 uses preset scale parameters to construct Gaussian difference scale space images of preoperative two-dimensional projection images and two-dimensional vascular images after background removal.

[0057] For ease of description, the DRR after removing the background will be denoted as DRR. - Let the two-dimensional blood vessel image be denoted as image3(x,y), and the Gaussian difference scale space image of these two images be denoted as DRR. - (x,y,σ), image3(x,y,σ), where σ is a preset scale parameter, DRR - (x,y,σ)=DRR - (x,y)*G(x,y,σ), image3(x,y,σ)=image3(x,y)*G(x,y,σ), where G(x,y,σ) is the Gaussian smoothing kernel function.

[0058] S33A3 involves scanning extreme points in the Gaussian difference-of-scale image and retaining those with contrast exceeding a contrast threshold as vascular feature points. Specifically, in DRR... -After scanning the extreme points in (x,y,σ) and image3(x,y,σ) to generate a Gaussian difference scale space image, each sampling point is scanned and compared with its surrounding 26 pixels (8 pixels in the neighborhood and 9*2 pixels in the adjacent layers above and below) to determine whether it is an extreme point. The local extreme points found in this way are the coarse vascular feature points (key points) of the image.

[0059] S33A4: A feature region of a preset size is selected centered on each blood vessel feature point. This feature region is divided into multiple sub-blocks. Seed points are determined based on the histogram of the directional gradients of the sub-blocks. The direction values ​​of the seed points are then used to determine the descriptors of the corresponding blood vessel feature points. Specifically, after selecting the coarse blood vessel feature points of the image, a difference algorithm is used to determine the position and scale of key points. Then, low-contrast extreme points are removed, and the Hessian matrix is ​​used to remove edge response interference caused by Gaussian difference operations, thereby optimizing the feature point detection results.

[0060] In addition to its coordinate values ​​(planar position and scale), the feature vector of a blood vessel feature point also needs to have its orientation value determined by the gradient direction of its neighboring pixels. For example, taking a 16×16 pixel region centered on the blood vessel feature point, dividing this region into 4×4 sub-blocks, and calculating the orientation gradient histogram of each sub-block, yields a seed point. Each feature point consists of 4×4 seed points, and each seed point is divided into 8 directions, thus forming a 4×4×8 dimensional feature vector as the descriptor of the corresponding feature point. This vector possesses rotation invariance, scale invariance, and other properties.

[0061] S33A5, using descriptors of vascular feature points in preoperative two-dimensional projection images and two-dimensional vascular images, determines mutually matching vascular feature points. Specifically, this can be done based on the descriptors of feature points in image3 and DRR. - The descriptors of the feature points in the data are used to calculate the distance (such as Euclidean distance, Hamming distance, etc.), and the matching is determined based on the distance, thereby identifying several pairs of vascular feature points.

[0062] S33A6: Randomly select multiple sets of matching points from the mutually matched vascular feature points, and use descriptors and binary images to calculate an affine matrix as image transformation relationship data. In practical applications, at least 3 sets of matching points need to be randomly selected. The calculated affine matrix is ​​used to translate, rotate, and stretch the image.

[0063] The DRR is processed using affine matrices to perform transformations such as translation, rotation, and stretching on its contents. The transformed result, DRR', can then be fused with the DSA.

[0064] This embodiment extracts local features from preoperative two-dimensional projection images and two-dimensional vascular images. Even if there are only a few vascular images in the image, a large number of vascular feature points can be extracted. The descriptors of these vascular feature points have rich information content, thereby enabling fast and accurate matching of vascular feature points in the two images. The affine matrix generated in this way is used to transform the image, which remains unchanged by image rotation, scaling, and brightness changes, and also maintains a certain degree of stability against viewpoint changes, affine transformations, and noise. Regarding the image registration in step S3 above, in one embodiment, the skeletal image in the image is used as the feature points for registration. Step S3 further includes the following operations:

[0065] S31B, the bone image is segmented from the preoperative two-dimensional projection image and the intraoperative two-dimensional image, respectively, to obtain the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image. Both CT and X-ray scans can visualize the bone, and in this embodiment, no contrast agent is required, so only one intraoperative two-dimensional image is needed. Since the bone images in DRR (from CTA) and DSA are clear enough and the bone features are obvious enough, the bone can be extracted from the image relatively easily using machine vision algorithms.

[0066] S32B performs registration between preoperative and intraoperative two-dimensional skeletal images. This embodiment selects the skeleton as the basis for registration, performing image registration without the aid of contrast agents. This reduces the adverse effects of contrast agents and high-dose radiation on patients and doctors, and improves the efficiency of image fusion.

[0067] Similar to step S33A above, step S32B may include the following operations:

[0068] S32B1 constructs Gaussian difference-of-scale spatial images of preoperative and intraoperative two-dimensional skeleton images using preset scale parameters. For ease of description, the preoperative two-dimensional skeleton image is denoted as DRR. b (x,y), the intraoperative two-dimensional skeletal image is denoted as DSA. b The Gaussian difference-scaled spatial image of these two images (x, y) is denoted as DRR. b (x,y,σ), DSA b (x,y,σ), where σ is a preset scaling parameter, DRR b (x,y,σ)=DRR b (x,y)*G(x,y,σ), DSA b (x,y,σ)=DSA b (x,y)*G(x,y,σ), where G(x,y,σ) is the Gaussian smoothing kernel function.

[0069] S32B2 scans extreme points in the Gaussian difference-of-scale image, retaining extreme points with contrast above a contrast threshold as skeletal feature points. Specifically, in DRR... b (x,y,σ) and DSA b After scanning the extreme points in (x, y, σ) to generate a Gaussian difference-of-scale image, each sampling point is scanned and compared with its surrounding 26 pixels (8 pixels in the neighborhood and 9*2 pixels in the adjacent layers above and below) to determine whether it is an extreme point. The local extreme points found in this way are the coarse skeletal feature points (keypoints) of the image.

[0070] S32B3: A feature region of a preset size is selected centered on each of the aforementioned skeletal feature points. This feature region is divided into multiple sub-blocks. Seed points are determined based on the histogram of the directional gradients of the sub-blocks. The direction values ​​of the seed points are then used to determine the descriptors of the corresponding skeletal feature points. Specifically, after selecting the coarse skeletal feature points of the image, a difference algorithm is used to determine the position and scale of the key points. Then, low-contrast extreme points are removed, and the Hessian matrix is ​​used to remove edge response interference caused by the difference of Gaussians operation, thereby optimizing the feature point detection results.

[0071] In addition to their coordinate values ​​(planar position and scale), the feature vectors of skeletal feature points also have their orientation values ​​determined by the gradient directions of their neighboring pixels. For example, taking a 16×16 pixel region centered on the skeletal feature point, dividing this region into 4×4 sub-blocks, and calculating the orientation gradient histogram of each sub-block yields a seed point. Each feature point consists of 4×4 seed points, and each seed point is divided into 8 directions, thus forming a 4×4×8 dimensional feature vector as the descriptor of the corresponding feature point. This vector possesses rotation invariance, scale invariance, and other properties.

[0072] S32B4, using descriptors of bone feature points in preoperative and intraoperative two-dimensional bone images, determines mutually matching bone feature points. Specifically, this can be based on DRR. b Descriptors of feature points in DSA b The descriptors of the feature points in the image are used to calculate the distance (e.g., Euclidean distance, Hamming distance, etc.), and a match is determined based on the distance, thereby identifying several pairs of blood vessel feature points. In step S32B5, multiple sets of matching point pairs are randomly selected from the mutually matched skeletal feature points, and an affine matrix is ​​calculated using the descriptors and the binary image as image transformation relationship data. In practical applications, at least three sets of matching points need to be randomly selected. The calculated affine matrix is ​​used for image translation, rotation, and stretching.

[0073] The DRR is processed using affine matrices to perform transformations such as translation, rotation, and stretching on its contents. The transformed result, DRR', can then be fused with the DSA.

[0074] This embodiment extracts local features from preoperative and intraoperative two-dimensional skeletal images. Even if only a few skeletal images exist in the images, a large number of skeletal feature points can be extracted. The descriptors of these skeletal feature points have rich information content, thereby enabling fast and accurate matching of skeletal feature points in the two images. The affine matrix generated in this way is used to transform the image, which remains unchanged by image rotation, scaling, and brightness changes, and also maintains a certain degree of stability against viewpoint changes, affine transformations, and noise. Furthermore, in the case of adopting the above two embodiments of steps S31A-S33A or S31B-S32B, in step S5, the image similarity can be calculated based on the number of matching points, the affine matrix, and the binary image, and it can be determined whether the similarity is higher than a threshold. The binary image refers to the image after binarizing DRR' and the image after binarizing DSA. If the similarity is higher than the threshold, the registered 2D image will be fused with the intraoperative 2D image; if the similarity is lower than the threshold, the user can be prompted to confirm whether to perform image fusion, or the process can return to step S1 to obtain intraoperative 2D images from different scanning angles and execute steps S2-S3 again.

[0075] Regarding the calculation of image similarity, a neural network model can be used as an example. Specifically, the number of matching points, the degree of deviation indicated by the affine matrix (translation, rotation, deformation), and the degree of overlap between the two binary images are taken as input data, and the neural network model outputs the image similarity.

[0076] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] Obviously, the above embodiments are merely illustrative examples for clear explanation and are not intended to limit the implementation. Those skilled in the art will recognize that other variations or modifications can be made based on the above description. It is neither necessary nor possible to exhaustively list all possible implementations here. However, obvious variations or modifications derived therefrom are still within the scope of protection of this invention.

Claims

1. An image fusion method for use in vascular interventional surgery, characterized in that, include: Acquire preoperative three-dimensional images and intraoperative two-dimensional images, wherein the preoperative three-dimensional images contain at least vascular images and the intraoperative two-dimensional images contain at least vascular interventional surgical instrument images; The preoperative three-dimensional image is projected based on the scanning angle of the intraoperative two-dimensional image to obtain a preoperative two-dimensional projection image; The preoperative two-dimensional projection image and the intraoperative two-dimensional image are registered to obtain image transformation relationship data; The preoperative two-dimensional projection image is processed using the transformation relationship data to obtain a registered two-dimensional image; The registered two-dimensional image is fused with the intraoperative two-dimensional image to obtain a fused two-dimensional image, which includes at least vascular images and vascular interventional surgical instrument images.

2. The image fusion method according to claim 1, characterized in that, After obtaining the fused two-dimensional image, the following is also included: Obtain the position information of the tip of the vascular interventional surgical instrument in the positioning coordinate system; Based on the mapping relationship between the positioning coordinate system and the navigation coordinate system, the position of the vascular interventional surgical instrument image in the fused two-dimensional image is updated using the location information.

3. The image fusion method according to claim 1, characterized in that, Registration of the preoperative two-dimensional projection image and the intraoperative two-dimensional image further includes: Acquire a first intraoperative two-dimensional image without contrast agent injection and a second intraoperative two-dimensional image with contrast agent injection; Two-dimensional vascular images were obtained based on the first intraoperative two-dimensional image and the second intraoperative two-dimensional image. The background is removed from the preoperative two-dimensional projection image, leaving only the vascular image; Using preset scale parameters, Gaussian difference scale space images of the preoperative two-dimensional projection image and the two-dimensional vascular image after removing the background are constructed; In the Gaussian difference scale space image, extreme points are scanned respectively, and extreme points with contrast higher than the contrast threshold are retained as vascular feature points. Each of the blood vessel feature points is centered on a feature region of a preset size. The feature region is divided into multiple sub-blocks. Seed points are determined based on the orientation gradient histogram of the sub-blocks. The orientation values ​​of the seed points are used to determine the descriptors of the corresponding blood vessel feature points. Using the descriptors of the vascular feature points in the preoperative two-dimensional projection image and the two-dimensional vascular image, the mutually matching vascular feature points are determined; Multiple sets of matching points are randomly selected from the mutually matched vascular feature points, and the affine matrix is ​​calculated using the descriptor and binary image pair as the image transformation relationship data.

4. The image fusion method according to claim 1, characterized in that, Registration of the preoperative two-dimensional projection image and the intraoperative two-dimensional image further includes: The bone image is segmented from the preoperative two-dimensional projection image and the intraoperative two-dimensional image to obtain the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image, respectively. Using preset scale parameters, Gaussian difference scale space images of the preoperative two-dimensional skeletal image and the intraoperative two-dimensional skeletal image are constructed; In the Gaussian difference scale space image, extreme points are scanned respectively, and extreme points with contrast higher than the contrast threshold are retained as skeletal feature points. Each of the aforementioned skeletal feature points is centered on a feature region of a preset size. The feature region is divided into multiple sub-blocks. Seed points are determined based on the orientation gradient histogram of the sub-blocks. The orientation values ​​of the seed points are used to determine the descriptors of the corresponding skeletal feature points. Using the descriptors of the bone feature points in the preoperative two-dimensional bone image and the intraoperative two-dimensional bone image, mutually matching bone feature points are determined; Multiple sets of matching point pairs are randomly selected from the mutually matched skeletal feature points, and the affine matrix is ​​calculated using the descriptor and the binary image as the image transformation relationship data.

5. The image fusion method according to claim 3 or 4, characterized in that, Fusing the registered two-dimensional image with the intraoperative two-dimensional image further includes: The image similarity is calculated based on the number of matching points in the group, the affine matrix, and the binary image, and it is determined whether the similarity is higher than a threshold. If the similarity is higher than a threshold, the registered two-dimensional image is fused with the intraoperative two-dimensional image.

6. The image fusion method according to claim 1, characterized in that, Before projecting the preoperative three-dimensional image based on the scanning angle of the intraoperative two-dimensional image, the method further includes: Acquire intraoperative scene images captured by multi-view camera equipment, including intraoperative imaging equipment and operating table; A three-dimensional coordinate system is constructed based on the fixed axis of the operating table. The structural features of the intraoperative imaging equipment are extracted from the intraoperative scene image, and the scanning angle is determined based on the position of the structural features in the three-dimensional coordinate system.

7. The image fusion method according to any one of claims 1-6, characterized in that, The preoperative three-dimensional image is a preoperative CTA image, and the intraoperative two-dimensional image is an intraoperative DSA image.

8. The image fusion method according to any one of claims 1-6, characterized in that, The images of the vascular interventional surgical instruments include catheter images and / or guidewire images.

9. An image fusion device, characterized in that, include: A processor and a memory connected to the processor; wherein the memory stores instructions executable by the processor, the instructions being executed by the processor to cause the processor to perform the image fusion method according to any one of claims 1-8.

10. A vascular interventional surgical system, characterized in that, include: The image fusion device of claim 9, and the positioning system of the vascular interventional surgical instrument; wherein the positioning system includes a position sensor and a position acquisition device, the position sensor is disposed on the vascular interventional surgical instrument, and the position acquisition device is used to determine the position information of the front end of the vascular interventional surgical instrument based on the position sensor.

Images