Systems and methods for medical image fusion

Abstract

CT angiography can be used to obtain a 3D anatomical model of one or more blood vessels in a patient, while simultaneously obtaining 2D images of the vessels based on perspective. The 3D model can be registered with the 2D images based on contrast agent injection sites identified on the 3D model and / or in the 2D images. A fused image can then be created to depict the overlaid 3D model and 2D image, for example, on a monitor or via a virtual reality headset. Injection sites can be determined automatically or based on user input, which may include bounding boxes drawn around the injection sites on the 3D model, selection of automatically segmented regions in the 3D model, etc.

Description

Technical Field

[0001] This application relates to the field of medical imaging. Background Technology

[0002] Medical imaging, such as X-ray fluoroscopy, can be used during surgical procedures, for example, to guide the insertion or implantation of interventional devices into the human body in real time. To visualize the devices and / or human organs (e.g., blood vessels) involved in these procedures, radiopaque contrast agents can be injected into the patient's blood vessels, allowing the devices and / or blood vessels to be delineated from other background objects in images captured by medical imaging devices (e.g., computed tomography (CT) scanners or X-ray scanners). However, currently available imaging techniques and contrast agents can have potential adverse effects on the human body, thus requiring the imaging process to be completed rapidly, as the contrast agent may need to be washed out shortly after injection and / or maintained at a low dose to avoid adverse effects. These constraints may limit the clinician's ability to monitor the implantation of medical devices during the procedure and may affect the speed and success rate of these procedures. Summary of the Invention

[0003] This document describes systems, methods, and apparatuses associated with medical image registration and fusion. An apparatus capable of performing such a task may include at least one processor configured to acquire a three-dimensional (3D) anatomical model characterizing one or more blood vessels of a patient, and two-dimensional (2D) perspective images of all or a subset of the one or more blood vessels, wherein the 2D perspective images may indicate that at least one of the one or more blood vessels has been injected with a contrast agent (e.g., contrast dye). The at least one processor may also be configured to determine the injection site of the contrast agent in the 2D perspective image and on the 3D anatomical model (e.g., automatically or based on user input), and to register the 3D anatomical model with the 2D perspective image based at least on the corresponding location of the injection site in the 2D perspective image and the 3D anatomical model, such that the 3D anatomical model is aligned (e.g., approximately) with respect to at least one of the one or more blood vessels with respect to the 2D perspective image. The at least one processor may also be configured to overlay (e.g., project) the 3D anatomical model onto the 2D perspective image and generate an image depicting the overlaid 3D anatomical model and the 2D perspective image. The images can then be displayed on a monitor or via a virtual reality (VR) head-mounted viewer, either or both of which can be part of the device, allowing the superimposed 3D vascular model and 2D perspective image to be visualized on the monitor or via the VR head-mounted viewer.

[0004] In one embodiment, at least one processor of the device can be configured to determine an injection site on a 3D anatomical model based on user input, which may indicate the location of the injection site on the 3D anatomical model. Such user input may include, for example, a bounding box drawn on the 3D anatomical model that may mark the injection site. In another embodiment, at least one processor of the device can be configured to determine the injection site on the 3D anatomical model by at least the following operations: The processor may segment the 3D anatomical model into multiple segments, each segment representing a patient's anatomy. The processor may also receive user input indicating which segment(s) among the multiple segments contain the injection site, and determine the injection site based on the user input.

[0005] In this embodiment, the 2D fluoroscopic image can be obtained using a medical imaging apparatus such as a C-arm X-ray machine, and at least one processor can be configured to determine the injection site on the 3D anatomical model by at least the following operations: The processor can determine the patient's pose and body shape based on information provided by a sensing device (e.g., a camera configured to capture images of the patient). The processor can also determine the relative pose and orientation of the 3D anatomical model and the sensing device, as well as the relative pose and orientation of the medical imaging device and the sensing device, and determine the injection site on the 3D anatomical model based at least on the patient's pose or body shape, the relative pose and orientation of the 3D anatomical model and the sensing device, and the relative pose and orientation of the medical imaging device and the sensing device.

[0006] In one embodiment, at least one processor of the device may also be configured to generate a 2D mask associated with one or more blood vessels of the patient based on a 2D perspective image, and to register the 2D mask with a 3D anatomical model based on the injection site. In another embodiment, at least one processor may also be configured to detect a medical device implanted inside one of the patient's one or more blood vessels based on a 2D perspective image, generate a 3D model of the medical device based on a predefined device model, and overlay the 3D model of the medical device onto a 3D image depicting the blood vessel in which the medical device is implanted. This 3D image may be an image used to obtain the 3D anatomical model or a 3D image registered with an image used to obtain the 3D anatomical model. The 3D anatomical model may be obtained based on a 3D image. Attached Figure Description

[0007] A more detailed understanding of the examples disclosed herein can be obtained from the following description, which is given by way of example in conjunction with the accompanying drawings.

[0008] Figure 1 This is a simplified diagram illustrating an example process for registering and / or fusing medical images according to some embodiments described herein.

[0009] Figure 2This is a simplified diagram illustrating examples of registering and / or fusing 3D anatomical models with 2D perspective images according to some embodiments described herein.

[0010] Figure 3 This is a simplified diagram illustrating an example of registering two images using an artificial neural network (ANN) according to some embodiments described herein.

[0011] Figure 4 This is a simplified diagram illustrating an example structure of a neural ordinary differential equation (ODE) network according to some embodiments described herein, which can be used to determine transformation parameters for registering two medical images.

[0012] Figure 5 This is a flowchart illustrating an example method for training a neural network to perform one or more tasks as described with respect to some embodiments provided herein.

[0013] Figure 6 This is a simplified block diagram illustrating an example system or device for performing one or more tasks as described with respect to some embodiments provided herein. Detailed Implementation

[0014] The present disclosure is illustrated by way of example and not limitation in the accompanying drawings. A detailed description of illustrative embodiments will now be described with reference to the various drawings. Although detailed examples of possible implementations are provided in this specification, it should be noted that these details are intended to be exemplary and in no way intended to limit the scope of this application.

[0015] Figure 1 Example processes for registering and / or fusing medical images according to one or more embodiments described herein are illustrated. Medical images can include various types of two-dimensional (2D) or three-dimensional (3D) images (including 3D models) and can depict various anatomical structures of the human body and / or medical devices implanted in the human body. For example, medical images can depict one or more blood vessels (such as coronary arteries or veins), one or more implantable devices (such as catheters or stents), etc. Medical images (including 3D models) can be obtained using various medical imaging and modeling techniques. For example, medical images can include a 3D anatomical model 102 (e.g., a 3D vascular model or image) obtained based on computed tomography (CT) angiography and a 2D medical image 104 (e.g., a fluoroscopic image) obtained based on X-ray fluoroscopy. The 3D anatomical model 102 can characterize one or more blood vessels of a patient (e.g., the model can also include other anatomical structures, such as the patient's bones and / or joints), while the 2D medical image 104 can be an image of all or a subset of the one or more blood vessels characterized by the 3D model.

[0016] In embodiments of this disclosure, the 3D model 102 can be obtained during preoperative steps (e.g., before beginning a surgical procedure for the patient), while the 2D medical image 104 can be obtained during the surgical procedure (e.g., after injecting a contrast agent (e.g., contrast dye) into one or more blood vessels of the patient characterized by the 3D anatomical model 102). Once obtained, the 2D medical image 104 can be registered with the 3D model 102 such that at least one blood vessel on the 3D anatomical model 102 can be aligned (e.g., approximately) with at least one blood vessel on the 2D medical image 104. The registered 3D anatomical model 102 and the 2D medical image 104 can then be fused together, for example, by overlaying (e.g., projecting) the 3D anatomical model 102 onto the 2D medical image 104 and generating an image 106 depicting the overlaid 3D anatomical model 102 and the 2D medical image 104. The fused image 106 can be used for various purposes. For example, since the 2D medical image 104 can be an image captured using a contrast agent, it can show which specific blood vessel has been injected with the contrast agent, the orientation of the contrast agent within the blood vessel, the orientation of an implanted medical device (e.g., a catheter) within the blood vessel, etc. Thus, by capturing multiple of these images and registering / fusing them with the 3D anatomical model 102, clinicians can monitor the placement and movement of the contrast agent and / or implanted medical device within the patient (e.g., relative to the 3D anatomical model) to ensure that the medical device is properly positioned.

[0017] In embodiments of this disclosure, fused image 106 (e.g., multiple such fused images) can be displayed on a monitoring device such as a computer monitor and / or a virtual reality (VR) headset (e.g., played back as an animation or video). In embodiments of this disclosure, a 2D vascular mask can be obtained based on 2D medical image 104 (e.g., using a pre-trained artificial intelligence (AI) segmentation model) and registered / fused with a 3D anatomical model (e.g., in addition to or instead of registering 2D medical image 104 with 3D anatomical model 102) to facilitate one or more clinical tasks. In embodiments of this disclosure, a 3D model of an implanted medical device can be reconstructed based on 2D medical image 104 and / or a predefined device model for the implanted medical device. The 3D device model can then be overlaid with a 3D medical image of the anatomical structure in which the medical device is implanted to depict the pose and / or orientation of the implanted medical device within the anatomical structure. The 3D medical image overlaid with the 3D device model can be an image used to generate 3D anatomical model 102 or another image registered with an image used to generate 3D anatomical model 102.

[0018] The registration of the 3D anatomical model 102 and the 2D medical image 104 can be done automatically (e.g., based on a pre-trained AI image registration model) and / or based on user input that can assist in the registration. For example, an interface (e.g., a graphical user interface) can be provided to the user, through which the 2D medical image 104 can be displayed, and the user can identify at least one blood vessel in the 2D medical image that may have been injected with contrast agent. In response to the identification of such a blood vessel in the 2D medical image 104, the user can indicate the location of the contrast agent injection site (e.g., the location of the catheter tip when the contrast agent is injected) and / or at least one blood vessel associated with the injection site in the 2D medical image 104 and / or on the 3D anatomical model 102. The user can provide indication, for example, by clicking or drawing a bounding box around a specific area of ​​the 2D medical image 104 and / or the 3D anatomical model 102 to mark the injection site. The injection site can then be used as a reference marker or feature to improve the accuracy of the registration and / or reduce the computations associated with the registration. In embodiments of this disclosure, using a pre-trained AI segmentation model, 2D medical image 104 and / or 3D anatomical model 102 can be automatically segmented into multiple segments, and key markers can be automatically marked (e.g., to indicate the femoral bifurcation, common femoral artery, common iliac artery, abdominal aorta, left ventricle, etc.). The user can then select one of these segments and / or key markers to indicate that the selected segment or marker includes an injection site or a blood vessel associated with the injection site.

[0019] In embodiments of this disclosure, sensing devices such as digital cameras, depth sensors, thermal sensors, and / or radar sensors may be positioned in an operating room and used to capture images of the patient (e.g., color images, depth images, etc.). The patient images can then be used to construct a human body model (e.g., a 3D body mesh) that characterizes the patient's pose, posture, and / or body shape when the 2D medical image 104 is captured. Based on the patient's pose, posture, and / or body shape characterized by the 3D human body model, the relative pose and orientation of the 3D anatomical model and the sensing devices, and the relative pose and / or orientation of the sensing devices and the medical scanner (e.g., a C-arm X-ray machine) used to capture the 2D medical image 104, the location of the injection site described herein can be automatically determined on the 3D anatomical model 102 and used to facilitate (e.g., refine) the registration of the 3D anatomical model 102 and the 2D medical image 104. In this example, CT images used to generate the 3D anatomical model 102 can be registered with the human body model (e.g., a 3D body mesh) to improve the accuracy of injection site determination.

[0020] Figure 2 An example is illustrated of a 3D anatomical model 202 (e.g., according to one or more embodiments of the present disclosure). Figure 1 Anatomical model 102) and 2D perspective image 204 (e.g., Figure 1 Examples of registration and / or fusion of 2D medical images 104. As shown, registration and / or fusion can be facilitated by user input indicating the location of contrast agent injection sites (e.g., catheter insertion sites) in the 2D fluoroscopic image 204 and / or on the 3D anatomical model 202. User input can be provided by a user based on the 2D fluoroscopic image 204. For example, an interface (e.g., a graphical user interface) can be provided, through which the 2D fluoroscopic image 204 can be displayed, and the user can identify and / or indicate (e.g., based on the contrast reflected by the 2D fluoroscopic image) at least one vessel 206 where contrast agent (e.g., contrast dye) can be injected. In response to identifying such a vessel 206 based on the 2D fluoroscopic image 204, the user (e.g., using the same interface) can indicate the location of the contrast agent injection site (e.g., catheter tip) and / or at least one vessel associated with the contrast agent injection in the 2D fluoroscopic image 204 and / or on the 3D anatomical model. Figure 2 As illustrated in the examples, a user can do this by drawing a bounding box 208 around a specific region of the 3D anatomical model 202 (and / or in the 2D perspective image 204) to indicate the location of the injection site and / or the blood vessels associated with the injection site. In some examples, the 3D anatomical model 202 and / or the 2D perspective image 204 can be automatically segmented into multiple segments, and key markers can be automatically labeled (e.g., using a pre-trained AI model), allowing the user to select one of the segments and / or key markers to indicate that the selected segment or marker includes the injection site or the blood vessels associated with the injection site.

[0021] Injection sites and / or blood vessels determined based on user input can be used as reference markers or features to facilitate (e.g., refine) the registration and / or fusion 210 of the 3D anatomical model 202 and the 2D perspective image 204. For example, the problem of image registration can be formulated as finding the geometric transformation T: I F →I M To move image I M With fixed image I F Alignment. This alignment can be achieved by optimizing a similarity metric L (e.g., a loss function) based on feature correspondences (e.g., lines, angles, contours, etc.) between the fixed and moving images. Thus, by determining the injection site and / or associated vessels on the 3D anatomical model 202 (e.g., which can be considered a 3D image) and the 2D perspective image 204, equivalent features on the two images can be detected more quickly and accurately (e.g., using deep neural networks to learn), leading to improved alignment of equivalent features (e.g., through translation, rotation, scaling, etc.) and better results in image registration / fusion.

[0022] Figure 3 An example is shown using an artificial neural network (ANN) 302 to register two images I. mov (e.g., source image) and I fix (e.g., target image, such as) Figure 2 Example of a 3D anatomical model (202). Image I fix and I mov One of them can be a 2D medical image, such as Figure 1 2D medical images 104 or Figure 2 204, while image I fix and I mov Another one could be a 3D image, such as Figure 1 3D anatomical model 102 or Figure 2 202. Neural network 302 can be configured to receive image I. fix and I mov (For example, as input), image I mov From the moving image domain (e.g., with image I) mov (Associated) transformation to a fixed image domain (e.g., with image I) fix (Associated), and generate image I reg (For example, as image I) mov (spatial transformation version), this image I reg Similar to image I fix (For example, in I) fix with I reg There is a minimized dissimilarity 304 between them. A neural network 304 can be trained to determine the method used to process image I. mov Transform into image I reg Multiple transformation parameters θ T This operation can be explained by the following formula:

[0023] I reg =I mov (θ(x)) (1)

[0024] Where x can represent the coordinates in the moving image domain, θ(x) can represent the mapping of x to the fixed image domain, and I mov (θ(x)) can represent one or more grid sampling operations (e.g., using sampler 306). θ can include parameters associated with an affine transformation model that allows translation, rotation, scaling, and / or skew of the input image. θ can also include parameters associated with a deformable field (e.g., a dense deformable field) that allows deformation of the input image. For example, θ can include stiffness parameters, B-spline control points, deformable parameters, etc.

[0025] In embodiments of this disclosure, the neural network 302 can be configured to determine the value θ of the transformation parameter based on a set of initial values ​​θ0 of the transformation parameter and an integral of an update (e.g., gradient update) of the transformation parameter determined by the neural network 302. T The initial values ​​θ0 of the transformation parameters can be obtained from a normal distribution (e.g., randomly), for example, based on an existing image registration model. In the example, neural network 302 may include a neural ordinary differential equation (ODE) network configured to determine the transformation parameters θ by solving the ordinary differential equations associated with the transformation parameters. T Such a neural ODE network may include one or more ODE layers or ODE blocks, each of which can be configured to determine (e.g., predict or estimate) a corresponding update (e.g., gradient update) to the transformation parameters based on the current or current state (e.g., current value) of the transformation parameters. For example, neural network 302 may be configured to determine the corresponding update to the transformation parameters through one or more iterations (e.g., through one or more ODE layers or blocks), and each update may be determined based on the current state of the transformation parameters associated with each of the one or more iterations. The update can then be used to obtain (e.g., derive) the final value θ of the transformation parameters using an ODE solver. T For example, an ODE solver can be used to integrate a corresponding update (e.g., gradient update) determined (e.g., predicted) by one or more ODE layers or blocks, and apply the integral of the update to the initial parameter value θ0 to derive the final value θ. T .

[0026] The operation of the aforementioned neural ODE network can be explained below. The image registration task is formulated as follows: Where θ can represent the transformation parameter described in this paper, and C can represent the parameter designed to indicate I. fix with I mov The loss (or cost) function (θ(x)) of the dissimilarity 304 can be used to derive the transformation parameter θ using gradient descent-based optimization techniques (such as those exemplified below):

[0027]

[0028] Where t can represent the iteration in the optimization process, and η t It can represent the optimization step size, and It can represent the current or present state θ of the transformation parameters. t The derivative of the loss function C (for example, representing the current value).

[0029] A neural ODE network can be trained to predict the expression shown in equation (2). The corresponding update (e.g., gradient update), and the update predicted by such a network, can be expressed as:

[0030] θ t+1 =θ t +f(θ t , ρ t (3)

[0031] Where f can be represented by μ t A parameterized neural ODE network. For sufficiently small t, updates can occur continuously (e.g., approximately continuously), as represented by the following ordinary differential equation:

[0032]

[0033] Among them, f μ It can represent a neural ODE network parameterized with μ.

[0034] Therefore, starting from the initial parameter value θ0, a neural ODE network can be trained to, for example, utilize an ODE solver to generate an output θ corresponding to the solution of the ordinary differential equation shown in (4). T (For example, the final values ​​of the transformation parameters) (For example, the function of an ODE network can be understood as solving an initial value problem over a time interval [0, T]). When the input of a neural ODE network includes images (such as...) Figure 3 Image I shown fix and I mov When the gradient update predicted by the neural ODE network (e.g., by the layers of the ODE network) is in the form of:

[0035]

[0036] Furthermore, the solution to the ordinary differential equation can be:

[0037]

[0038] and

[0039] θ t+dt =θ t +f μ (I mov (θ t (x)), I fix ,t)*dt (7)

[0040] (6) can represent the continuous derivation of parameters at time T, and (7) can represent the step in the derivation process (e.g., from t to t+dt).

[0041] Once the transformation parameters θ are obtained TThe values, which can be used, for example, via one or more resampling operations that can be performed using sampler 306, to resample the input image I. mov Transform into I reg During the training of the neural ODE network, image I can be used. reg With input image I fix The images are compared, and the dissimilarity 304 between them can be determined based on a loss function (e.g., a loss function based on Euclidean distance, cross-correlation, normalized cross-correlation, etc.). The dissimilarity 304 can then be used to guide the tuning of network parameters, for example, with the aim of minimizing the dissimilarity 304.

[0042] Figure 4 The neural ODE network 402 is shown (e.g., Figure 3 An example structure of a neural network 302 is shown, which can be configured to determine parameters (e.g., transformation parameters) for registering a first medical image with a second medical image. The figure illustrates an ODE layer or block 404 associated with a hidden state θ(t) of the transformation parameters; however, those skilled in the art will understand that the neural ODE network 402 can include multiple such layers or blocks, and the transformation parameters can be tuned by a series of transformations involving multiple hidden states. As shown, the ODE layer or block 404 can include one or more convolutional layers 406, one or more batch normalization (BN) layers 408, one or more activation functions 410 (e.g., rectified linear unit (ReLU) activation functions), one or more pooling layers (not shown), and / or one or more fully connected layers (not shown). Each convolutional layer 406 can include multiple convolutional kernels or filters with corresponding weights, configured to receive images (e.g., source image I described herein) from images received by the neural ODE network 402. mov and / or target image I fix Feature extraction. The operation of convolutional layer 406 can be followed by batch normalization (BN) (e.g., via BN layer 408) and / or linear or non-linear activation (e.g., using ReLU 410), and the features extracted by convolutional layer 406 can be downsampled via a shrinking path (e.g., including one or more pooling layers and / or one or more fully connected layers) to reduce redundancy and / or size of the extracted features. In some examples, the downsampled features can then be processed via a dilation path (e.g., including one or more transposed convolutional layers and / or one or more non-pooling layers), during which the features can be upsampled to a higher resolution.

[0043] The neural ODE network 402 can determine transformation parameters for spatial alignment of an image of interest using features extracted via the convolution operations described herein. The neural ODE network 402 can predict the transformation parameters, for example, by means of the hidden states θ(t) of the transformation parameters successively transformed by one or more ODE layers or blocks 404. Each transformation may correspond to transforming the hidden state of the parameter from θ(t) to θ(t+Δt), where Δt may represent the transformation or optimization step or size. When Δt approaches zero (e.g., when the transformation step is sufficiently small), the final state of the transformation parameters (e.g., θ(t=T)) can be obtained by solving the ODE associated with the transformation parameters (e.g., as illustrated in Equations 4-7). An ODE solver can be used to evaluate the amount of transformation (e.g., adjustment) determined and / or applied by the ODE block 404. (For example, as illustrated in Equations 6 and / or 7 as described herein), and the fault tolerance level of the ODE solver can determine the number of transformations and / or evaluations to be performed before obtaining the final values ​​of the transformation parameters. The ODE solver can be implemented using various numerical analysis techniques. For example, the ODE solver may include an Euler solver (e.g., based on the Euler method for solving ODEs), a Runge-Kutta (RK) solver such as the RK2 or RK4 solver (e.g., based on the Runge-Kutta (RK) method for solving ODEs), an adaptive step-size solver (e.g., based on an adaptive step-size method for solving ODEs), etc. The ODE solver can be a standalone solver (e.g., separate from the neural ODE network 402) or it can be part of the neural ODE network 402 (e.g., the ODE solver itself can be learned through training). The fault tolerance level of the ODE solver can be configurable (e.g., as a hyperparameter of the neural ODE network 402) and can be assigned the same or different values ​​for training and inference purposes.

[0044] The injection site indication described herein can be used to guide feature extraction and / or transform parameter determination processes associated with image registration and / or fusion. For example, the initial transform parameter value θ0 for registering the 3D anatomical model and the 2D perspective image described herein can be determined based on the location of the injection site in a 2D perspective image. This location can be determined using a convolutional neural network (such as a convolutional neural network with a U-shaped encoder-decoder architecture). The neural ODE network can be trained using a constraint loss, under which updated transform parameters that deviate significantly from the initial transform parameter value θ0 can be penalized.

[0045] Figure 5A flowchart illustrating an example process 500 for training a neural network (e.g., an AI model implemented by a neural network) to perform one or more tasks described herein. As shown, the training process 500 may include: at 502 initializing the execution parameters of the neural network (e.g., weights associated with the individual layers of the neural network), for example by sampling from a probability distribution or by replicating the parameters of another neural network with a similar structure. The training process 500 may also include: at 504 processing the input (e.g., a training image) using the currently assigned parameters of the neural network; and at 506 predicting the desired outcome (e.g., image transformation parameters). At 508, the prediction may be compared to a gold standard to determine, for example, a loss associated with the prediction based on a loss function (such as the mean squared error between the prediction and the gold standard, L1 norm, L2 norm, etc.). At 510, this loss may be used to determine whether one or more training termination criteria are met. For example, a training termination criterion may be determined to be met if the loss is below a threshold or if the change in loss between two training iterations is below a threshold. If the termination criterion is met at 510, training can end; otherwise, at 512, for example, before training returns to 506, the currently assigned network parameters can be adjusted by backpropagating the gradient descent of the loss function through the network.

[0046] For the sake of simplicity, the training steps are depicted and described in a specific order herein. However, it should be understood that training operations can occur in various orders, simultaneously, and / or with other operations not presented or described herein. Furthermore, it should be noted that not all operations that may be included in the training method are depicted and described herein, and not all exemplified operations need to be performed.

[0047] The systems, methods, and / or apparatuses described herein may be implemented using one or more processors, one or more storage devices, and / or other suitable auxiliary devices (such as display devices, communication devices, input / output devices, etc.). Figure 6An example device 600 is illustrated that can be configured to perform the tasks described herein. As shown, device 600 may include a processor (e.g., one or more processors) 602, which may be a central processing unit (CPU), graphics processing unit (GPU), microcontroller, reduced instruction set computer (RISC) processor, application-specific integrated circuit (ASIC), application-specific instruction set processor (ASIP), physical processing unit (PPU), digital signal processor (DSP), field-programmable gate array (FPGA), or any other circuitry or processor capable of performing the functions described herein. Device 600 may also include communication circuitry 604, memory 606, mass storage device 608, input device 610, and / or communication link 612 (e.g., communication bus) through which one or more components shown in the figures exchange information.

[0048] Communication circuitry 604 can be configured to send and receive information using one or more communication protocols (e.g., TCP / IP) and one or more communication networks, including local area networks (LANs), wide area networks (WANs), the Internet, and wireless data networks (e.g., Wi-Fi, 3G, 4G / LTE, or 5G networks). Memory 606 may include a storage medium (e.g., a non-transitory storage medium) configured to store machine-readable instructions that, when executed, cause processor 602 to perform one or more functions described herein. Examples of machine-readable media may include volatile or non-volatile memory, including but not limited to semiconductor memory (e.g., electrically programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM)), flash memory, etc.). Mass storage device 608 may include one or more disks, such as one or more internal hard disks, one or more removable disks, one or more magneto-optical disks, one or more CD-ROMs or DVD-ROMs, etc., on which instructions and / or data may be stored for operation of processor 602. Input device 610 may include a keyboard, mouse, voice-controlled input device, touch-sensitive input device (e.g., touch screen), etc., for receiving user input from device 600.

[0049] It should be noted that device 600 can operate as a standalone device or can be connected to other computing devices (e.g., networked or clustered) to perform the tasks described herein. And even in Figure 6 Only one example of each component is shown in the figure, and those skilled in the art will understand that device 600 may include multiple instances of one or more components shown in the figure.

[0050] Although this disclosure has been described according to certain embodiments and generally associated methods, changes and variations of the embodiments and methods will be apparent to those skilled in the art. Therefore, the above description of exemplary embodiments does not limit this disclosure. Other changes, substitutions, and modifications are possible without departing from the spirit and scope of this disclosure. Furthermore, unless specifically stated otherwise, discussions using terms such as “analyze,” “determine,” “enable,” “identify,” and “modify” refer to the actions and processes of a computer system or similar electronic computing device that manipulate and transform data representing physical (e.g., electronic) quantities within the registers and memories of the computer system into other data representing physical quantities within the computer system's memory or other such information storage, transmission, or display devices.

[0051] It should be understood that the above description is intended to be illustrative and not restrictive. Many other embodiments will become apparent to those skilled in the art after reading and understanding the above description.

Claims

1. A method for medical image fusion, the method comprising: Obtain a three-dimensional (3D) anatomical model representing one or more blood vessels in the patient; Obtain two-dimensional (2D) perspective images of all or a subset of the one or more blood vessels, wherein the 2D perspective images indicate that at least one of the one or more blood vessels has been injected with a contrast agent; The 3D anatomical model is overlaid onto the 2D perspective image, and an image depicting the overlaid 3D anatomical model and the 2D perspective image is generated; Determine the injection site of the contrast agent in the 2D fluoroscopic image; The injection site is determined on the 3D anatomical model; the 2D perspective image is obtained using a medical imaging device, and wherein determining the injection site on the 3D anatomical model includes: determining the patient's pose and body shape based on information provided by a sensing device; determining the relative pose and orientation of the 3D anatomical model and the sensing device based on the patient's pose and body shape; determining the relative pose and orientation of the medical imaging device and the sensing device; and determining the injection site on the 3D anatomical model based at least on the patient's pose or body shape, the relative pose and orientation of the 3D anatomical model and the sensing device, and the relative pose and orientation of the medical imaging device and the sensing device; and The 3D anatomical model is registered with the 2D perspective image based at least on the corresponding position of the injection site in the 2D perspective image and the 3D anatomical model, such that the 3D anatomical model is approximately aligned with the 2D perspective image with respect to at least one of the one or more blood vessels associated with the injection site.

2. The method according to claim 1, further comprising: The image depicting the superimposed 3D vascular model and the 2D perspective image is displayed on a monitor or via a virtual reality headset.

3. The method according to claim 1, wherein, The injection site is determined on the 3D anatomical model based on user input indicating the location of the injection site on the model.

4. The method according to claim 3, wherein, The user input includes a bounding box drawn on the 3D anatomical model, which marks the injection site.

5. The method according to claim 1, wherein, Determining the injection site on the 3D anatomical model includes: The 3D anatomical model is divided into multiple segments, each segment representing the patient's anatomical structure; Receive user input, the user input indicating that one or more of the plurality of segments include the injection site; and The injection site is determined on the 3D anatomical model based on the user input.

6. The method according to claim 1, further comprising: A 2D mask associated with one or more blood vessels of the patient is generated based on the 2D perspective image, and the 2D mask is registered with the 3D anatomical model based on the injection site.

7. The method according to claim 1, further comprising: The medical device implanted inside one of the patient's one or more blood vessels is detected based on the 2D perspective image. Generate a 3D model of the medical device based on a predefined device model; and The 3D model of the medical device is overlaid onto a 3D image depicting one of the blood vessels into which the medical device is implanted.

8. A computer program product comprising instructions that, when run on a computer, cause the computer to perform the method as described in any one of claims 1-7.

Images