Providing projected images

Abstract

A computer-implemented method for providing projection images representing an interventional device is provided. The method includes inputting x-ray projection image data into a neural network. The x-ray projection image data represents the interventional device from a first perspective of a projection x-ray imaging system relative to the interventional device. In response to the input, the neural network generates predicted x-ray projection image data representing the interventional device from a second perspective of the projection x-ray imaging system relative to the interventional device. The neural network is trained to generate the predicted x-ray projection image data using training data including a plurality of x-ray projection training images representing the interventional device from the first perspective and, for each x-ray projection training image, a corresponding ground truth x-ray projection training image representing the interventional device from the second perspective.

Description

[Technical Field]

[0001] SUMMARY A computer-implemented method, computer program product, and system are disclosed that relate to providing projection images representative of an interventional device. [Background technology]

[0002] Many interventional medical procedures are performed using projection x-ray imaging systems. The two-dimensional "projection" images provided by such imaging systems facilitate navigation of interventional devices, such as guidewires and catheters, within anatomical structures. However, it is difficult to mentally visualize the three-dimensional shape of the interventional device from the projection images. This hinders navigation of the interventional device.

[0003] To overcome this limitation, biplane projection X-ray imaging systems have been developed. These systems can simultaneously acquire projection images representing an interventional device from two different, often orthogonal, viewpoints. The information provided by the different viewpoints makes it easier for users to visualize its three-dimensional shape. However, such imaging systems have a large footprint. Acquiring images from a second viewpoint also increases the amount of X-ray radiation delivered to the patient.

[0004] Alternative approaches are also available to overcome the challenge of visualizing the 3D shape of an interventional device from projection images. One approach is to acquire multiple projection images of the interventional device from different viewpoints using a projection X-ray imaging system. However, this approach has the disadvantage of requiring the projection X-ray imaging system to be repositioned, which also increases the amount of X-ray radiation delivered to the patient. Another approach involves tracking the 3D shape of the interventional device. Various tracking systems can be used to track the 3D shape. The shape information augments the projection images generated by the projection X-ray imaging system. However, this approach incurs increased procedural complexity due to the need to set up a tracking system and align the coordinate system of the tracking system with the coordinate system of the projection X-ray imaging system.

[0005] Therefore, there remains a need for improvements that facilitate understanding the three-dimensional shape of interventional devices from X-ray projection images.

[0006] Document WO 2022 / 106377 A1 discloses a computer-implemented method for providing a neural network for predicting the three-dimensional shape of an interventional device placed within a vascular region. The method includes training the neural network to predict the three-dimensional shape of the interventional device constrained by the vascular region from received X-ray image data and received volumetric image data. The training includes constraining adjustment of parameters of the neural network so that the three-dimensional shape of the interventional device predicted by the neural network fits within the three-dimensional shape of the vascular region represented by the received volumetric image data. This document also discloses a computer-implemented method for predicting the three-dimensional shape of an interventional device placed within a vascular region. The method includes receiving volumetric image data representing a three-dimensional shape of a vascular region; receiving X-ray image data representing one or more two-dimensional projections of an interventional device within the vascular region; inputting the received X-ray image data and the received volumetric image data into a neural network trained to predict the three-dimensional shape of the interventional device constrained by the vascular region from the received X-ray image data and the received volumetric image data; and predicting, in response to the input, the three-dimensional shape of the interventional device constrained by the vascular region using the neural network from the received X-ray image data and the received volumetric image data. Summary of the Invention [Problem to be solved by the invention]

[0007] There remains a need for improvements that facilitate understanding the three-dimensional shape of interventional devices from x-ray projection images. [Means for solving the problem]

[0008] According to one aspect of the present disclosure, a computer-implemented method for providing a projection image representative of an interventional device is provided, the method comprising: receiving x-ray projection image data representing the interventional device from a first perspective of a projection x-ray imaging system relative to the interventional device; inputting the X-ray projection image data into a neural network; generating predicted x-ray projection image data representing the interventional device from a second viewpoint of a projection x-ray imaging system relative to the interventional device, the second viewpoint being different from the first viewpoint, using a neural network; Including, The neural network is trained to generate predicted X-ray projection image data representing the interventional device from the second perspective using training data having a plurality of X-ray projection training images representing the interventional device from a first perspective and, for each X-ray projection training image, a corresponding ground truth X-ray projection training image representing the interventional device from a second perspective.

[0009] In this method, predicted X-ray projection image data is generated from X-ray projection image data representing the interventional device from a first viewpoint of the projection X-ray imaging system. The predicted X-ray projection image data represents the interventional device from a second viewpoint of the projection X-ray imaging system. As a result, this method provides X-ray projection data for the interventional device from different viewpoints. The two viewpoints facilitate a user's visualization of the three-dimensional shape of the interventional device. Furthermore, because the predicted X-ray projection image data for the second viewpoint is generated without the need to actually acquire X-ray projection image data from this viewpoint, this method facilitates a user's visualization of the three-dimensional shape of the interventional device without increasing the amount of X-ray radiation delivered to the patient. In contrast to the method disclosed in WO 2022 / 106377 A1, neither the training of the neural network nor the predictions made by the neural network require volumetric image data.

[0010] Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples which proceeds with reference to the accompanying drawings. [Brief explanation of the drawings]

[0011] [Figure 1] 1 is a flowchart illustrating an example of a computer-implemented method for providing a projection image representative of an interventional device, according to some aspects of the present disclosure. [Figure 2] 2 is a schematic diagram illustrating an example of a system 200 for providing projection images representing an interventional device, according to some aspects of the present disclosure. FIG. [Figure 3] 1 is an example of X-ray projection image data 120a representing an interventional device 110 from a first viewpoint 130a of a projection X-ray imaging system 140 relative to the interventional device, according to some aspects of the present disclosure. [Figure 4] 10 is an example of a predicted X-ray projection image 120Pb representing the interventional device 110 from a second viewpoint 130b of a projection X-ray imaging system 140 relative to the interventional device, according to some aspects of the present disclosure. [Figure 5] 1 is a first example of a neural network 150 trained to generate predicted X-ray projection image data 120Pb representing an intervention device 110 from a first viewpoint 130a using training data having an X-ray projection training image 120a' representing the intervention device 110 from a second viewpoint 130b and a corresponding ground truth X-ray projection training image 120b' representing the intervention device 110 from a second viewpoint 130b, according to some embodiments of the present disclosure. [Figure 6] 10 is a second example of training a neural network 150 to generate predicted X-ray projection image data 120Pb representing an intervention device 110 from a first viewpoint 130a using training data having an X-ray projection training image 120a' representing an intervention device 110 from a first viewpoint 130a and a corresponding ground truth X-ray projection training image 120b' representing the intervention device 110 from a second viewpoint 130b, according to some embodiments of the present disclosure. [Figure 7]1 is an example of a neural network 150 trained to generate constrained predicted X-ray projection image data 120Pb representing an interventional device 110 from a second viewpoint 130b, where the predicted X-ray projection image data 120Pb is constrained by angiography image data 170b, I2, in accordance with some aspects of the present disclosure. [Figure 8] 10 is a schematic diagram including predicted X-ray projection image data 120Pb, 120Pc representing the intervention device 110 from a second viewpoint 130b and a third viewpoint 130c of a projection X-ray imaging system 140, respectively, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0012] Examples of the present disclosure are provided with reference to the following description and drawings. In this description, for purposes of explanation, many specific details of particular examples are set forth. Reference herein to an "example," "implementation," or similar language means that a feature, structure, or characteristic described in connection with an example is included in at least that example. It should also be understood that features described in connection with one example may be used in another example, and that not all features are necessarily repeated in each example for the sake of brevity. For example, features described in connection with a computer-implemented method may be implemented in a computer program and in a system in a corresponding manner.

[0013] In the following description, reference is made to a computer-implemented method that includes providing projection images representing an interventional device. In some examples, the interventional device is an intravascular device. For example, reference is made to examples in which the interventional device depicted in the projection images is a guidewire. However, it should be understood that the computer-implemented methods disclosed herein may similarly be used to provide projection images representing other types of interventional devices. For example, the interventional device may be used to perform medical procedures on other portions of the anatomy, including the lungs and associated airways. As some examples, the interventional device may alternatively be a catheter, a thrombectomy device, an intravascular ultrasound (IVUS) imaging device, an endobronchial ultrasound (EBUS) device, an optical coherence tomography (OCT) imaging device, a blood pressure and / or flow sensor device, a TEE probe, etc.

[0014] It should be noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including stored computer-readable instructions that, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program. The computer program may be provided by dedicated hardware or hardware capable of executing software in association with appropriate software. When provided by a processor, the functions of the method features may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. One or more functions of the method features may be provided by a processor shared within a networked processing architecture, such as, for example, a client / server architecture, a peer-to-peer architecture, the Internet, or the cloud.

[0015] Explicit use of the terms "processor" or "controller" should not be construed as exclusively referring to hardware capable of executing software, and can implicitly include, but is not limited to, digital signal processor "DSP" hardware, read-only memory "ROM" for storing software, random access memory "RAM," non-volatile storage, and the like. Furthermore, examples of the present disclosure can take the form of a computer-usable storage medium, or a computer program accessible from a computer-readable storage medium, the computer program providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable storage medium or computer-readable storage medium can be any device that can have, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or device or propagation medium. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer diskettes, random access memory "RAM," read-only memory "ROM," rigid magnetic disks, and optical disks, current examples of which include compact disk-read-only memory "CD-ROM," compact disk-read / write "CD-R / W," Blu-Ray, and DVD.

[0016] As noted above, there remains a need for improvements that facilitate understanding the three-dimensional shape of interventional devices from x-ray projection images.

[0017] FIG. 1 is a flowchart illustrating an example of a computer-implemented method for providing projection images representing an interventional device according to some aspects of the present disclosure. FIG. 2 is a schematic diagram illustrating an example of a system 200 for providing projection images representing an interventional device according to some aspects of the present disclosure. The system 200 illustrated in FIG. 2 includes one or more processors 210. It should be noted that the operations described in connection with the method illustrated in FIG. 1 may also be performed by one or more processors 210 of the system 200 illustrated in FIG. 2. Similarly, the operations described in connection with one or more processors 210 of the system 200 may also be performed in the method described with reference to FIG. 1.

[0018] Referring to FIG. 1 , a computer-implemented method for providing a projection image representative of an interventional device 110 includes: receiving S110 X-ray projection image data 120a representing the interventional device 110 from a first viewpoint 130a of a projection X-ray imaging system 140 relative to the interventional device; Step S120 of inputting the X-ray projection image data 120a into the neural network 150; In response to the input, predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b of the projection x-ray imaging system 140 relative to the interventional device is generated. P a step S130 of generating a second viewpoint 130b, where the second viewpoint 130b is different from the first viewpoint 130a; Including, The neural network 150 generates a plurality of X-ray projection training images 120a' representing the interventional device 110 from a first perspective 130a. 1..n and each X-ray projection training image 120a' 1..n , and a corresponding ground truth X-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n and using the training data having the predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b. PIt is trained to generate b.

[0019] 1, X-ray projection image data 120a is received in act S110. The X-ray projection image data 120a represents the intervention device 110 from a first perspective 130a of a projection X-ray imaging system 140 relative to the intervention device.

[0020] The X-ray projection image data 120a received in operation S110 may be generated by a projection X-ray imaging system. A projection X-ray imaging system typically includes a support arm, such as a so-called "C-arm" or "O-arm," that supports an X-ray source-detector configuration. A projection X-ray imaging system may alternatively include a support arm having a shape different from these examples. A projection X-ray imaging system may alternatively include a C-less system in which the X-ray source and detector are not paired and can be positioned independently of each other. A projection X-ray imaging system typically generates the X-ray projection image data with the support arm held in a stationary position relative to the imaging region during image data acquisition. The X-ray projection image data 120a may be generated, for example, by the projection X-ray imaging system 140 shown in FIG. 2. As an example, the X-ray projection image data may be generated by a Philips Azurion 7 X-ray imaging system commercially available from Philips Healthcare of Vest, The Netherlands.

[0021] In general, the x-ray projection image data 120a received in act S110 may represent one or more still images or may represent a temporal sequence of images. The temporal sequence of images may be generated substantially in real time. Thus, the x-ray projection image data 120a may represent the interventional device 110 in real time.

[0022] In general, the x-ray projection image data 120a received in act S110 may represent any region of interest within an anatomical structure. For example, the region of interest may be a portion of the brain, heart, lungs, etc. In some examples, the interventional device 110 is positioned within the vasculature, and the x-ray projection image data 120a also represents the vasculature.

[0023] The X-ray projection image data 120a received in operation S110 may be received from a variety of sources. For example, the X-ray projection image data 120a may be received from a projection X-ray imaging system, such as the projection X-ray imaging system 140 shown in FIG. 2, or may be received, for example, from a computer-readable storage medium or from the Internet or the cloud. The X-ray image data may be received by one or more processors 210 shown in FIG. 2. The X-ray projection image data 120a may be received via any form of data communication, including wired communication, optical communication, and wireless communication. As some examples, when wired or optical communication is used, communication may be via signals transmitted over electrical or optical cables, and when wireless communication is used, communication may be via, for example, RF or optical signals.

[0024] As described above, the X-ray projection image data 120a received in operation S110 represents the interventional device 110 from a first viewpoint 130a of the projection X-ray imaging system 140 relative to the interventional device. FIG. 3 is an example of X-ray projection image data 120a representing the interventional device 110 from a first viewpoint 130a of the projection X-ray imaging system 140 relative to the interventional device, according to some aspects of the present disclosure. The X-ray projection image data 120a shown on the left side of FIG. 3 includes the interventional device 110 in the form of a guidewire positioned within the brain. A corresponding first viewpoint 130a of the projection X-ray imaging system 140 relative to the interventional device is shown on the right side of FIG. 3. The first viewpoint 130a can generally be any viewpoint relative to the interventional device 110. The first viewpoint 130a can be defined in any coordinate system. For example, the viewpoint may be defined, for example, in a spherical or Cartesian coordinate system. A first viewpoint 130a of the projection X-ray imaging system 140 is defined relative to the interventional device 110. A known transformation between the orientation of the interventional device and the patient can be used to further define the first viewpoint 130a relative to the patient. For example, in the example shown in FIG. 3, the first viewpoint 130a also represents a so-called anterior-posterior "AP" view of the brain.

[0025] 1, in operation S120, the x-ray projection image data 120a is input to a neural network 150. The neural network 150 generates a plurality of x-ray projection training images 120a' representing the interventional device 110 from a first perspective 130a. 1..n and each X-ray projection training image 120a' 1..n , and a corresponding ground truth X-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n and using the training data having the predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b. P It is trained to generate b.

[0026] The neural network 150 may be implemented using various types of architectures. For example, the neural network 150 may have a convolutional neural network (CNN), an autoencoder, a transforming autoencoder (TAE), a generative adversarial network (GAN), a capsule network, or a recurrent network architecture. The neural network may also be implemented using a combination of such architectures. In an example where the X-ray projection image data 120a input to the neural network 150 represents a temporal sequence of images, the neural network may include a recurrent neural network (RNN) or a long short-term memory (LSTM) cell. Such cells may be used to generate an output based on both the current image and previous images in the temporal sequence. The use of such cells enables the neural network 150 to obtain temporal information from the X-ray projection image data 120a.

[0027] In one example, the neural network 150 has a TAE architecture. This example uses training data including X-ray projection training images 120a′ representing the interventional device 110 from a first viewpoint 130a and corresponding ground truth X-ray projection training images 120b′ representing the interventional device 110 from a second viewpoint 130b to generate predicted X-ray projection image data 120b representing the interventional device 110 from the second viewpoint 130b, in accordance with some aspects of the present disclosure. P A first example of a neural network 150 trained to generate b will be described with reference to FIG.

[0028] In the example described with reference to Figure 5, the neural network 150 includes a transform autoencoder. e and decoder 150 d Includes: Encoder 150 e , and decoder 150 d may include one or more convolutional layers. eis trained to learn a latent space representation LR of the input X-ray projection image data 120a, and the latent space representation of the input X-ray projection image data 120a is transformed by the orientation data 160, T, and the decoder 150 d When decoded by the encoder 150, the decoded latent space representation represents the interventional device 110 from the second viewpoint 130b. Consequently, during inference, the instance of the X-ray projection image data 120a is input to the neural network 150. e generates a latent space representation LR of the input X-ray projection image data 120a. A transformation T is then applied to the latent space representation to provide a transformed latent space representation TLR. The transformed latent space representation TLR is then output to the decoder 150 d , and the decoder 150 outputs a decoded latent space representation representing the intervention device 110 from the second viewpoint 130b.

[0029] 5, the transformation T is a spatial transformation that represents the change in viewpoint between the first viewpoint 130a and the second viewpoint 130b. The transformation T is used to represent ... 1..n and for each x-ray projection training image 120a', a corresponding ground truth x-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n and thus are associated with the training data. In general, the neural network 150 can be trained using any transformation T, which can represent any change in perspective between the first perspective 130a and the second perspective 130b. As an example, the first perspective 130a and the second perspective 130b can be orthogonal to each other. This change in perspective is useful in many medical procedures.

[0030] An example of a transformation T is given below in Equation 1 for a transformation that represents a rotation of the viewpoint of a projection X-ray imaging system in the xy plane through 90 degrees about the y axis in Cartesian space.

number

[0031] If prediction is desired for a single change in viewpoint, the neural network 150 may be trained using a set of X-ray projection training images representing the interventional device 110 from a first viewpoint 130a and a second viewpoint 130b and the corresponding transformation T. During inference, the same transformation T is applied by the neural network to the latent space representation LR. The transformation T may be hidden from the user and applied to the latent space representation LR within the neural network during inference, or it may be applied as an input to the neural network during inference.

[0032] If predictions are desired for multiple changes in viewpoint, multiple neural networks may be trained, each trained using training data and corresponding transformations for different changes in viewpoint. Alternatively, a single neural network may be trained to make predictions for multiple changes in viewpoint. In this case, the neural network 150 is trained using training data including a set of X-ray projection training images representing the interventional device 110 from a first viewpoint 130a and a second viewpoint 130b, and the corresponding transformations, for each change in viewpoint.

[0033] When a single neural network is trained to make predictions for multiple changes in viewpoint, during training, a transformation T is applied as an input to the neural network. The neural network then learns to base its predictions on the desired transformation T. In inference, a transformation T for a desired change in viewpoint is input to the neural network, and a prediction is made for that transformation. In this case, the transformation T may be selectable automatically or based on user input. If the transformation is automatically selectable, the neural network may be controlled to make sequential predictions for different changes in viewpoint. Thus, the neural network may be controlled to make near real-time predictions for multiple different changes in viewpoint.

[0034] In the examples described above where a transformation T is input to a neural network, the transformation T is input to the neural network in the form of orientation data. In these examples, the method described with reference to FIG. receiving orientation data 160, T, defining a second viewpoint 130b of the projection X-ray imaging system 140; inputting the received orientation data 160, T, into a neural network 150; Including, The neural network 150 generates predicted x-ray projection image data 120 representing the interventional device 110 from the second perspective 130b based on the orientation data 160, T. P It is trained to generate b.

[0035] In some examples, before inputting the X-ray projection image data 120a into the neural network 150, a segmentation operation is performed on the X-ray projection image data 120a to identify interventional devices. This is shown in the lower portion of FIG. 5, labeled "Segment." Various segmentation algorithms, including model-based segmentation, watershed-based segmentation, region growing, level setting, graph cut, etc., may be used for this purpose. The neural network may also be trained to segment the X-ray projection image data 120a to identify interventional devices. Identifying interventional devices in this manner before inputting the X-ray projection image data 120a into the neural network facilitates the neural network's focus on interventional devices, thereby improving the predictions made by the neural network. In one example, the X-ray projection image data 120a input into the neural network represents only interventional devices. Data representing only interventional devices may also be extracted from the segmented X-ray projection image data 120a. Therefore, features representing tissue, bone, and vasculature may be removed from the x-ray projection image data 120a input to the neural network 150. Such features may contain large variations and interfere with the predictions of the neural network 150, and removing such features may result in improved reliability of the neural network's predictions.

[0036] Generally, the neural network 150 Receive the training data, A plurality of X-ray projection training images 120a' in the training data representing the interventional device 110 from a first perspective 130a. 1..n For each of the X-ray projection training image 120a' 1..n is input to neural network 150, The neural network 150 is used to generate a predicted X-ray projection image 120 representing the interventional device 110 from a second perspective 130b. P b 1..n Generate Predicted X-ray projection images 120 P b 1..n and a corresponding ground truth X-ray projection training image 120b' representing the interventional device 110 from a second perspective 130b. 1..n and adjusting parameters of the neural network 150 based on the difference between Repeat the input, generation, and refinement until a stopping criterion is met. 130b, thereby generating predicted X-ray projection image data 120 representing the interventional device 110 from the second viewpoint 130b. P It is trained to generate b.

[0037] The training data used in these operations includes a plurality of x-ray projection training images 120a' representing the interventional device 110 from a first perspective 130a. 1..n and for each x-ray projection training image 120a', a corresponding ground truth x-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n The training data includes an X-ray projection training image 120a'. 1..n may represent the same type of interventional device for which predictions are made during inference. 1..n may represent an interventional device in the region of interest of the same type in which the interventional device will be placed in the inference. The training data may represent the interventional device 110 during treatment on different subjects. For example, a plurality of X-ray projection training images 120a' 1..n may include images from tens, or hundreds, or thousands, or more, of medical procedures performed on different subjects. The subjects may have different ages, body mass indices, genders, etc. Thus, the training data may represent interventional devices in a variety of different shapes.

[0038] In the exemplary neural network 150 shown in FIG. 5, the neural network 150 is trained by inputting X-ray projection training images 120a′ 1..n The input of is shown on the left side of Figure 5. During training, the encoder 150 e is each input X-ray projection training image 120a' 1..n A transform T is then applied to the latent space representation to provide a transformed latent space representation TLR. The transformed latent space representation TLR is then passed to decoder 150. d is input to the decoder 150 d is a predicted X-ray projection image 120 representing the interventional device 110 from a second viewpoint 130b. P b 1..n The parameters of the neural network 150 are then used to generate the predicted x-ray projection image 120 P b 1..n and a corresponding ground truth X-ray projection training image 120b' representing the interventional device 110 from a second perspective 130b. 1..n The difference is calculated using a loss function, as shown on the right side of Figure 5. Loss functions such as the L1 norm, L2 norm, or negative log-likelihood may be used for this purpose. Also, a loss function may be used that causes the neural network 150 to learn a distribution from which the latent space representation is sampled to resemble a reference distribution. For example, the value of a loss function such as the Kullback-Leibler "KL" divergence may be used for this purpose, which uses a standard Gaussian distribution as the reference distribution.

[0039] If predictions are desired for multiple different viewpoints in the inference, the training operations described above are performed using training data including each change in viewpoint, i.e., a set of X-ray projection training images representing the interventional device 110 from a first viewpoint 130a and a second viewpoint 130b, and the corresponding transformation T. Thus, in this case, the transformation T corresponding to the change in viewpoint between the first viewpoint 130a and the second viewpoint 130b is also input into the neural network during training.

[0040] During training, in the training method described above, the X-ray projection training image 120a' 1..n into the neural network 150, and the predicted X-ray projection image 120 P b 1..n The operations of generating the loss function and adjusting the parameters of the neural network 150 are repeated until a stopping criterion is met. Training is terminated when the value of the loss function meets the stopping criterion. This indicates that the trained neural network 150 is able to predict, with an acceptable level of accuracy, an X-ray projection image representing the interventional device 110 from the second viewpoint 130b.

[0041] As described above, training a neural network 150 involves inputting a training data set into the neural network and iteratively adjusting the neural network's parameters until the trained neural network provides accurate outputs. Training is often performed using a dedicated neural processor, such as a graphics processing unit (GPU), neural processing unit (NPU), or tensor processing unit (TPU). Training often employs a centralized approach, in which cloud- or mainframe-based neural processors are used to train the neural network. Following its training with the training data set, the trained neural network can be deployed to a device for analyzing new input data during inference. Processing requirements during inference are significantly lower than those required during training, allowing the neural network to be deployed to a variety of systems, such as laptop computers, tablets, and mobile phones. Inference can be performed, for example, by a central processing unit (CPU), GPU, NPU, or TPU on a server or in the cloud.

[0042] Therefore, the process of training the neural network 150 described above involves training the encoder 150 e and its decoder 150 dThe training involves adjusting the parameters of the neural network. The parameters, more specifically the weights and biases, control the behavior of the activation function in the neural network. In supervised learning, the training process automatically adjusts the weights and biases so that when presented with input data, the neural network accurately provides the corresponding expected output data. To do this, a loss function or error value is calculated based on the difference between the predicted output data and the expected output data. The loss function value can be calculated using functions such as the negative log-likelihood loss, mean absolute error (or L1 norm), mean squared error, root mean squared error (or L2 norm), Huber loss, or (binary) cross-entropy loss. Other loss functions, such as the Kullback-Leibler divergence, are used to calculate the X-ray projection training images 120a'. 1..n It may additionally be used when training a variational autoencoder to ensure that the distribution of the latent space encoding generated from is similar to a standard Gaussian distribution with mean 0 and standard deviation 1. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function meets a stopping criterion. In some cases, training is terminated when the value of the loss function meets one or more of several criteria.

[0043] Various methods for solving loss minimization problems are known, including gradient descent and quasi-Newton algorithms. Various algorithms have been developed to implement these methods and their variations, including, but not limited to, stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg-Marquardt, momentum methods, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax "optimizers." These algorithms use chaining techniques to calculate the derivatives of the loss function with respect to the model parameters. This process is called backpropagation because the derivatives are calculated starting at the last layer, or output layer, and moving toward the first layer, or input layer. These derivatives inform the algorithm how the model parameters must be adjusted to minimize the error function. That is, adjustments to the model parameters are made starting at the output layer and working backward through the network until the input layer is reached. In the first training iteration, the initial weights and biases are often randomized. The neural network then predicts output data, which is also random. Backpropagation is then used to adjust the weights and biases. The training process is performed iteratively by adjusting the weights and biases at each iteration. Training is terminated when the error, or the difference between the predicted output data and the expected output data, is within an acceptable range for the training data or some validation data. The neural network can then be deployed, and the trained neural network will make predictions for new input data using the trained values ​​of its parameters. If the training process is successful, the trained neural network will accurately predict the expected output data from the new input data.

[0044] Returning to the method shown in FIG. 1, inference continues to operation S130, where, in response to the input in operation S120, predicted x-ray projection image data 120 is generated. P b is generated using a neural network.P b represents the interventional device 110 from a second viewpoint 130b of the projection X-ray imaging system 140 relative to the interventional device. Referring to the example neural network 150 shown in Figure 5, this operation is performed using a neural network 150 trained using the techniques described above.

[0045] Next, the predicted X-ray projection image data 120 P 4 illustrates a predicted X-ray projection image 120 representing the interventional device 110 from a second viewpoint 130b of a projection X-ray imaging system 140 relative to the interventional device, in accordance with some aspects of the present disclosure. P 4 is an example of predicted X-ray projection image data 120b representing the interventional device 110 from the second viewpoint 130b. Compared to the image representation of the corresponding X-ray projection image data 120a of FIG. 3 input to the neural network 150, the second viewpoint 130b is rotated 90 degrees relative to that of FIG. 3. The example shown in FIG. 4 represents a so-called lateral view of the brain. As can be seen, the predicted X-ray projection image data 120b representing the interventional device 110 from the second viewpoint 130b P b, i.e., by providing the image representation shown in Fig. 4, the user can better visualize the three-dimensional shape of the interventional device. Furthermore, because the predicted X-ray projection image data for the second viewpoint is generated without having to actually acquire X-ray projection image data from this viewpoint, the method facilitates the user's visualization of the three-dimensional shape of the interventional device without increasing the amount of X-ray radiation dose delivered to the patient.

[0046] FIG. 6 illustrates a diagram illustrating the generation of predicted x-ray projection image data 120 representing the interventional device 110 from a first perspective 130a using training data including x-ray projection training images 120a′ representing the interventional device 110 from a second perspective 130b and corresponding ground truth x-ray projection training images 120b′ representing the interventional device 110 from a second perspective 130b, in accordance with some aspects of the present disclosure. P6 is a second example of training a neural network 150 to generate a neural network b. The neural network 150 shown in FIG. 6 is a capsule network implementation of the neural network shown in FIG. 5 and can be used as an alternative to the neural network shown in FIG. 5. Capsule neural networks are further described in Hinton, G.E. et al., "Transforming Auto-Encoders," Artificial Neural Networks and Machine Learning - ICANN 2011, Lecture Notes in Computer Science, 6791: 44-51. In the neural network shown in FIG. 6, the segment encapsulated in the elliptical boundary is a capsule C that includes a neural network such as the neural network 150 shown in FIG. 1..j The left neural network encoder 150 of each capsule e Additionally, it outputs a confidence measure (denoted by p in FIG. 6) in the generated output, in the range [0, 1]. The decoder 150 of the right neural network of each capsule e produces an output. The reliability measure of each encoder in the capsule is multiplied with the output of the decoder for the capsule, and the capsule outputs are combined to provide a combined output for the capsule. The reliability measure p can also be output so that the user can modulate the reliability in the system accordingly. Training of the neural network is performed in the same manner as described above for the example shown in Figure 5, and all capsules C 1..jare trained simultaneously. Compared to the neural network shown in Figure 5, the use of a capsule implementation shown in Figure 6 has the advantage of obtaining consensus from the outputs of each of the capsules. Each capsule may focus on a different portion of the image (e.g., one capsule may focus only on the device tip, while another capsule may focus on a larger area at the distal end of the device). This is achieved by an additional set of weights used to weight the input to each of the capsules. The outputs from each of these capsules must agree to produce a combined output that produces a low error when compared to ground truth.

[0047] Variations on the above method are also contemplated, as described in the examples below.

[0048] In one example, the interventional device 110 is positioned within the vasculature, and the x-ray projection image data 120a also represents the vasculature. In this case, the x-ray projection training image 120a' 1..n , and if the ground truth X-ray projection training image 120b′ also represents the vasculature, the neural network 150 generates predicted X-ray projection image data 120b that also represents the vasculature. P b. The interventional device 110 can be placed in any part of the vasculature. For example, it can be placed in the brain or in the peripheral vasculature, such as in the arm or leg. The vasculature provides useful context for the shape of the interventional device, and the resulting predicted x-ray projection image data 120 P By providing a prediction of the vasculature in b, a user may gain an improved understanding of its shape. In this example, a GAN may be used as the neural network 150. The neural network in this example is trained in a similar manner as described above with reference to FIG. 5.

[0049] In another embodiment, the X-ray projection image data 120a, the predicted X-ray projection image data 120 P b, X-ray projection training image 120a' 1..n, and ground truth X-ray projection training image 120b' 1..n each includes a temporal sequence of images representing the interventional device 110. In this example, the neural network 150 uses the current x-ray image in the input x-ray projection image data 120a and one or more historical x-ray images in the input x-ray projection image data 120a to generate predicted x-ray projection image data 120b for the current x-ray image in the input x-ray projection image data 120a. P b. In this example, the encoder 150 shown in FIG. e and decoder 150 d may contain RNN or LSTM cells, as described above.

[0050] In another embodiment, predicted X-ray projection image data 120 representing the interventional device 110 from a second perspective 130b is P b is constrained using angiographic image data 170b representing the vasculature. In this example, the interventional device 110 is positioned within the vasculature, and the method described with reference to FIG. receiving angiographic image data 170b representing the vasculature; Predicted X-ray projection image data 120 P The received angiographic image data 170b is used to generate predicted x-ray projection image data 120b representing the interventional device 110 from a second perspective 130b, such that b represents the interventional device within the vasculature. P constraining b; Includes.

[0051] In this example, the angiographic image data may be provided in the form of preoperative computed tomography (CT) angiographic images representing the vasculature. The CT images may be acquired from a CT imaging system. Alternatively, the angiographic image data may be provided in the form of intraoperative 3D rotational angiographic (3DRA) images obtained using a projection X-ray imaging system. The 3DRA images may be acquired from a cone beam CT (CBCT) imaging system. Alternatively, the angiographic image data may be provided in the form of intraoperative projection images obtained from a biplane projection X-ray imaging system representing the vasculature from a second perspective 130b. Such data is often acquired before or during a medical investigation and includes the second perspective 130b of the vasculature. In some examples, the angiographic image data is acquired after injection of a contrast agent into the vasculature to enhance visibility of the vasculature and before navigation of a device within the vasculature. If the angiographic image data is generated by a biplane projection X-ray imaging system, the image data representing the vasculature from the second perspective 130b may be provided directly by one of the biplane images. Similarly, if the angiographic image data is provided by a 3D RA image, image data representing the vasculature from the second viewpoint 130b may be provided by selecting data from a viewpoint matching the second viewpoint 130b. If the angiographic image data is provided by a CT image, image data representing the vasculature from the second viewpoint 130b may be provided in the form of a digitally reconstructed radiograph "DRR" generated by positioning a virtual source and a virtual detector at the second viewpoint 130b relative to the CT image and projecting virtual X-rays emitted by the virtual X-ray source through the CT data onto the virtual X-ray detector. Thus, in this case, the angiographic image data includes a computed tomography CT image representing the vasculature, and the method comprises: projecting the angiography image data to provide a projection of a CT image representing the vasculature from a second viewpoint 130b; Including, Predicted X-ray projection image data 120 P Constraining b is performed using projections of the CT images.

[0052] In this example, the constraint operation is applied to the predicted x-ray projection image data 120 after being output by the neural network 150. P b. In other words, it can be described as a post-processing operation applied to the output of the neural network. The constraining operation constrains the predicted x-ray projection image data 120 to represent the interventional device within the vasculature. P This can be performed by deforming the interventional device within b. In other words, the shape of the interventional device is deformed to fit within the vasculature. Known deformation techniques, such as the use of displacement fields, can be used for this purpose. Displacement fields can be used to specify in which direction pixels in an image must move to generate a new image. Thus, the displacement fields can be used to define how pixels belonging to the interventional device should be moved to fit within the vasculature. The deformation can also be performed subject to the mechanical constraints of the interventional device. Therefore, constraints on factors such as the connectivity between sections of the interventional device and the amount of curvature of the interventional device can also be applied in this operation to ensure that the prediction of the interventional device shape is valid.

[0053] In another example, the predicted x-ray projection image data 120 after being output by the neural network 150 may be P Rather than constraining b, the neural network 150 uses the constrained predicted x-ray projection image data 120 P In this example, the neural network 150 is trained to generate constrained predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b. P b) is trained to generate predicted X-ray projection image data 120 P b is constrained by angiographic image data 170b, and the training data further includes angiographic images representing the vasculature.

[0054] This example shows the constrained predicted X-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b. P 7, which is an example of a neural network 150 trained to generate predicted X-ray projection image data 120. P b is constrained by angiographic image data 170b and I2 according to some aspects of the present disclosure. In this example, X-ray projection image data 120a is input to the neural network 150, as shown on the left side of FIG. 7. In the illustrated example, the interventional device is segmented from the X-ray projection image data 120a labeled "Device Navigation on AP," and only the X-ray projection image data 120a representing the interventional device is input to the neural network. The X-ray projection image data 120a may be provided by a projection X-ray imaging system. For example, it may be provided by one of the planes of a biplane projection X-ray imaging system. In the illustrated embodiment, biplane angiographic images I1 and I2 are acquired prior to inference. The biplane angiographic images represent the vasculature in which the interventional device will be positioned in the inference. The biplane angiographic image I1 is acquired from a first viewpoint 130a, i.e., a viewpoint from which projection is made using the X-ray projection image data 120a. The biplane angiographic image I2 is acquired from a second viewpoint 130b, and therefore the predicted X-ray projection image 120 P b represents the shape of the vasculature from the viewpoint generated by the neural network 150. In this example, angiographic image data 170b in the form of biplane angiographic image I2 is input to the neural network 150, which generates constrained predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b. P b. Constrained predicted X-ray projection image data 120 P b may be overlaid on the biplane angiography image I2 to provide anatomical context for the shape of the interventional device as seen from the second viewpoint 130b. Such an overlaid image is shown in the image labeled "Device Navigation on Virtual Biplane."

[0055] The training data used to train the neural network 150 in this example is the x-ray projection training images 120a' 1..n , and the corresponding ground truth X-ray projection training image 120b'. 1..n , and angiographic image data 170b. The neural network implicitly learns to associate the curvature of the vasculature from the input angiographic image data 170b with the curvature of the interventional device. An additional loss function may be used to penalize predictions of the interventional device 110 that do not overlap with the vasculature in the angiographic image data 170b, in order to train the neural network to associate the curvature of the vasculature from the input angiographic image data 170b with the curvature of the interventional device.

[0056] In a related example, the angiography image data 170b in the previous example includes a CT image representing the vasculature, which also represents a portion of the interventional device 110. In this example, the angiography image data 170b is projected from a second viewpoint 130b to provide a projection of both the vasculature and the interventional device. The neural network prediction also includes the predicted x-ray projection image data 120. P b is constrained to match the shape of the interventional device in the projection of the angiographic image data 170b. Thus, in this example, the shape of the interventional device from the second view, as determined from the angiographic image data 170b, is used by the neural network as a constraint in its prediction.

[0057] In this example, the angiographic image data includes computed tomography CT images representing the vasculature and at least a portion of the interventional device 110, and the method described with reference to FIG. 7 includes: projecting the angiography image data to provide a projection of a CT image representing the vasculature from a second viewpoint 130b; Including, Predicted X-ray projection image data 120 P Constraining b is performed using projections of the CT images.

[0058] The method also includes: Segmenting a portion of the interventional device 110 in the angiographic image data; projecting the segmented portion of the interventional device to provide a projection of the segmented portion of the interventional device from a second perspective 130b; Predicted X-ray projection image data 120 P deriving predicted X-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b using the projected segmented portion of the interventional device, such that b corresponds to the projected segmented portion of the interventional device; P constraining b; Includes.

[0059] An example of this is shown in FIG. 8, which illustrates predicted X-ray projection image data 120 representing the interventional device 110 from a second perspective 130b and a third perspective 130c, respectively, of a projection X-ray imaging system 140, in accordance with some aspects of the present disclosure. P b, 120 P8 is a schematic diagram including a CT image (above right) and a CT image (b). In the illustrated example, the upper left portion of FIG. 8 shows a CT image representing the vasculature and a portion of the interventional device 110. The CT image is a so-called pre-CT image generated before the live X-ray projection image 120a shown in the upper right portion of FIG. 8. The central arrow in the CT image indicates a first viewpoint 130a of the projection X-ray imaging system, from which the current live X-ray projection image 120a shown in the upper right side of FIG. 8 is generated. Now, referring to the image in the upper right side of FIG. 8, the live X-ray projection image 120a is a current image generated by the projection X-ray imaging system from the direction 130a shown in the CT image. A path ABCD indicates the extent of the interventional device in the current projection image 120a. A first portion AB of the path ABCD also corresponds to the projection shape of the interventional device obtained by projecting the CT image from the direction 130a. A second dashed portion of the path ABCD, i.e., BCD, indicates a further extent of the interventional device in the current projection image 120a. The neural network predictions are shown in the lower right and lower left portions of FIG. 8 for the second viewpoint 130b and the third viewpoint 130c, respectively. In the example shown in FIG. 8, predictions are made for both the second viewpoint 130b and the third viewpoint 130c. However, it should be noted that predictions may alternatively be made for only one viewpoint, i.e., the second viewpoint 130b. The image in the lower right portion of FIG. 8 shows the predicted shape of the interventional device from the second viewpoint 130b along a path A-D'. This path includes a portion AB corresponding to the projected shape of the interventional device obtained by projecting the CT image from viewpoint 130b, and a portion B-D' showing the predicted shape of the interventional device predicted by the neural network. The lower left portion of FIG. 8 similarly shows the projected and predicted shapes of the interventional device. In this image, portions AB and B-D' represent the projected and predicted shapes of the interventional device for the third viewpoint 130c, respectively.

[0060] In another example, device type data 180 indicating the type of interventional device 110 represented in the x-ray projection image data 120a is input to the neural network during training and also during inference. In this example, the method described with reference to FIG. receiving device type data 180 indicative of the type of interventional device 110 represented in the x-ray projection image data 120a; inputting the received device type data into a neural network 150; Including, The neural network 150 generates predicted x-ray projection image data 120 representing the interventional device 110 from the second perspective 130b based on the device type data 180. P It is trained to generate b.

[0061] Training the neural network using device type data allows the neural network to make predictions, for example, across different types of interventional devices. In this example, the device type data may refer to a category of interventional device, such as a "guidewire" or an "IVUS imaging device," or may refer to a more specific identifier, such as the supplier or model number of the interventional device. Examples of device type 180 being input to the neural network are shown in FIGS. 5 and 7. The device type data may be manually selected by a user, as shown in these examples, or the device type may be user-selectable from a menu of options. Alternatively, the device type data may be automatically detected. In the latter case, the device type may be detected by performing an object detection operation on the x-ray projection image data 120a using a feature detector. Alternatively, the device type may be automatically detected before the device is inserted into a patient from video data generated during a medical procedure using object detection techniques.

[0062] In another example, device constraint data is used to constrain the predictions of the neural network. In this example, the method described with reference to FIG. receiving device constraint data defining mechanical and / or dimensional constraints of the interventional device 110 represented in the x-ray projection image data 120a; Predicted X-ray projection image data 120 P Using the received device constraint data, predictive x-ray projection image data 120 is generated such that b represents the interventional device 110 within the mechanical and / or dimensional constraints defined by the device constraint data. P constraining b; Includes.

[0063] An example of a mechanical constraint that may be applied in this example is a limit on the shape of the interventional device, such as the amount of curvature. For example, it may be specified that the curvature of the interventional device be below a certain limit. Another example of a mechanical constraint is the connectivity of the interventional device. For example, it may be specified that the interventional device be fully connected along its length. An example of a dimensional constraint is the length of the interventional device. For example, it may be specified that the length of the interventional device be within a specified range. Predicted X-ray projection image data 120 P The constraints on b may be implemented using various techniques. For example, the constraints may be implemented within the neural network by penalizing predictions that deviate from the constraints. Alternatively, post-processing operations may be performed on the predicted x-ray projection image data 120 generated by the neural network 150. P In this approach, the predicted X-ray projection image data 120 generated by the neural network 150 may be P b may be deformed using the displacement field, as described above, and any violation of the constraints in the neural network's predictions identifies the locations where the constraints are violated and predicts deformations that ensure the constraints are satisfied. Pb. In another post-processing approach, a projection of a 3D computer-aided design (CAD) model representing the interventional device from a first perspective 130a may be deformed so that a prediction of the shape of the interventional device from a second perspective 130b satisfies mechanical and / or dimensional constraints. Applying such constraints may ensure that the prediction of the interventional device shape is valid. Such constraints may be applied in combination with the constraints described above, in which the interventional device is deformed to fit within the vasculature.

[0064] In another example, the confidence value is calculated based on the predicted x-ray projection image data 120 P In this example, the method described with reference to FIG. Predicted X-ray projection image data 120 P calculating a confidence value p of b; outputting an indication of the confidence value p; Includes.

[0065] The confidence value p may be calculated using various techniques. For example, the neural network 150 may calculate the confidence value using a dropout technique. This involves repeatedly inputting the same data into the neural network 150 and randomly removing a percentage of neurons from the neural network at each iteration to determine the neural network's output. The neural network's output is then analyzed to provide a mean and variance. The mean represents the final output, and the magnitude of the variance indicates whether the neural network is consistent in its predictions, in which case the variance is small and the confidence value is relatively high, or whether the neural network is inconsistent in its predictions, in which case the variance is larger and the confidence value is relatively low. Alternatively, the confidence may be calculated using, for example, Kullback-Leibler "KL" divergence between the distribution from which the latent space representation LR is sampled and the distribution across the currently trained encoding. This divergence indicates how well the input sequence is represented by the learned encoding. Alternatively, the confidence value may be calculated based on the predicted X-ray projection images 120. P The confidence may be calculated based on the clarity of the image representing b. For example, a blurred image indicates a low confidence, while a clearer image indicates a higher confidence. A low confidence value may indicate that the trained neural network 150 is not suited to process the input X-ray projection image data 120a. For example, the input data may be out of distribution compared to the data used to train the neural network.

[0066] The indication of the confidence value p may be output in various ways. For example, the confidence value p may be output as a numerical value. Alternatively, a graphical representation of the confidence value p may be displayed on the predicted X-ray projection image data 120. PThe confidence value p may be generated by multiplying the image representation of b by a confidence value p, which results in a relatively high contrast image when the confidence is relatively high and a relatively low contrast image when the confidence is relatively low. The graphical representation of the confidence value p may alternatively be provided as a heatmap image. The output of the confidence value p in this example allows the user to modulate the confidence in the neural network's prediction accordingly.

[0067] As described above, the source of training data for training the neural network 150 is projection images from historical procedures performed using a biplane projection X-ray imaging system. Such procedures typically result in a single pair of views of the interventional device per procedure. This limits the amount of training data available for training the neural network. In one example, the neural network is trained using synthetic X-ray projection training images. In this example, the training data used to train the neural network 150 includes a plurality of synthetic X-ray projection training images representing the interventional device 110 from a first perspective 130a and, for each synthetic X-ray projection training image, a corresponding ground truth synthetic X-ray projection training image representing the interventional device from a second perspective 130b. The synthetic X-ray projection training images are generated by projecting 3D tracking data representing the shape of the interventional device from each of the first perspective 130a and the second perspective 130b.

[0068] In this example, the operation of projecting 3D tracking data to provide X-ray projection training images representing the interventional device from a first viewpoint 130a and a second viewpoint 130b can be performed by positioning a virtual source and a virtual detector at the desired viewpoints 130a, 130b relative to the 3D tracking data and projecting virtual X-rays emitted by the virtual X-ray source through the 3D tracking data onto the virtual X-ray detector.

[0069] In this example, synthetic x-ray projection training images are generated by projecting 3D tracking data representing the shape of the interventional device, so a data set from one historical procedure can be used to generate multiple pairs of viewpoints of training data. This increases the amount of training data provided per procedure and also facilitates the generation of training data with any desired pairs of viewpoints 130a, 130b. In this example, the tracking data can be provided by a tracking system such as an electromagnetic tracking system, a fiber optic-based tracking system, or a dielectric mapping system. An example of an electromagnetic tracking system is disclosed in U.S. Patent Application Publication No. 2020 / 397510. An example of an fiber optic-based tracking system using strain sensors to determine the position of the interventional device is disclosed in U.S. Patent Application Publication No. 2012 / 323115.

[0070] In another example, a computer program is provided having instructions that, when executed by one or more processors, cause the one or more processors to perform a method for providing projection images representative of an interventional device 110, the method comprising: receiving S110 X-ray projection image data 120a representing the interventional device 110 from a first viewpoint 130a of a projection X-ray imaging system 140 relative to the interventional device; Step S120 of inputting the X-ray projection image data 120a into the neural network 150; In response to the input, predicted x-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b of the projection x-ray imaging system 140 relative to the interventional device is generated. P a step S130 of generating a second viewpoint 130a, where the second viewpoint 130a is different from the first viewpoint 130a; and The neural network 150 generates a plurality of X-ray projection training images 120a' representing the interventional device 110 from a first perspective 130a. 1..n and each X-ray projection training image 120a' 1..n, and a corresponding ground truth X-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n and generating predicted X-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b using training data including P It is trained to generate b.

[0071] In another example, a system 200 for providing a projection image representative of an interventional device 110 is provided. The system includes: receiving x-ray projection image data 120a representing the interventional device 110 from a first viewpoint 130a of a projection x-ray imaging system 140 relative to the interventional device; The X-ray projection image data 120a is input to the neural network 150; In response to the input, the neural network 150 is used to generate predicted x-ray projection image data 120 representing the interventional device 110 from a second perspective 130b of the projection x-ray imaging system 140 relative to the interventional device. P b, where the second viewpoint 130b is different from the first viewpoint 130a. one or more processors 210 configured to: The neural network 150 generates a plurality of X-ray projection training images 120a' representing the interventional device 110 from a first perspective 130a. 1..n and for each x-ray projection training image 120a', a corresponding ground truth x-ray projection training image 120b' representing the interventional device 110 from a second viewpoint 130b. 1..n and generating predicted X-ray projection image data 120 representing the interventional device 110 from a second viewpoint 130b using training data including P It is trained to generate b.

[0072] An example of a system 200 is shown in Figure 2. The system 200 also includes an interventional device 110, a medical imaging system for generating X-ray projection image data 120a, such as the projection X-ray imaging system 140 shown in Figure 2, a patient couch 220, and predicted X-ray projection image data 120a. P Note that the display device may include a monitor 230 for displaying b, and one or more of a user interface device (not shown in FIG. 2) configured to receive user input in the manner described above, such as a mouse, touch screen, keyboard, joystick, etc.

[0073] The above examples should be understood to be illustrative of the present disclosure, not limiting. Further examples are contemplated. For example, examples described in connection with a computer-implemented method may also be provided correspondingly by a computer program, a computer-readable storage medium, or the system 200. It should be understood that features described with respect to any one example may be used alone or in combination with other described features, or with one or more features of another of the examples, or with combinations of other examples. Furthermore, equivalents and modifications not described above may be used without departing from the scope of the invention, as defined in the appended claims. In the claims, the word "comprises" does not exclude other elements or operations, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Any reference signs in the claims should not be construed as limiting their scope.

Claims

1. 1. A computer-implemented method for providing a projection image representative of an interventional device, comprising: receiving x-ray projection image data representing the interventional device from a first perspective of a projection x-ray imaging system relative to the interventional device; inputting the X-ray projection image data into a neural network; using the neural network to generate predicted x-ray projection image data representing the interventional device from a second viewpoint of the projection x-ray imaging system relative to the interventional device, the second viewpoint being different from the first viewpoint; and the neural network is trained to generate the predicted x-ray projection image data representing the interventional device from the second perspective using training data having a plurality of x-ray projection training images representing the interventional device from the first perspective and, for each x-ray projection training image, a corresponding ground truth x-ray projection training image representing the interventional device from the second perspective. Computer-implemented methods.

2. The method further comprises: receiving orientation data defining the second viewpoint of the projection X-ray imaging system; inputting the received orientation data into the neural network; and the neural network is trained to generate the predicted x-ray projection image data representing the interventional device from the second perspective based on the orientation data. The computer-implemented method of claim 1 .

3. 3. The computer-implemented method of claim 2, further comprising a transform autoencoder having the neural network, an encoder, and a decoder, wherein the encoder is trained to learn a latent space representation of the input X-ray projection image data, and when the latent space representation of the input X-ray projection image data is transformed by the orientation data and decoded by the decoder, the decoded latent space representation represents the interventional device from the second viewpoint.

4. 4. The computer-implemented method of claim 1, wherein the interventional device is positioned within a vasculature, and the received X-ray projection image data, the predicted X-ray projection image data, the X-ray projection training images, and the ground truth X-ray projection training images further represent the vasculature.

5. 5. The computer-implemented method of claim 1, wherein the received X-ray projection image data, the predicted X-ray projection image data, the X-ray projection training images, and the ground truth X-ray projection training images each comprise a temporal sequence of images representing the interventional device, and the neural network is trained to generate the predicted X-ray projection image data for the current X-ray image in the input X-ray projection image data using a current X-ray image in the input X-ray projection image data and one or more historical X-ray images in the input X-ray projection image data.

6. The interventional device is positioned within the vasculature, and the method further comprises: receiving angiographic image data representative of the vasculature; using the received angiographic image data to constrain the predicted x-ray projection image data representing the interventional device from the second perspective such that the predicted x-ray projection image data represents the interventional device within the vasculature; The computer-implemented method of claim 1 , comprising:

7. 7. The computer-implemented method of claim 6, wherein the neural network is trained to generate constrained predicted X-ray projection image data representing the interventional device from the second viewpoint, the predicted X-ray projection image data being constrained by the angiographic image data, and the training data further includes angiographic images representing the vasculature.

8. The received angiographic image data includes a computed tomography CT image representative of the vasculature, the method further comprising: projecting the angiographic image data to provide a projection of a previous day's CT image representing the previous day's vasculature from the second perspective; and constraining the predicted X-ray projection image data is performed using the projections of the CT images.

8. A computer-implemented method according to claim 6 or claim 7.

9. The received angiographic image data further represents at least a portion of the interventional device, and the method further comprises: segmenting the portion of the interventional device in the angiographic image data; projecting the segmented portion of the interventional device to provide a projection of the segmented portion of the interventional device from the second perspective; using the projected segmented portion of the interventional device to constrain predicted x-ray projection image data representing the interventional device from the second perspective such that the predicted x-ray projection image data corresponds to the projected segmented portion of the interventional device; The computer-implemented method of claim 8 , comprising:

10. The method further comprises: receiving device type data indicative of the type of the interventional device represented in the x-ray projection image data; inputting the received device type data into the neural network; Including, the neural network is trained to generate the predicted x-ray projection image data representing the interventional device from the second perspective based on the device type data. A computer-implemented method according to any one of claims 1 to 9.

11. The method further comprises: receiving device constraint data defining mechanical and / or dimensional constraints of the interventional device represented in the x-ray projection image data; constraining the predicted x-ray projection image data using the received device constraint data such that the predicted x-ray projection image data represents the interventional device within the mechanical and / or dimensional constraints defined by the device constraint data; 11. The computer-implemented method of claim 1, comprising:

12. The method further comprises: calculating a confidence value for the predicted X-ray projection image data; outputting an indication of said confidence value; 12. The computer-implemented method of claim 1, comprising:

13. 13. The computer-implemented method of claim 1, wherein the training data includes a plurality of synthetic X-ray projection training images representing the interventional device from the first viewpoint and, for each synthetic X-ray projection training image, a corresponding ground truth synthetic X-ray projection training image representing the interventional device from the second viewpoint, the synthetic X-ray projection training images being generated by projecting 3D tracking data representing a shape of the interventional device from each of the first viewpoint and the second viewpoint.

14. The computer-implemented method of claim 1 , wherein the first viewpoint and the second viewpoint are orthogonal to each other.

15. The neural network receiving the training data; For each of a plurality of the x-ray projection training images in the training data representing the interventional device from the first perspective, inputting the X-ray projection training images into the neural network; generating a predicted x-ray projection image representing the interventional device from the second perspective using the neural network; adjusting parameters of the neural network based on differences between the predicted x-ray projection images and the corresponding ground truth x-ray projection training images representing the interventional device from the second perspective; repeating said inputting, said generating, and said adjusting until a stopping criterion is met; 15. The computer-implemented method of claim 1, wherein the method is trained to generate the predicted X-ray projection image data representing the interventional device from the second view by:

Images