Determining an interventional device shape

Abstract

A computer-implemented method of providing a neural network for predicting a three-dimensional shape of an interventional device arranged within a vascular region, comprising training (S140) a neural network (140) to predict a three-dimensional shape of the interventional device constrained by the vascular region (150) from received X-ray image data (120) and received volumetric image data (110). The training comprises constraining an adjustment of parameters of the neural network such that the three-dimensional shape of the interventional device predicted by the neural network (150) fits within a three-dimensional shape of the vascular region represented by the received volumetric image data (110).

Description

Technical Field

[0001] This disclosure relates to determining the three-dimensional shape of an interventional device positioned within a vascular region. A computer-implemented method, processing apparatus, system, and computer program product are also disclosed. Background Technology

[0002] Many interventional medical procedures are performed under X-ray imaging. The two-dimensional images provided by X-ray imaging assist physicians in navigating interventional devices, such as guidewires and catheters, within anatomical structures. Dense areas of anatomical structures, such as bones and interventional devices, are highly visible under X-ray imaging. However, soft tissue anatomical areas, such as the vascular system, are often less visible under X-ray imaging, which limits navigation. Mentally mapping the shape of the interventional device seen in a two-dimensional X-ray image to a three-dimensional anatomical structure can also be challenging.

[0003] By way of example, in prostate artery embolization (PAE), interventional devices such as guidewires and catheters are guided through the vascular system to the treatment site of injected microparticles under X-ray imaging in order to block small blood vessels and thus impede blood supply to the prostate. The vascular system is often not very visible in X-ray images, and psychologically mapping the shape of the interventional device seen in the X-ray image to the three-dimensional anatomy can be challenging.

[0004] To improve visibility of the vascular system, contrast agents are often used to capture two-dimensional intraoperative digital subtraction angiography (DSA) images. However, some patients may have adverse reactions to contrast agents, limiting their use. To address the challenge of psychologically mapping the shape of the interventional device in X-ray images to three-dimensional anatomy, preoperative three-dimensional images of the vascular system, such as computed tomography angiography (CTA) images, can be obtained prior to such interventions. CTA images serve as a roadmap to guide the interventional procedure. However, despite these measures, physicians often need to take multiple two-dimensional X-ray images of the vascular system from different projection angles to confirm the location of the interventional device.

[0005] Generating multiple two-dimensional X-ray images of the vascular system from different projection angles also has its drawbacks. In addition to the increased radiation dose, the desired projection angle may be unavailable because certain configurations of the C-arm supporting the X-ray source and detector are obstructed by the position of the patient or patient table.

[0006] Therefore, there is still room for improvement in the way that the shape of the interventional device placed in the vascular area is determined based on X-ray images. Summary of the Invention

[0007] According to a first aspect of this disclosure, a computer-implemented method is provided for providing a neural network for predicting the three-dimensional shape of an interventional device positioned within a vascular region. The method includes:

[0008] Receive volumetric image data representing the three-dimensional shape of the vascular region;

[0009] Receive X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region;

[0010] Receive ground truth shape data of the interventional device, representing the three-dimensional shape of the interventional device within the vascular region corresponding to one or more two-dimensional projections of the interventional device; and

[0011] The neural network is trained to predict the three-dimensional shape of an interventional device constrained by the vascular region based on received X-ray image data and received volumetric image data by: inputting the received X-ray image data and received volumetric image data into the neural network; adjusting the parameters of the neural network based on a first loss function, the first loss function representing the difference between the three-dimensional shape of the interventional device predicted by the neural network and the received actual interventional device shape data; and constraining the adjustment such that the three-dimensional shape of the interventional device predicted by the neural network fits within the three-dimensional shape of the vascular region represented by the received volumetric image data.

[0012] According to a second aspect of this disclosure, a computer-implemented method for predicting the three-dimensional shape of an interventional device positioned within a vascular region is disclosed. The method includes:

[0013] Receive volumetric image data representing the three-dimensional shape of the vascular region;

[0014] Receive X-ray image data representing a two-dimensional projection of the interventional device within the vascular region; and

[0015] The received X-ray image data and the received volumetric image data are input into a neural network, which is trained to predict the three-dimensional shape of the interventional device constrained by the vascular region based on the received X-ray image data and the received volumetric image data; and in response to the input, the neural network is used to predict the three-dimensional shape of the interventional device constrained by the vascular region based on the received X-ray image data and the received volumetric image data.

[0016] Further aspects, features, and advantages of this disclosure will become apparent from the following description, made with reference to the accompanying drawings. Attached Figure Description

[0017] Figure 1 This is a schematic diagram of an example neural network.

[0018] Figure 2 This is a schematic diagram of an example neuron.

[0019] Figure 3 This is a schematic diagram illustrating an example where a desired X-ray imaging view is obstructed during an interventional procedure.

[0020] Figure 4 This is a flowchart illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device, according to some aspects of this disclosure.

[0021] Figure 5 This is a schematic diagram illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure.

[0022] Figure 6 This is a flowchart illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device, and further includes optional operations S170 and S180.

[0023] Figure 7 This is a schematic diagram illustrating a second example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure.

[0024] Figure 8 This is a flowchart illustrating an example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure.

[0025] Figure 9 This is a schematic diagram illustrating an example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure.

[0026] Figure 10 This is a schematic diagram of a system 300 for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. Detailed Implementation

[0027] Examples of this disclosure are provided with reference to the following description and accompanying drawings. In this specification, for purposes of explanation, numerous specific details of certain examples are set forth. References to “example,” “implementation,” or similar language in the specification mean that a feature, structure, or characteristic described in connection with the example is included in at least one example. It should also be appreciated that a feature described with respect to one example may also be used in another example, and for the sake of brevity, not all features need to be repeated in each example. For example, features described with respect to a computer-implemented method may be implemented in a corresponding manner in a processing apparatus, in a system, and in a computer program product.

[0028] In the following description, reference is made to a computer-implemented method involving predicting the three-dimensional shape of an interventional device positioned within a vascular region. Reference is made to an X-ray imaging procedure in which an interventional device in the form of a guidewire is positioned within a vascular system. However, it should be appreciated that the examples of the computer-implemented methods disclosed herein can be used with other types of interventional devices that can be used within a vascular system besides guidewires, such as, but not limited to: catheters, intravascular ultrasound imaging devices, optical coherence tomography devices, catheters with sensing capabilities such as spectral sensing, guide sheaths, laser plaque resection devices, mechanical plaque resection devices, blood pressure devices and / or flow sensor devices, shape sensing devices, TEE probes, needles, biopsy needles, ablation devices, balloons, stents, or implants, etc.

[0029] Note that the computer-implemented methods disclosed herein can be provided as a non-transient computer-readable storage medium, comprising computer-readable instructions stored thereon that, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods can be implemented in a computer program product. The computer program product can be provided by dedicated hardware or hardware capable of running software associated with suitable software. When provided by a processor, the functionality of the method features can be provided by a single dedicated processor, a single shared processor, or multiple individual processors, some of which may be shared. The explicit use of the terms “processor” or “controller” should not be construed as specifically referring to hardware capable of running software, and may implicitly include, but is not limited to, digital signal processor “DSP” hardware, read-only memory “ROM” for storing software, random access memory “RAM”, non-volatile storage devices, etc. Furthermore, examples of this disclosure may take the form of a computer program product accessible from a computer-usable storage medium or a computer-readable storage medium, the computer program product providing program code for use by or in conjunction with a computer or any instruction-running system. For the purposes of this description, a computer-usable storage medium or computer-readable storage medium can be any means capable of including, storing, communicating, propagating, or transmitting programs for use by or in connection with an instruction-running system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system or device or a propagation medium. Examples of computer-readable media include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), rigid disk, and optical disc. Current examples of optical discs include optical disc read-only memory (CD-ROM), optical disc read / write (CD-R / W), and Blu-ray. TM and DVD.

[0030] This article also references various neural networks and how to train them by automatically adjusting their parameters. Figure 1 This is a schematic diagram of an example neural network. Figure 1 Example neural network 1 is a deep feedforward neural network, which includes neurons arranged in an input layer, three hidden layers h1–h3, and an output layer. Figure 1 Example neural network 1 in the example has the following inputs to neurons in its input layer: Input 1 - Input k The input data is received in the form of digital or binary input values, processed by neurons in its hidden layers h1–h3, and output by neurons in its output layer. 1..n The input data can represent, for example, image data or location data. Each neuron in the input layer represents a portion of the input data, such as, for example, pixels of an image.

[0031] Figure 2 This is a schematic diagram of an example neuron. Figure 2 The example neuron illustrated can be used to provide Figure 1 The neurons in the hidden layers h1–h3, and Figure 1 Neurons in the output layer. Figure 2 The example neuron in the example includes a summation part labeled with sigma symbols, and an activation function labeled F(S), which represents a sigma-shaped activation function. In the operation, the data input I0–I j-1 Multiply by the corresponding weight w0–w j-1 This sum is added together with the bias term B. The result is fed into the activation function to determine the neuron's output. The bias term shifts the activation function left or right and implicitly sets a threshold for neuron activation, thus controlling the neuron's output. Weights determine the strength of the connections between neurons in the network. Weights, biases, and neuron connections are examples of "trainable parameters" of a neural network, which are "learned" during the neural network's "training" process, or in other words, can be trained. Various activation functions can be used, such as the sigmoid function, the Tanh function, the step function, the rectified linear unit "ReLU", leaky ReLU, softmax, and the Swish function.

[0032] As mentioned above, training a neural network involves automatically adjusting the weights and biases described above. Supervised learning involves providing the neural network with a training dataset that includes input data and corresponding expected output data. The training dataset represents the input data that the neural network will likely be used to analyze after training. During supervised learning, the weights and biases are automatically adjusted so that, when presented with the input data, the neural network accurately provides the corresponding expected output data.

[0033] Training a neural network typically involves feeding a large training dataset into the network and iteratively tuning its parameters until the trained network provides accurate output. Training is usually performed using a graphics processing unit (GPU) or a dedicated neural processor, such as a neural processing unit (NPU) or a tensor processing unit (TPU). Therefore, training often employs a centralized approach, where cloud-based or host-based neural processors are used to train the neural network. After training with the training dataset, the trained neural network can be deployed to devices for analyzing new input data; this process is known as "inference." The processing requirements during inference are significantly lower than those required during training, allowing neural networks to be deployed on a wide variety of systems, such as laptops, tablets, mobile phones, and so on. For example, inference can be performed by a central processing unit (CPU), GPU, NPU, TPU, on a server, or in the cloud.

[0034] As mentioned above, in supervised learning, weights and biases are automatically adjusted so that the neural network accurately provides the expected output data when presented with the input training data. The value of the loss function, or error, is calculated based on the difference between the predicted and expected output data. The loss function can be calculated using functions such as negative log-likelihood loss, mean squared error, mean absolute error, Huber loss, Dice coefficient loss, or cross-entropy loss. During training, the value of the loss function is typically minimized, and training terminates when the value of the loss function meets a stopping criterion. Sometimes, training terminates when the value of the loss function meets one or more of several criteria. Various methods are known for solving this minimization problem, such as gradient descent, quasi-Newton methods, etc. Various algorithms have been developed to implement these methods and their variations, including but not limited to stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg-Marquardt, Momentum, Adam, Nadam, Adagrad, Adadelta, RMSProp, and the Adammax "optimizer".

[0035] These algorithms use chain rules to calculate the derivative of the loss function with respect to the model parameters. This process is called backpropagation because the derivative is calculated by moving from the last layer or output layer towards the first layer or input layer. These derivatives tell the algorithm how to tune the model parameters to minimize the error function. That is, adjustments to the model parameters are made starting from the output layer and working backward through the network until the input layer is reached. In the first training iteration, the initial weights and biases are often randomized. The neural network then predicts the output data, which is also randomized. Backpropagation is then used to adjust the weights and biases. The training process is performed iteratively by adjusting the weights and biases in each iteration. Training terminates when the error or difference between the predicted output data and the expected output data is within an acceptable range for the training data or for some validation data. Subsequently, the neural network can be deployed, and the trained neural network uses the trained values ​​of its parameters to predict new input data. If the training process is successful, the trained neural network accurately predicts the expected output data based on the new input data.

[0036] It should be recognized that although the training process described above involves a deep feedforward neural network, this is only used as an example, and other neural networks can be trained in a similar manner.

[0037] As mentioned above, the combination of the low visibility of soft tissue under X-ray imaging and the two-dimensional nature of X-ray images confuses the psychological task of mapping X-ray images of interventional devices onto the three-dimensional vascular system and thus navigating the interventional device within the body. Radiologists can therefore take multiple X-ray images of the vascular system from different projection angles. However, obtaining an image from the desired projection angle is sometimes confused by the patient's position. By way of example, Figure 3 This is a schematic diagram illustrating an example where a desired X-ray imaging view is obstructed during an interventional procedure. Figure 3 In some cases, the patient's legs obstruct the C-arm supporting the X-ray imaging source and detector from moving to the desired position and thus obtaining the desired view. In such situations, the radiologist may subsequently attempt to image the patient with multiple suboptimal views, thereby increasing the X-ray dose to the patient. Many similar situations exist where the patient or patient table obstructs obtaining certain desired X-ray imaging views.

[0038] To address one or more of the aforementioned drawbacks, a method is provided for determining the three-dimensional shape of an interventional device positioned within a vascular region. (Reference) Figures 4-6 The method is described. Figure 4 This is a flowchart illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device, according to some aspects of this disclosure. (Reference) Figure 4The method includes providing a neural network for predicting the three-dimensional shape of an interventional device positioned within a vascular region, and includes:

[0039] Receive volumetric image data 110 representing the three-dimensional shape of the vascular region;

[0040] S120 receives X-ray image data 120 representing one or more two-dimensional projections of an interventional device within a vascular region;

[0041] S130 receives actual three-dimensional shape data 130 of the interventional device within a vascular region corresponding to one or more two-dimensional projections of the interventional device; and

[0042] The S140 neural network 140 is trained by the following operations to predict the three-dimensional shape of an interventional device constrained by a vascular region 150 based on received X-ray image data 120 and received volumetric image data 110: the received X-ray image data 120 and received volumetric image data 110 are input into the S150 neural network 140, and the parameters of the S160 neural network 140 are adjusted based on a first loss function 160, which represents the difference between the three-dimensional shape of the interventional device predicted by the neural network 150 and the received actual interventional device shape data 130, and the adjustment is constrained such that the three-dimensional shape of the interventional device predicted by the neural network 150 fits within the three-dimensional shape of the vascular region represented by the received volumetric image data 110.

[0043] Figure 5 This is a schematic diagram illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. Various neural networks can be used as neural network 140, including convolutional neural networks, encoder-decoder networks, generative adversarial networks, capsule networks, regressive networks, reinforcement learning agents, Bayesian networks, and recurrent networks. During training operation S140, neural network 140 receives volumetric image data 110 and X-ray image data 120 representing the three-dimensional shape of a vascular region, and is trained to predict the three-dimensional shape of an interventional device constrained by a vascular region 150. Training can be performed in accordance with the above reference. Figure 1 and Figure 2The described method is performed in a consistent manner. Training data is provided in the form of ground-state interventional device shape data 130 representing the three-dimensional shape of the interventional device within the vascular region. The value of the first loss function 160 is calculated based on the difference between the three-dimensional shape of the interventional device predicted by the neural network 150 and the received ground-state interventional device shape data 130. The value of the first loss function 160 is used to adjust the parameters of the neural network 140. The adjustment is also constrained such that the three-dimensional shape of the interventional device predicted by the neural network 150 fits within the three-dimensional shape of the vascular region represented by the received volumetric image data 110. This constraint can be implemented by adjusting the parameters of the neural network 140 using constraint-based learning techniques, such as Ren, H. et al., 2018, entitled “Adversarial constraint learning for structured prediction” (Proc. 27). th The technique disclosed in the Joint Conference on Artificial Intelligence, 2637-2643, arXiv:1805.10561v2. Therefore, the constraint ensures that the three-dimensional shape of the interventional device predicted by the neural network 150 is an effective shape.

[0044] Volumetric image data 110, X-ray image data 120, and real-world interventional device shape data 130 can be received from various sources, including databases, imaging systems, computer-readable storage media, the cloud, and so on. Data can be received using any form of data communication (such as wired or wireless data communication), and can be received via the Internet, Ethernet, or by means of portable computer-readable storage media (such as USB storage devices, optical discs, or hard disks, etc.).

[0045] Volumetric image data 110 may be provided, for example, by one or more of the following:

[0046] Computed tomography image data;

[0047] Contrast-enhanced computed tomography image data;

[0048] 3D ultrasound image data,

[0049] Cone-beam computed tomography (CBCT) image data;

[0050] Magnetic resonance imaging data;

[0051] Anatomical atlas model data; and

[0052] Reconstructed volumetric image data is generated by reconstructing X-ray image data representing one or more two-dimensional projections of a vascular region.

[0053] The actual intervention device shape data 130 may include one or more of the following:

[0054] Computed tomography image data;

[0055] Contrast-enhanced computed tomography image data;

[0056] Cone-beam computed tomography (CBCT) image data;

[0057] Fiber optic shape sensing position data generated by multiple fiber optic shape sensors mechanically coupled to the interventional device, such as the sensors disclosed in document WO2007 / 109778A1;

[0058] Electromagnetic tracking position data generated by one or more electromagnetic tracking sensors or transmitters mechanically coupled to the interventional device, such as the sensors disclosed in document WO 2015 / 165736 A1;

[0059] Dielectric mapping position data generated by one or more dielectric sensors mechanically coupled to the interventional device, such as the sensors disclosed in document US 2019 / 254564A1; and

[0060] Ultrasonic tracking position data generated by one or more ultrasonic tracking sensors or transmitters mechanically coupled to the interventional device, such as that disclosed in document WO 2020 / 030557 A1.

[0061] In some examples, the actual intervention device shape data 130 may alternatively be provided by the kinematic model of the continuous robotic system.

[0062] In some implementations, the coordinate systems of volumetric image data 110, X-ray image data 120, and real-world interventional device shape data 130 can be registered with each other to facilitate the calculation of the first loss function 160 and / or constraint operations.

[0063] When volumetric image data 110, X-ray image data 120, or real-world interventional device shape data 130 are provided by image data, segmentation techniques can be used as appropriate to extract the interventional device and vascular system. Segmentation techniques such as thresholding, template matching, active contouring, applying region growing algorithms, model-based segmentation, manual segmentation, and neural network-based segmentation methods can be used. The segmentation result can be represented as a binary mask, spatial coordinates in image space, parametric equations, 2D spline curves, wireframe models, etc. Various suitable segmentation techniques are disclosed in the document "Bloodvessel segmentation algorithms – Review of methods, datasets and evaluation metrics" by Moccia, S. et al. (Computer Methods and Programs in Biomedicine, Vol. 158, pp. 71-91).

[0064] Figure 6 This is a flowchart illustrating a first example of a method for providing a neural network for predicting the three-dimensional shape of an interventional device, and further includes optional operations S170 and S180. Figure 6 The flowchart shown corresponds to Figure 4 The flowchart in the document includes operations S170 and S180, which are independent and optional, and can also be implemented together.

[0065] In operating S170, refer to the above. Figure 4The described method includes calculating the estimated uncertainty of the 3D shape of the interventional device predicted by the neural network 150, S170. In one example, the value representing the uncertainty can be calculated by comparing the output with the value of a loss function for the predicted 3D shape of the interventional device. This allows the network to learn features associated with higher errors in the 3D estimation of the device and therefore with higher uncertainty, such as a shortened 2D view of the device. In another example, a dropout technique can be used to estimate the uncertainty in the network prediction. Neurons in the neural network are randomly selected and their outputs are removed from the computation. By repeatedly running the inference using dropout, slightly different prediction outputs can be obtained each time the inference is run. The mean and variance of the prediction outputs can then be calculated, where the mean indicates the final output and the variance indicates the uncertainty. For example, if the variance is high, then the uncertainty of the network output is also high. In another example, the neural network can be a Bayesian network that directly predicts a distribution defined by the mean and variance, and the output 3D shape of the device is a sample from that distribution. In yet another example, the uncertainty can be calculated by processing the prediction output of the neural network, for example by calculating the distance between a point on the 3D shape of the interventional device and the centerline of the shape. Alternatively, the estimated uncertainty can be based, for example, on the difference between the predicted three-dimensional shape of the interventional device constrained by the vascular region 150 and the volumetric image data 110 representing the three-dimensional shape of the vascular region. The estimated uncertainty can be displayed, for example, numerically, or as a heatmap indicating the spatial distribution of the calculated estimation accuracy.

[0066] In operating S180, refer to the above. Figure 4 The described method includes segmenting the X-ray image data 120 received in S180 to provide one or more two-dimensional projections of the interventional device, and inputting the X-ray image data received in S150 into a neural network 140, which includes inputting the segmented received X-ray image data into the neural network 140. Various segmentation techniques can be used in this operation, including those mentioned above.

[0067] In some implementations, training the S140 neural network 140 includes constraint conditioning such that the three-dimensional shape of the interventional device predicted by the neural network 150 satisfies one or more mechanical constraints of the interventional device. Mechanical constraints may, for example, represent the length of the interventional device in the predicted three-dimensional shape, or the maximum curvature of the predicted three-dimensional shape of the interventional device, or the continuity of the predicted three-dimensional shape of the interventional device. These and other constraints may be implemented in the conditioning S160 operation by one or more additional loss functions, for example, adding a regularization component that penalizes the distance between predicted points and the centerline of the shape, or the distance between predicted points and their nearest neighbors, such as multi-scale loss deployed in a global nested edge detector, geodesic loss, or a loss that operates on the parametric representation of the shape to promote continuity (e.g., spline curve), the values ​​of which are also minimized during the training operation S140. Mechanical constraints may be provided, for example, in the form of a model of the interventional device.

[0068] In some embodiments, the location of the anatomical feature can be used to further constrain and adjust in operation S160, thereby improving the accuracy of the interventional device's predicted three-dimensional shape. For example, if dense bone structures are detected in X-ray image data 120 and also in volumetric image data 110, these can be used to improve the accuracy of the interventional device's predicted three-dimensional shape. In these embodiments, the received volumetric image data 110 also represents the three-dimensional shape of the anatomical feature, and the received X-ray image data 120 also represents a two-dimensional projection of the anatomical feature; and training the neural network 140 in S140 further includes:

[0069] The neural network 140 is trained to predict the position of anatomical features relative to the three-dimensional shape of the interventional device based on the received X-ray image data 120; and

[0070] The adjustment is constrained based on the difference between the predicted position of the anatomical features relative to the three-dimensional shape of the interventional device and the position of the anatomical features relative to the three-dimensional shape of the vascular region in the received volumetric image data 110.

[0071] Figure 7 This is a schematic diagram illustrating a second example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. Figure 7 The method shown in the figure corresponds to Figure 5 The method, and further includes a projection operation labeled “projection”, wherein the three-dimensional shape of the interventional device predicted by the neural network 150 is projected onto the surface to obtain a “2D projection” 180. Figure 7 The illustrated method also includes calculating a second loss function 170 based on the difference between the 2D projection 180 and the input X-ray image data 120. Figure 7In the illustrated method, the value of the second loss function is also used to adjust the parameters of the neural network 140 during training. More specifically... Figure 7 The method shown in the figure corresponds to Figure 4 or Figure 6 The method involves adjusting the parameters of the neural network 140 in S160 based on a second loss function 170, which represents the difference between a two-dimensional projection 180 of the three-dimensional shape of the interventional device predicted by the neural network 150 and the received X-ray image data 120; the two-dimensional projection 180 of the three-dimensional shape of the interventional device and the received X-ray image data are projected onto a common surface. This is achieved by projecting the three-dimensional shape of the interventional device predicted by the neural network 150 onto a two-dimensional surface of the X-ray detector providing the X-ray image data 120. Typically, this common surface will be planar, or include planar segments, or in some embodiments may include a curved surface. In doing so, the second loss function provides additional constraints for adjusting the parameters of the neural network in operation S160. Using the second loss function in this way improves the accuracy of the predicted three-dimensional shape of the interventional device.

[0072] The aspects of the training method described above can be provided by a processing device including one or more processors configured to execute the method. The processing device can be, for example, a cloud-based processing system, a server-based processing system, or a host-based processing system, and in some examples, its one or more processors may include one or more neural processors or neural processing units (NPUs), one or more CPUs, or one or more GPUs. It is also contemplated that the processing system can be provided by a distributed computing system. The processing device can communicate with one or more non-transitory computer-readable storage media that collectively store instructions for executing the method and associated data.

[0073] The example described above for the trained neural network 140 can be used to make predictions on new data during inference. The trained neural network can be deployed, for example, to systems such as desktop computers, laptops, tablets, mobile phones, etc. For example, inference can be performed by a central processing unit (CPU), GPU, NPU, TPU, on a server, or in the cloud. Figure 8 This is a flowchart illustrating an example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. (Reference) Figure 8 A computer-implemented method for predicting the three-dimensional shape of an interventional device positioned within a vascular region, comprising:

[0074] Receive volumetric image data 210 representing the three-dimensional shape of the vascular region;

[0075] S220 receives X-ray image data 220 representing a two-dimensional projection of the interventional device within the vascular region; and

[0076] The received X-ray image data 220 and the received volumetric image data 210 are input into the neural network 140 in S230. The neural network 140 is trained to predict the three-dimensional shape of the interventional device constrained by the vascular region 150 based on the received X-ray image data 220 and the received volumetric image data 210. In response to the input S230, the neural network 140 is used to predict the three-dimensional shape of the interventional device constrained by the vascular region 150 based on the received X-ray image data 220 and the received volumetric image data 210 in S240.

[0077] The predicted three-dimensional shape of an interventional device constrained by a vascular region can be output by displaying the predicted three-dimensional shape on a display device or storing it in a memory device, etc.

[0078] By providing the three-dimensional shape of the interventional device constrained by the vascular region, the above method provides improved visualization of the device and vascular system without requiring additional X-ray imaging.

[0079] The volumetric image data 210 used by the trained neural network 140 can be provided from any of the sources mentioned above. For example, in some embodiments, the volumetric image data 210 is provided from preoperative computed tomographic images of the vascular system of the same patient. However, in some embodiments, where the vascular system does not exhibit high inter-patient variability, a model of a vascular region that may originate from different patients or a statistical model of a vascular region that may originate from multiple different patients can be used as the volumetric image data 210.

[0080] In some embodiments, the location of the anatomical feature can be used to adjust the parameters of the neural network 140 during inference to improve the accuracy of the predicted three-dimensional shape of the interventional device. In these embodiments, the received volumetric image data 210 also represents the three-dimensional shape of the anatomical feature visible in the X-ray image data, and the received X-ray image data 220 also represents a two-dimensional projection of the anatomical feature; and the neural network 140 predicts the three-dimensional position of the anatomical feature relative to the three-dimensional shape of the interventional device based on the received X-ray image data 220. The parameters of the neural network 140 can be adjusted during inference to improve the accuracy of the predicted three-dimensional shape of the interventional device by constraining the adjustment based on the difference between the predicted position of the anatomical feature relative to the interventional device and the position of the anatomical feature relative to the three-dimensional shape of the vascular region in the received volumetric image data 210.

[0081] The X-ray image data 220 input to the trained neural network 140 can typically consist of one or more X-ray images. In some embodiments, a single X-ray image is provided, and the neural network 140 is trained to predict the three-dimensional shape of the interventional device constrained by the vascular region 150 based on one and only one two-dimensional projection of the interventional device within the vascular region and received volumetric image data 110. In other embodiments, more than one X-ray image can form the basis for the predicted three-dimensional shape of the interventional device constrained by the vascular region. In these embodiments, multiple X-ray images acquired at time intervals are provided, and the neural network 140 is trained to predict multiple three-dimensional shapes of the interventional device constrained by the vascular region 150 based on multiple two-dimensional projections of the interventional device within the vascular region and received volumetric image data 110, corresponding to multiple shapes of the interventional device in the multiple X-ray images. Multiple two-dimensional projections 180 can be generated based on each predicted three-dimensional shape. Various neural networks that can combine multiple images or image sequences as input can be used as the neural network 140, including recurrent neural networks, long short-term memory (LSTM) networks, temporal convolutional networks, transformers, and so on.

[0082] Figure 9 This is a schematic diagram illustrating an example of a method for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. Figure 9 The schematic diagram corresponds to Figure 8 The method, and further includes operation S250, which in Figure 8 The diagram is optionally illustrated using a dashed outline. In operation S250, the predicted three-dimensional shape of the interventional device, predicted by the neural network 140, is projected onto at least one surface to provide at least one predicted two-dimensional projection of the interventional device. The surface may, for example, represent the surface of the X-ray radiation receiving detector of an X-ray imaging system. The projection can be provided at any desired projection angle. In doing so, a projected image is generated, representing the image that has been provided by the X-ray imaging system at that projection angle. The surface can be planar or curved, or virtually any desired shape. The predicted two-dimensional projection(s) of the interventional device can be output by displaying the projection on a display device or storing it in a memory device, etc.

[0083] In some embodiments, the predicted two-dimensional projection of the interventional device may include a projection different from the projection of the interventional device represented by the received X-ray image data 220. The use of multiple projections generated in this way can be used to provide a “virtual” biplane projection of the interventional device. In some embodiments, a virtual multiplane projection of the interventional device can be provided by providing second, third, fourth, etc., projections of the predicted three-dimensional shape of the interventional device through operation S250. In some embodiments, multiple surfaces may intersect each other. Projections onto multiple intersecting surfaces are often referred to as “X-plane” images. The generation of projections different from the projection of the interventional device represented by the received X-ray image data 220 alleviates the need to take multiple X-ray images at different projection angles. Furthermore, it provides the desired projection when the desired projection cannot be achieved due to the position of the patient or bed.

[0084] In some embodiments, the two-dimensional projection of the interventional device is the same projection as that of the interventional device represented by the received X-ray image data 220; the two-dimensional projection of the three-dimensional shape of the interventional device, and the received X-ray image data, are projected onto a common surface. In these embodiments, reference above... Figure 7 The second loss function 170 described can be used to adjust the parameters of the neural network 140 during inference to improve the accuracy of the intervention device in predicting the three-dimensional shape.

[0085] In some embodiments, the volumetric image data 210 used by the neural network 140 can be acquired during the interventional procedure. In these embodiments, in addition to the patient's vascular region, the three-dimensional volumetric image data can include the three-dimensional shape of the interventional device at the time the volumetric image is acquired. If the neural network's prediction is accurate, the three-dimensional shape of the interventional device predicted by the neural network 140 based on one or more two-dimensional X-ray images of the interventional device acquired after the volumetric image acquisition should correspond to the three-dimensional shape of the interventional device in the volumetric image. This correspondence can be evaluated by one or more additional loss functions, the values ​​of which are also minimized during the training operation S140. The loss function can measure the similarity between the known device shape and the predicted device shape, or it can compare the positions of the known device shape and the predicted device shape in the constrained vascular region, etc. In some embodiments, a two-dimensional projection of the three-dimensional shape of the interventional device in the volumetric image can be appended as "Frame 0" to a two-dimensional X-ray image of the interventional device acquired after the volumetric image acquisition to create a sequence; the two-dimensional projection of the three-dimensional shape of the interventional device, and the received X-ray image data, are projected onto a common surface. The neural network 140 can be trained to predict the three-dimensional shape of the interventional device in one of the input X-ray images (e.g., the most recent X-ray image) based on the sequence generated above and the received volumetric image data, or multiple three-dimensional shapes of the interventional device constrained by the vascular region, corresponding to multiple shapes of the interventional device in multiple X-ray images. In the case of predicting multiple three-dimensional shapes of the interventional device corresponding to multiple shapes of the interventional device in multiple X-ray images, both a first loss function and a second loss function can be used to adjust the parameters of the neural network 140 during inference to improve the accuracy of the predicted three-dimensional shape of the interventional device; the first loss function compares the known three-dimensional device shape with the three-dimensional device shape predicted from “frame 0”; the second loss function compares each two-dimensional projection S250 of the predicted three-dimensional shape of the interventional device with the input two-dimensional X-ray image of the interventional device.

[0086] A system for predicting the three-dimensional shape of interventional devices is also provided. Furthermore, Figure 10 This is a schematic diagram of a system 300 for predicting the three-dimensional shape of an interventional device according to some aspects of this disclosure. System 300 includes one or more processors 390 configured to perform one or more of the operations described above with respect to the computer-implemented reasoning methods. The system may also include an X-ray imaging system, such as... Figure 10The C-arm X-ray imaging system illustrated herein, or an X-ray imaging system with another type of support structure for the C-arm, including O-arms and U-arms, etc., can generate X-ray image data 220 representing a two-dimensional projection of the interventional device within the vascular region, which is used as input to an inference method. System 300 may also include, for example, Figure 10 The illustration includes one or more displays, and / or user interface devices (such as a keyboard or touchscreen), and pointing devices (such as a mouse, and / or a hospital bed) for the execution of the control method.

[0087] The examples above are to be understood as illustrative rather than restrictive in relation to this disclosure. Other examples are also contemplated. For example, examples described with respect to computer-implemented methods may also be provided by a computer program product, or by a computer-readable storage medium, or by a processing apparatus, or by system 300 in a corresponding manner. It should be understood that features described with respect to any example may be used alone or in combination with other described features, and may also be used in combination with one or more features of another example or in combination with other examples. Furthermore, equivalents and modifications not described above may be employed without departing from the scope of the invention as defined in the claims. In the claims, the word "comprising" does not exclude other elements or operations, and the words "a" or "an" do not exclude a plurality. Although specific measures are recited in dissimilar dependent claims, this does not indicate that combinations of these measures cannot be advantageously used. Any reference numerals in the claims should not be construed as limiting their scope.

Claims

1. A computer-implemented method for providing a neural network for predicting the three-dimensional shape of an interventional device positioned within a vascular region, the method comprising: Receive (S110) volumetric image data representing the three-dimensional shape of the blood vessel region (110). Receive (S120) X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region (120). Receive (S130) real-world interventional device shape data (130), the real-world interventional device shape data representing the three-dimensional shape of the interventional device within the vascular region corresponding to the one or more two-dimensional projections of the interventional device; and The neural network (140) is trained (S140) to predict the three-dimensional shape of the interventional device constrained by the vascular region (150) based on received X-ray image data (120) and received volumetric image data (110): the received X-ray image data (120) and the received volumetric image data (110) are input (S150) into the neural network (140); and the parameters of the neural network (140) are adjusted (S160) based on a first loss function (160), the first loss function representing the difference between the three-dimensional shape of the interventional device predicted by the neural network (150) and the received actual interventional device shape data (130); and the adjustment is constrained such that the three-dimensional shape of the interventional device predicted by the neural network (150) is adapted to fit within the three-dimensional shape of the vascular region represented by the received volumetric image data (110).

2. The computer-implemented method according to claim 1, wherein, The parameters of the neural network (140) are adjusted (S160) based on a second loss function (170), which represents the difference between the two-dimensional projection (180) of the three-dimensional shape of the interventional device predicted by the neural network (150) and the received X-ray image data (120); the two-dimensional projection (180) of the three-dimensional shape of the interventional device and the received X-ray image data are projected onto a common surface.

3. The computer-implemented method according to claim 1 or claim 2, further comprising calculating (S170) the estimation uncertainty of the three-dimensional shape of the interventional device predicted by the neural network (150).

4. The computer-implemented method according to any one of claims 1-2, wherein, The volumetric image data (110) includes one or more of the following: Computed tomography image data; Contrast-enhanced computed tomography image data; 3D ultrasound image data; Cone-beam computed tomography (CBCT) image data; Magnetic resonance imaging data; Anatomical atlas model data; and Reconstructed volumetric image data generated by reconstructing X-ray image data representing one or more two-dimensional projections of the vascular region.

5. The computer-implemented method according to any one of claims 1-2, comprising: Segmenting (S180) the received X-ray image data (120) to provide the one or more two-dimensional projections of the interventional device, and wherein inputting (S150) the received X-ray image data (120) into the neural network (140) includes inputting the segmented received X-ray image data into the neural network (140).

6. The computer-implemented method according to any one of claims 1-2, wherein, The actual situation intervention device shape data (130) includes one or more of the following: Computed tomography image data; Contrast-enhanced computed tomography image data; Cone-beam computed tomography (CBCT) image data; Fiber optic shape sensing position data generated by multiple fiber optic shape sensors mechanically coupled to the interventional device; Electromagnetic tracking position data generated by one or more electromagnetic tracking sensors or transmitters mechanically coupled to the interventional device; Dielectric mapping position data generated by one or more dielectric sensors mechanically coupled to the interventional device; as well as Ultrasonic tracking position data generated by one or more ultrasonic tracking sensors or transmitters that are mechanically coupled to the interventional device.

7. The computer-implemented method according to any one of claims 1-2, wherein, Training (S140) the neural network (140) includes constraining the adjustment such that the three-dimensional shape of the interventional device predicted by the neural network (150) satisfies one or more mechanical constraints of the interventional device.

8. The computer-implemented method according to any one of claims 1-2, wherein, The neural network (140) includes one or more of the following: convolutional neural network, encoder-decoder network, generative adversarial network, capsule network, regression network, reinforcement learning agent, recurrent neural network, long short-term memory network, temporal convolutional network, and transformer.

9. The computer-implemented method according to any one of claims 1-2, wherein, The received volumetric image data (110) further represents the three-dimensional shape of the anatomical feature, wherein the received X-ray image data (120) further represents the two-dimensional projection of the anatomical feature; and wherein training (S140) the neural network (140) further includes training the neural network (140) to predict the position of the anatomical feature relative to the three-dimensional shape of the interventional device based on the received X-ray image data (120); and further includes constraining the adjustment based on the difference between the predicted position of the anatomical feature relative to the three-dimensional shape of the interventional device and the position of the anatomical feature relative to the three-dimensional shape of the vascular region in the received volumetric image data (110).

10. A computer-implemented method for predicting the three-dimensional shape of an interventional device positioned within a vascular region, the method comprising: Receive (S210) volumetric image data representing the three-dimensional shape of the blood vessel region (210). Receive (S220) X-ray image data representing one or more two-dimensional projections of the interventional device within the vascular region; and The received X-ray image data (220) and the received volumetric image data (210) are input (S230) into a neural network (140) trained to predict the three-dimensional shape of the interventional device constrained by the vascular region (150) based on the received X-ray image data (220) and the received volumetric image data (210), and in response to the input (S230), the neural network (140) is used to predict (S240) the three-dimensional shape of the interventional device constrained by the vascular region (150) based on the received X-ray image data (220) and the received volumetric image data (210).

11. The computer-implemented method of claim 10, further comprising projecting (S250) a predicted three-dimensional shape of the interventional device predicted by the neural network (140) onto at least one surface to provide at least one predicted two-dimensional projection of the interventional device.

12. The computer-implemented method according to claim 11, wherein, The projection (S250) includes projecting the predicted three-dimensional shape of the interventional device, predicted by the neural network (140), onto a plurality of intersecting surfaces.

13. The computer-implemented method according to any one of claims 10-12, wherein, The neural network (140) is trained to predict the three-dimensional shape of the interventional device constrained by the vascular region (150) based on one and only one two-dimensional projection of the interventional device within the vascular region and the received volumetric image data (110).

14. A system (300) for predicting the three-dimensional shape of an interventional device disposed within a vascular region; said system comprising one or more processors (390) configured to perform the method of claim 10.

15. A computer program product comprising instructions which, when executed by one or more processors (390), cause the one or more processors to perform the method according to claim 1 or claim 10.

Images