Determining the location of an interventional device
Patent Information
- Application Number
- JP2023528478
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2020-11-24
- Filing Date
- 2021-11-18
- Publication Date
- 2024-11-07
- Estimated Expiration
- Not applicable · inactive patent
AI Technical Summary
Existing interventional medical procedures face challenges in accurately determining the position of devices like guidewires and catheters under X-ray imaging due to high interpatient variability, obscuration by dense anatomical structures, and radiolucent sections, which impede navigation and visualization.
A computer-implemented method using a neural network to predict the position of interventional device portions by training on temporal shape data and ground truth position data, employing LSTM cells to adjust parameters based on a loss function, improving visibility and navigation accuracy.
Enhances the visibility and navigation of interventional devices by accurately predicting their positions, even in challenging anatomical environments, thereby improving procedural accuracy and efficiency.
Abstract
Description
[Technical Field]
[0001] FIELD OF THE DISCLOSURE Computer-implemented methods, processing devices, systems, and computer program products are disclosed that relate to determining the position of portions of an interventional device. [Background technology]
[0002] Many interventional medical procedures are performed under live x-ray imaging. The two-dimensional images generated during live x-ray imaging assist physicians by providing visualization of both the anatomical structures and the interventional devices, such as guidewires and catheters, used in the procedure.
[0003] As an example, endovascular procedures require interventional devices to be navigated to specific locations within the cardiovascular system. Navigation often begins at femoral, brachial, radial, jugular, or leg access points, from which the interventional device is routed through the vasculature to the location where the imaging or therapeutic procedure will occur. The vasculature typically has high inter-patient variability, even higher during disease, which can hinder navigation of the interventional device. For example, navigation through an abdominal aortic aneurysm through the ostia of the renal vessels can be difficult because the aneurysm reduces the ability to use the vessel wall to aid in device positioning and cannulation.
[0004] During such procedures, portions of interventional devices, such as guidewires and catheters, may become obscured or even invisible under x-ray imaging, further hindering navigation of the interventional device. The interventional device may, for example, be hidden behind dense anatomical structures. Radiolucent sections of the interventional device and image artifacts may also confound determination of the path of the interventional device within the anatomical structures. Summary of the Invention [Problem to be solved by the invention]
[0005] To address these shortcomings, various techniques have been developed, including the use of radiopaque fiducial markers on the interventional device and the interpolation of segmented images. However, there remains room for improvement in determining the position of an interventional device under x-ray imaging. [Means for solving the problem]
[0006] According to a first aspect of the present disclosure, a computer-implemented method is provided for providing a neural network for predicting the position of each of a plurality of portions of an interventional device, the method comprising: Time steps t1..t n receiving temporal shape data representing the shape of the interventional device in a sequence of receiving S12 interventional device ground truth position data representing the position of each of a plurality of portions of the interventional device at each time step in the sequence; training the neural network to predict, for each current time step in the sequence, a position of each of a plurality of positions of the interventional device at the current time step in the sequence from the temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence by inputting received temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence into the neural network and adjusting parameters of the neural network based on a loss function representing the difference between the predicted position of each part of the interventional device at the current time step and the position of each corresponding part of the interventional device 100 at the current time step from the received interventional device ground truth position data; Includes:
[0007] According to a second aspect of the present disclosure, a computer-implemented method for predicting the position of each of a plurality of portions of an interventional device is provided, the method comprising: receiving temporal shape data representing a shape of an interventional device at a sequence of time steps; inputting received temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence into a neural network trained to predict the position of each of a plurality of portions of the interventional device at a current time step in the sequence from the temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence; and generating, in response to the input, a predicted position of each of the plurality of portions of the interventional device at the current time step in the sequence using the neural network; Includes:
[0008] Further aspects, features, and advantages of the present disclosure will become apparent from the following description of examples which proceeds with reference to the accompanying drawings. [Brief explanation of the drawings]
[0009] [Figure 1] 1 shows an x-ray image of the human anatomy, including the tips of catheters and guidewires. [Figure 2] 1 is a flowchart of an exemplary method for providing a neural network for predicting a position of a portion of an interventional device, according to some aspects of the present disclosure. [Figure 3] FIG. 1 is a schematic diagram illustrating an example method for providing a neural network for predicting a position of a portion of an interventional device, according to some aspects of the present disclosure. [Figure 4] FIG. 1 is a schematic diagram illustrating an example of an LSTM cell. [Figure 5] 1 is a flowchart illustrating an exemplary method for predicting a position of a portion of an interventional device, according to some aspects of the present disclosure. [Figure 6] An x-ray image of the human anatomy is shown, including a catheter and guidewire, with the predicted location of the otherwise invisible portion of the guidewire indicated. [Figure 7] FIG. 2 is a schematic diagram illustrating a system 200 for predicting the position of a portion of an interventional device. DETAILED DESCRIPTION OF THE INVENTION
[0010] Examples of the present disclosure are provided with reference to the following description and drawings. In this description, for purposes of explanation, many specific details of several examples are set forth. Reference herein to an "example," "implementation," or similar language means that a feature, structure, or characteristic described in connection with an example is included in at least that example. It should also be understood that features described in connection with one example may also be used in other examples, and that not all features are necessarily duplicated in each example for the sake of brevity. For example, features described in connection with a computer-implemented method may be correspondingly implemented in a processing device, a system, and a computer program product.
[0011] In the following description, reference will be made to a computer-implemented method involving predicting the position of an interventional device within the vasculature. Reference will be made to a live X-ray imaging procedure in which an interventional device in the form of a guidewire is navigated within the vasculature. However, it should be understood that the example computer-implemented methods disclosed herein may be used with other types of interventional devices other than guidewires, such as, but not limited to, catheters, intravascular ultrasound imaging devices, optical coherence tomography devices, introducer sheaths, laser atherectomy devices, mechanical atherectomy devices, blood pressure and / or flow sensor devices, TEE probes, needles, biopsy needles, ablation devices, balloons, or endografts. It should also be understood that the example computer-implemented methods disclosed herein may be used with other types of imaging procedures, such as, but not limited to, computed tomography imaging, ultrasound imaging, and magnetic resonance imaging. It should also be understood that the example computer-implemented methods disclosed herein may be used with interventional devices positioned in anatomical regions other than the vasculature, including, but not limited to, the digestive tract, respiratory pathways, urinary tract, etc., if desired.
[0012] It should be noted that the computer-implemented methods disclosed herein may be provided as a non-transitory computer-readable storage medium including stored computer-readable instructions that, when executed by at least one processor, cause the at least one processor to perform the method. In other words, the computer-implemented methods may be implemented in a computer program product. The computer program product may be provided by dedicated hardware or hardware capable of executing software in association with appropriate software. When provided by a processor or "processing unit," the functionality of the method features may be provided by a single dedicated processor, by a single shared processor, or by multiple individual processors, some of which may be shared. The explicit use of the terms "processor" or "controller" should not be construed as exclusively referring to hardware capable of executing software, and may implicitly include, but is not limited to, digital signal processor (DSP) hardware, read-only memory (ROM), random access memory (RAM), non-volatile storage, etc., for storing software. Furthermore, examples of the present disclosure may take the form of a computer program product accessible from a computer-usable or computer-readable storage medium, the computer program product providing program code for use by or in connection with a computer or any instruction execution system. For purposes of this description, a computer-usable or computer-readable storage medium can be any device that can have, store, communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, device, or propagation medium. Examples of computer-readable media include semiconductor or solid state memory, magnetic tape, removable computer diskettes, random access memory "RAM," read-only memory "ROM," rigid magnetic disks, and optical disks.Current examples of optical disks are compact disk read-only memory (CD-ROM), optical disk read / write (CD-R / W), and Blu-ray. TM , including DVD.
[0013] FIG. 1 shows an X-ray image of the human anatomy, including the tip of a catheter and a guidewire. In FIG. 1, dense regions of the anatomy, such as the ribs, are highly visible as darker areas in the image. The catheter and the tip of the guidewire extending therefrom are also highly visible. However, soft tissue regions, such as the vasculature, are poorly visible and therefore provide little guidance during navigation under X-ray imaging. Image artifacts, labeled "obstructions" in FIG. 1, and other features in X-ray images that appear similar to the guidewire may also interfere with clear visualization of the guidewire in X-ray images. A further complication is that some portions of the guidewire are poorly visible under X-ray imaging. For example, the tip of the guidewire is clearly visible in FIG. 1, but portions of the guidewire are poorly visible or completely invisible, such as the portion labeled "invisible portion." The visibility of portions of other interventional devices may similarly be impaired when imaged by X-ray and other imaging systems.
[0014] The present inventors have discovered an improved method for determining the position of a portion of an interventional device. Figure 2 is a flowchart of an exemplary method for providing a neural network for predicting the position of a portion of an interventional device, according to some aspects of the present disclosure. This method will be described with reference to Figures 2-4. With reference to Figure 2, the method includes providing a neural network for predicting the position of each of a plurality of portions of the interventional device 100; A sequence of time steps t1..t n a step S110 of receiving temporal shape data 110 representing the shape of the interventional device 100 at For each time step t1 in the sequence. nreceiving interventional device ground truth position data 120 representing positions of a plurality of portions of the interventional device 100 at step S120; For each current time step t in the sequence n for one or more historical time steps t1..t in the sequence n-1 inputting S140 the received temporal shape data 110 representing the shape of the interventional device 100 at the current time step t n , the predicted positions 140 of each part of the interventional device 100 at the current time step t from the received interventional device ground truth position data 120. n and adjusting S150 the parameters of the neural network 130 based on a loss function that represents the difference between the positions of each corresponding part of the interventional device 100 at one or more historical time steps t1 in the sequence. n-1 From the temporal shape data 110 representing the shape of the interventional device 100 at the current time step t n a step S130 of training a neural network 130 to predict the position 140 of each of the plurality of portions of the interventional device 100 at the Includes:
[0015] 3 is a schematic diagram illustrating an example method for providing a neural network for predicting a position of a portion of an interventional device according to some aspects of the present disclosure. FIG. 3 includes a neural network 130 including a plurality of long short-term memory (LSTM) cells. The operation of each LSTM cell is described below with reference to FIG. 4.
[0016] Referring to FIG. 3, during the training operation S130, for example, at time steps t1..t n-1 Temporal shape data 110, which may be in the form of a time sequence of segmented X-ray images generated in step t1, t2, t3, t4, t5, t6, t7, t8, t9, t10, t11, t12, t13, t14, t15, t16, t17, t18, t19, t20, t21, nrepresents the shape of the guidewire in x-ray images. Various known segmentation techniques can be used to extract the shape of the interventional device or guidewire from x-ray images. For example, a segmentation technique such as that disclosed in Honnorat, N., et al., "Robust guidewire segmentation through boosting, clustering and linear programming," 2010 IEEE International Symposium on Biomedical Imaging: From Nano to Macro, Rotterdam, 2010, pp. 924-927, may be used. The x-ray image provides the shape of the guidewire in two dimensions. Portions of the guidewire can then be identified, for example, by defining one or more groups of pixels on the guidewire in the x-ray image. These portions can be defined arbitrarily or at regular intervals along the length of the guidewire. The position of each portion of the guidewire is then determined at each time step t1...t2. n The image may be provided in two dimensions at
[0017] In general, the temporal shape data 110 may include a temporal sequence of x-ray images including the interventional device 100, or a temporal sequence of computed tomography images including the interventional device 100, or a temporal sequence of ultrasound images including the interventional device 100, or a temporal sequence of magnetic resonance images including the interventional device 100, or a temporal sequence of positions provided by a plurality of electromagnetic tracking sensors or emitters mechanically coupled to the interventional device 100, or a temporal sequence of positions provided by a plurality of fiber optic shape sensors mechanically coupled to the interventional device 100, or a temporal sequence of positions provided by a plurality of dielectric sensors mechanically coupled to the interventional device 100, or a temporal sequence of positions provided by a plurality of ultrasound tracking sensors or emitters mechanically coupled to the interventional device 100. Thus, it is also contemplated that the temporal shape data 110 may be provided as three-dimensional shape data.
[0018] Time steps t1..tn-1 Simultaneously with the generation of the X-ray images at each time step t1..t n Corresponding interventional device ground truth position data 120 may also be generated, representing the position of each of the multiple portions of the interventional device 100 at time t1. The interventional device ground truth position data 120 serves as training data. In the example shown in FIG. 3, the ground truth position data 120 is provided by the same X-ray image data used to provide the temporal shape data 130. Thus, it is contemplated to provide the ground truth position data as two-dimensional position data. Furthermore, the same position of the guidewire is determined at each time step t1..t2. n may be used to provide both ground truth position data 120 and temporal shape data 110 in
[0019] Providing ground truth position data 120 from other sources is also contemplated. In some implementations, ground truth position data 120 may originate from a source different from that of temporal shape data 110. Ground truth position data 120 may be provided, for example, by a temporal sequence of computed tomography images including interventional device 100. Thus, providing ground truth position data as three-dimensional position data is also contemplated. The computed tomography images may be, for example, cone beam computed tomography, CBCT, or spectral computed tomography images. Ground truth position data 120 may alternatively be provided by a temporal sequence of ultrasound images including interventional device 100, or indeed a temporal sequence of images from another imaging modality, such as magnetic resonance imaging.
[0020] In other implementations, the ground truth position data 120 may be provided by a tracking sensor or emitter mechanically coupled to the interventional device. In this regard, an electromagnetic tracking sensor or emitter, such as that disclosed in WO 2015 / 165736 A1, or a fiber optic shape sensor, such as that disclosed in WO 2007 / 109778 A1, a dielectric sensor, such as that disclosed in U.S. Patent Application Publication No. 2019 / 254564 A1, or an ultrasonic tracking sensor or emitter, such as that disclosed in WO 2020 / 030557 A1, is mechanically coupled to the interventional device 100 and is transmitted at each time step t1...t n can be used to provide a time sequence of positions corresponding to the position of each sensor or emitter in
[0021] When the ground truth position data 120 is provided by a source different from the source of the temporal shape data 110, the coordinate system of the ground truth position data 120 may be aligned to the coordinate system of the temporal shape data 110 to facilitate calculation of the loss function.
[0022] The temporal shape data 110 and ground truth position data 120 may be received from a variety of sources including databases, imaging systems, computer-readable storage media, the cloud, etc. The data may be received using any form of data communication such as wired or wireless data communication, via the internet, Ethernet, or by transferring the data using a portable computer-readable storage medium such as a USB memory device, optical disk, or magnetic disk.
[0023] Returning to FIG. 3, neural network 130 then calculates one or more historical time steps t1..t n-1 The image is trained to predict σ from temporal shape data 110 of the shape of a temporal sequence of X-ray images at a current time step t in the sequence. nTraining of the neural network 130 of FIG. 3 may be performed as described in more detail in Alahi, A., et al., "Social LSTM: Human Trajectory Prediction in Crowded Spaces," 2016 IEEE Conference on Computer Vision and Pattern Recognition "CVPR," 10.1109 / CVPR.2016.110. The input to the neural network 130 is the position of each of the multiple parts of the interventional device. For each part of the interventional device, the LSTM cell calculates the position of each of the multiple parts of the interventional device over one or more history time steps t1...t2. n-1 Using the position of the part from the current time step t n Predict the position of the part.
[0024] In some implementations, neural network 130 includes multiple outputs, each output representing the current time step t in the sequence. n 3, training is performed over one or more historical time steps t1..t n-1 The positions of each part of the interventional device are input to the neural network from the current time step t n , and the predicted positions 140 of each part of the interventional device 100 at the current time step t from the received interventional device ground truth position data 120. n This is done by adjusting the parameters of the neural network using a loss function that represents the difference between the position of each corresponding part of the interventional device 100 at t and the position of each corresponding part of the interventional device 100 at t. In these embodiments, each output of the neural network 130 may include a corresponding input, as shown in FIG. 3, where the corresponding input is a function of one or more historical time steps (t 1 ..t 2 ) in the sequence. n-1) and receive temporal shape data (110) representing the shape of the interventional device (100) in the form of a position of a portion of the interventional device in the X-ray image. As described above, the position of a portion of the guidewire can be identified from the input X-ray image 110, for example, by defining one or more groups of pixels on the guidewire within the segmented X-ray image.
[0025] More specifically, the neural network 130 shown in FIG. 3 includes multiple outputs, each of which is a function of the current time step (t n ) based at least in part on the predicted positions of one or more neighboring portions of the interventional device (100) at the current time step (t n ) to predict the positions 140 of different parts of the interventional device 100. This functionality is provided by a pooling layer, which allows for the sharing of information in the hidden state between neighboring LSTM cells. This captures the influence of neighboring parts of the device on the predicted device part's motion. This improves prediction accuracy by preserving position information about neighboring parts of the interventional device, and thus the continuity of the interventional device shape. The extent of the neighborhood, i.e., the number of neighbors and the extent to which their positions are used to predict the position of the interventional device part, can range from the immediate neighborhood to the entire interventional device. The extent of the neighborhood can also depend on the flexibility of the device. For example, a rigid device may use a relatively large neighborhood, while a flexible device may use a relatively small neighborhood. An alternative to the illustrated pooling layer includes applying constraints to the output of the neural network by removing predicted positions that break the continuity of the device or predict the curvature of the interventional device beyond a predetermined value.
[0026] In some implementations, the neural network shown in FIG. 3 may be provided by an LSTM cell. For example, each block labeled LSTM in FIG. 3 may be provided by an LSTM cell as shown in FIG. 4. The position of each part of the interventional device may be predicted by an LSTM cell. However, the features of the items labeled LSTM may be provided to the LSTM by other types of neural networks. The features of the items labeled LSTM may be provided by, for example, a recurrent neural network, an RNN, a convolutional neural network, a CNN, a temporal convolutional neural network, a TCN, and a transformer.
[0027] The training operation S130 starts at the current time step t n , and the predicted positions 140 of each part of the interventional device 100 at the current time step t from the received interventional device ground truth position data 120. n This involves adjusting S150 the parameters of the neural network 130 based on a loss function that represents the difference between the positions of each corresponding part of the interventional device 100 at the time of the intervention.
[0028] The training operation S130 will be described in more detail with reference to Figure 4, which is a schematic diagram illustrating an exemplary LSTM cell. The LSTM cell shown in Figure 4 can be used to implement the LSTM cell of Figure 3. Referring to Figure 4, the LSTM cell receives three inputs: h t-1 , c t-1 and x t and two outputs, namely, h t and c t The sigma and tanh labels represent the sigmoid and tanh activation functions, respectively, and the "x" and "+" symbols represent pointwise multiplication and pointwise addition operations, respectively. At time t, the output h t represents the hidden state, and the output c t represents the cell state, and the input x t represents the current data input. Moving from left to right in Figure 4, the first sigmoid activation function provides the forget gate. Its input, i.e., ht-1 and x t Each represents the previous cell's hidden state, and the current data input is concatenated and passed through a sigmoid activation function. The output of the sigmoid activation function is then used to calculate the previous cell state c t-1 The forget gate is multiplied by the current cell state c t It controls the amount of information from previous cells included in the cell state c. Its contribution is included via point-wise addition, represented by the "+" sign. Moving to the right of Figure 1, the input gates control the amount of information from previous cells included in the cell state c. t Controls the update of the hidden state of the previous cell h t-1 , and the current data input x t are concatenated and passed through a sigmoid activation function, and then through a tanh activation function. The pointwise multiplication of the outputs of these functions determines the amount of information added to the cell state via pointwise addition, represented by the "+" sign. The result of the pointwise multiplication is the current cell state c t To provide the previous cell state c t-1 Moving further to the right of Figure 1, the next hidden state h t The hidden state contains information about the previous input and is used for prediction. The next hidden state h t To determine the hidden state of the previous cell, h t-1 , and the current data input x t-1 are concatenated and passed through a sigmoid activation function. The new cell state c t is passed through the tanh activation function. The output of the tanh and sigmoid activation functions is then used to generate the next hidden state h t are multiplied to determine the information in
[0029] As with other neural networks, training of the LSTM cell shown in Figure 4, and therefore the neural networks in which it may be used, is performed by adjusting parameters, namely, weights and biases. Referring to Figure 4, the four activation functions at the bottom of Figure 4 are controlled by weights and biases, which are identified in Figure 4 using the symbols w and b. In the LSTM cell shown, each of these four activation functions typically has two weight values, namely, each x t One for the input, and one for each h t-1 , one for σ and one bias value b. Thus, the example LSTM cell shown in Figure 4 typically includes eight weight parameters and four bias parameters.
[0030] Thus, the operation of the LSTM cell shown in Figure 4 is controlled by the following equation: f t =σ((w hf ×h t-1 )+(w xf ×x t )+b f ) Equation 1 u t =σ((w hu ×h t-1 )+(w xu ×x t )+b u ) Equation 2 c~ t =tanh((w hc ×h t-1 )+(w xc ×x t )+b c ) Equation 3 o t =σ((w ho ×h t-1 )+(w xo ×x t )+b o ) Equation 4 c t =[c~ t +u t ]+[c t-1 +f t ] Formula 5 y t =[o t×tanhc t ] Formula 6
[0031] Thus, training a neural network including the LSTM cell shown in FIG. 4, as well as other neural networks, involves adjusting the weights and biases of the activation function. Supervised learning involves providing the neural network with a training data set that includes input data and corresponding expected output data. The training data set represents input data that the neural network is likely to use for post-training analysis. During supervised learning, the weights and biases are automatically adjusted so that when presented with input data, the neural network accurately provides the corresponding expected output data.
[0032] Training a neural network typically involves inputting a large training data set into the neural network and iteratively adjusting neural network parameters until the trained neural network provides accurate outputs. Training is usually performed using dedicated neural processors such as graphics processing units (GPUs), neural processing units (NPUs), or tensor processing units (TPUs). Thus, training typically employs a centralized approach, in which cloud- or mainframe-based neural processors are used to train the neural network. Following its training with the training data set, the trained neural network can be deployed to analyze new input data, a process called "inference." Processing requirements during inference are significantly lower than those required during training, allowing neural networks to be deployed on a variety of systems, such as laptop computers, tablets, and mobile phones. Inference can be performed, for example, on a central processing unit (CPU), GPU, NPU, TPU, server, or in the cloud.
[0033] As outlined above, the process of training a neural network involves adjusting the weights and biases of the activation function. In supervised learning, the training process automatically adjusts the weights and biases so that, when presented with input data, the neural network accurately provides the corresponding expected output data. The value of the loss function, or error, is calculated based on the difference between the predicted output data and the expected output data. The value of the loss function may be calculated using functions such as negative log-likelihood loss, mean squared error, Huber loss, or cross entropy. During training, the value of the loss function is typically minimized, and training is terminated when the value of the loss function meets a stopping criterion. In some cases, training is terminated when the value of the loss function meets one or more of several criteria.
[0034] Various methods for solving loss minimization problems are known, including gradient descent and quasi-Newton algorithms. Various algorithms have been developed to implement these methods and their variations, including, but not limited to, stochastic gradient descent (SGD), batch gradient descent, mini-batch gradient descent, Gauss-Newton, Levenberg-Marquardt, momentum methods, Adam, Nadam, Adagrad, Adadelta, RMSProp, and Adamax "optimizers." These algorithms use chaining techniques to calculate the derivatives of the loss function with respect to the model parameters. This process is called backpropagation because the derivatives are calculated starting at the last layer, or output layer, and moving toward the first layer, or input layer. These derivatives inform the algorithm how the model parameters must be adjusted to minimize the error function. That is, adjustments to the model parameters are made starting at the output layer and working backward through the network until the input layer is reached. In the first training iteration, the initial weights and biases are often randomized. The neural network then predicts output data, which is similarly random. Backpropagation is then used to adjust the weights and biases. The training process is performed iteratively by adjusting the weights and biases at each iteration. Training is terminated when the error, or the difference between the predicted output data and the expected output data, is within an acceptable range for the training data or some validation data. The neural network can then be deployed, and the trained neural network will make predictions for new input data using the trained values of its parameters. If the training process is successful, the trained neural network will accurately predict the expected output data from the new input data.
[0035] 3 and 4 serve as examples only, and it should be understood that other neural networks may similarly be used to implement the functionality of the methods described above. Alternative neural networks to the LSTM neural network 130 may also be trained to perform the desired predictions during the training operation S130, including, but not limited to, recurrent neural networks, RNNs, convolutional neural networks, CNNs, temporal convolutional neural networks, TCNs, and transformers.
[0036] In some implementations, the training of the neural network in operation S130 is further constrained. In one exemplary implementation, the temporal shape data 110, or the interventional device ground truth position data 120, comprises a temporal sequence of X-ray images including the interventional device 100, where the interventional device 100 is positioned within a vascular region. In this example, the method described above further comprises: extracting vascular image data representing the shape of a vascular region from the temporal shape data 110 or the interventional device ground truth position data 120; The step S130 of training the neural network 130 further includes: current time step t n The adjustment S150 includes constraining the predicted position 140 of each of the plurality of regions of the interventional device 100 in the vascular region to fit within the shape of the vascular region represented by the extracted vascular image data.
[0037] In doing so, the position of the portion of the interventional device can be predicted with greater accuracy. The constraints can be applied by calculating a second loss function based on the constraints and incorporating this second loss function, along with the previous loss function, into the objective function, the value of which is then minimized during the training operation S130.
[0038] Vascular image data representative of the shape of a vascular region can be determined from X-ray images, for example, by providing the temporal sequence 110 of X-ray images as one or more Digital Subtraction Angiography (DSA) images.
[0039] Aspects of the training methods described above may be provided by a processing device having one or more processors configured to execute the methods. The processing device may be, for example, a cloud-based processing system, a server-based processing system, or a mainframe-based processing system, and in some examples, the one or more processors may include one or more neural processors or neural processing units (NPUs), one or more CPUs, or one or more GPUs. It is also contemplated that the processing device may be provided by a distributed computing system. The processing device may be in communication with one or more non-transitory computer-readable storage media that collectively store instructions for executing the methods and associated data.
[0040] The above-described example of a trained neural network 130 can be used to make predictions about new data in a process called "inference." The trained neural network may be deployed on a system, such as a laptop computer, tablet, or mobile phone. The inference may be performed, for example, on a central processing unit "CPU," GPU, NPU, server, or in the cloud. FIG. 5 is a flowchart illustrating an example method for predicting the position of a portion of an interventional device, according to some aspects of the present disclosure. Referring to FIG. 5, a computer-implemented method for predicting the position of each of multiple portions of the interventional device 100 includes: A sequence of time steps t1..t n a step S210 of receiving temporal shape data 210 representing the shape of the interventional device 100 at The temporal shape data 210 is calculated for one or more historical times t1..t n-1 From the temporal shape data 210 representing the shape of the interventional device 100 at the current time step t nt1..t2. The neural network 130 is trained to predict the position 140 of each of the plurality of portions of the interventional device 100 at one or more historical times t1..t2. n-1 and a step S220 of inputting received temporal shape data 210 representing the shape of the interventional device 100 at a current time step t in the sequence, using a neural network in response to the input S220. n generating a predicted position 140 for each of the plurality of portions of the interventional device 100 at step S230; Includes:
[0041] current time step t n The predicted positions 140 of each of the portions of the interventional device 100 in the image may be output, such as by displaying the predicted positions 140 on a display device or storing them in a memory device.
[0042] As mentioned above, the temporal shape data 210 may include, for example, a temporal sequence of X-ray images including the interventional device 100; or a temporal sequence of computed tomography images including the interventional device 100; or a temporal sequence of ultrasound images including the interventional device 100; or a temporal sequence of positions provided by multiple electromagnetic tracking sensors or emitters mechanically coupled to the interventional device 100; or a temporal sequence of positions provided by multiple fiber optic shape sensors mechanically coupled to the interventional device 100; or a temporal sequence of positions provided by multiple dielectric sensors mechanically coupled to the interventional device 100; or a temporal sequence of positions provided by a plurality of ultrasonic tracking sensors or emitters mechanically coupled to the interventional device 100; Includes:
[0043] The current time step t in the sequence predicted by the neural network 130 nThe predicted positions 140 of each of the portions of the interventional device 100 at the current time step t when the temporal shape data 210 does not clearly identify the interventional device. n Thus, in one example, the temporal shape data 210 includes a temporal sequence of X-ray images including the interventional device 100, and the inference method includes: current time step t n displaying a current X-ray image from the temporal sequence corresponding to displaying within the current x-ray image a predicted position 140 of at least one portion of the interventional device 100 within the current x-ray image; Includes:
[0044] In doing so, the inference method mitigates drawbacks associated with poor visibility on the part of the interventional device.
[0045] Other sources of temporal shape data 210, such as those described above during the training operation S130, may similarly be received and displayed in a corresponding manner during inference.
[0046] 6 shows an x-ray image of the human anatomy, including a catheter and guidewire, with the predicted locations of otherwise invisible portions of the guidewire displayed. The predicted locations of portions of the interventional device 100 may be displayed, for example, as an overlay on the current x-ray image.
[0047] In some examples, a confidence score may also be calculated and displayed on the display for the displayed position of the interventional device. The confidence score may be provided as an overlay on the predicted position(s) of the portion(s) of the interventional device 100 in the current x-ray image. The confidence score may be provided, for example, as a heat map of the probability that the device position is correct. Other forms of presenting the confidence score may alternatively be used, including displaying its numerical value, displaying a bar graph, etc. The confidence score may be calculated using the output of the neural network, which may be provided, for example, by a Softmax layer at the output of each LSTM cell in FIG. 3.
[0048] A system 200 is also provided for predicting the position of each of a plurality of portions of the interventional device 100. Additionally, FIG. 7 is a schematic diagram illustrating a system 200 for predicting the position of a portion of an interventional device. The system 200 includes one or more processors 270 configured to perform one or more of the operations described above in connection with the computer-implemented inference method. The system may also include an imaging system, such as the X-ray imaging system 280 shown in FIG. 7 or another imaging system. In use, the X-ray imaging system 280 captures time steps t1...t in the form of a sequence of X-ray images. n Temporal shape data 210 representing the shape of the interventional device 100 at time t may be generated, which may be used as input to the method. The system 200 may also include one or more display devices, such as shown in FIG. 7, and / or user interface devices, such as a keyboard, and / or a pointing device, such as a mouse, for controlling the execution of the method, and / or a patient bed.
[0049] The above examples should be understood as illustrating, not limiting, the present disclosure. Further examples are contemplated. For example, examples described in connection with a computer-implemented method may also be provided correspondingly by a computer program product, a computer-readable storage medium, a processing device, or the system 200. It should be understood that features described in connection with any one example may be used alone or in combination with other described features, and may also be used in combination with one or more features of another example or with combinations of other examples. Furthermore, equivalents and modifications not described above may also be used without departing from the scope of the invention as defined in the appended claims. In the claims, the word "comprises" does not exclude other elements or operations, and the indefinite articles "a" or "an" do not exclude a plurality. The mere fact that certain features are recited in mutually different dependent claims does not indicate that a combination of these features cannot be advantageously used. Any reference signs in the claims should not be construed as limiting their scope.
Claims
1. 1. A computer-implemented method for providing a neural network for predicting a position of each of a plurality of portions of an interventional device, comprising: receiving temporal shape data representing a shape of an interventional device at a sequence of time steps, the temporal shape data comprising a temporal sequence of x-ray images generated at the time steps, the x-ray images including the interventional device; receiving interventional device ground truth position data representative of a position of each of a plurality of portions of the interventional device at each time step in the sequence; For each current time step in the sequence, inputting the received temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence into a neural network; and adjusting parameters of the neural network based on a loss function that represents a difference between the predicted position of each part of the interventional device at the current time step and a position of each corresponding part of the interventional device at the current time step from the received interventional device ground truth position data; training a neural network to predict a position of each of the plurality of portions of the interventional device at a current time step in the sequence from the temporal shape data representing a shape of the interventional device at one or more historical time steps in the sequence by 23. A computer-implemented method comprising:
2. The interventional device ground truth position data, a temporal sequence of computed tomography images including the interventional device; or a time sequence of ultrasound images including the interventional device; or a time sequence of magnetic resonance images including said interventional device; or a temporal sequence of positions provided by a plurality of electromagnetic tracking sensors or emitters mechanically coupled to the interventional device; or a temporal sequence of positions provided by a plurality of fiber optic shape sensors mechanically coupled to the interventional device; or a temporal sequence of positions provided by a plurality of dielectric sensors mechanically coupled to the interventional device; or a temporal sequence of positions provided by a plurality of ultrasonic tracking sensors or emitters mechanically coupled to the interventional device; The computer-implemented method of claim 1 , comprising:
3. The computer-implemented method of claim 1 or 2, wherein the neural network has a plurality of outputs, each output configured to predict a position of a different portion of the interventional device at the current time step in the sequence.
4. 4. The computer-implemented method of claim 1, wherein each output is configured to predict a position of the different part of the interventional device at the current time step in the sequence based at least in part on the predicted positions of one or more nearby parts of the interventional device at the current time step.
5. the neural network comprises a LSTM neural network having a plurality of LSTM cells, each LSTM cell having an output configured to predict a position of a different portion of the interventional device at the current time step in the sequence; 4. The computer-implemented method of claim 3, wherein for each LSTM cell, the cell is configured to predict a position of the portion of the interventional device at the current time step in the sequence based on the received temporal shape data representing the shape of the interventional device at the one or more historical time steps in the sequence and predicted positions of one or more neighboring portions of the interventional device at the current time step.
6. The computer-implemented method of claim 1, further comprising a step of segmenting each X-ray image in the sequence to respectively provide a shape of the interventional device at each time step.
7. The interventional device is positioned within a vascular region, and the computer-implemented method further comprises: extracting vascular image data representative of a shape of the vascular region from the temporal shape data or the interventional device ground truth position data; Further comprising: and training the neural network further comprises constraining the adjusting step such that the predicted positions of each of the plurality of portions of the interventional device at the current time step in the sequence fit within a shape of the vascular region represented by the extracted vascular image data.
10. The computer-implemented method of claim 1.
8. The computer-implemented method of claim 7 , wherein the temporal sequence of X-ray images comprises digital subtraction angiography images.
9. 2. The computer-implemented method of claim 1, wherein the interventional device comprises a guidewire, a catheter, an intravascular ultrasound imaging device, an optical coherence tomography device, an introducer sheath, a laser atherectomy device, a mechanical atherectomy device, a blood pressure and / or flow sensor device, a TEE probe, a needle, a biopsy needle, an ablation device, a balloon, or an endograft.
10. 10. A processing device for providing a neural network for predicting the position of each of a plurality of parts of an interventional device, the processing device having one or more processors configured to perform the method of any one of claims 1 to 9.
11. 1. A computer-implemented method for predicting a position of each of a plurality of portions of an interventional device, comprising: receiving temporal shape data representing a shape of an interventional device at a sequence of time steps, the temporal shape data comprising a temporal sequence of x-ray images generated at the time steps, the x-ray images including the interventional device; inputting the received temporal shape data representing the shape of the interventional device at one or more historical time steps in the sequence into a neural network provided by the computer-implemented method of any one of claims 1 to 9; in response to the input, using the neural network to generate a predicted position for each of the plurality of portions of the interventional device at the current time step in the sequence, the plurality of portions including at least one portion that is not visible in the x-ray image generated at the current time step; 13. A computer-implemented method comprising: displaying a current X-ray image from the time sequence corresponding to the current time step; displaying the predicted location of at least one invisible portion of the interventional device within the current X-ray image; 12. The computer-implemented method of claim 11, comprising:
13. calculating a confidence score for the at least one displayed location; displaying the calculated confidence scores; The computer-implemented method of claim 11 , further comprising:
14. 12. A system for predicting a position of each of a plurality of portions of an interventional device, the system having one or more processors configured to perform the method of claim 11.
15. A computer program comprising instructions which, when executed by one or more processors, cause the one or more processors to perform the method of claim 1 or claim 11.
