Device and method for near real-time prediction of exposure maps for interventional radiology
A multilayer neural network-based method generates real-time exposure maps during interventional radiology, addressing the inadequacies of existing methods by providing accurate, patient-specific dose control.
Patent Information
- Application Number
- JP2024522083
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-22
- Filing Date
- 2022-10-17
- Publication Date
- 2024-11-13
AI Technical Summary
Current methods for predicting radiation exposure maps during interventional radiology are not sufficiently accurate or real-time, failing to account for patient-specific anatomy and leading to inadequate dose control during procedures.
A method utilizing a multilayer neural network, trained with a simulation module and patient data, generates real-time exposure maps by correlating radiographic images with acquisition parameters, allowing for dynamic dose control during surgery.
The method provides near real-time, accurate exposure maps that enable physicians to adjust radiation doses based on patient-specific anatomy, reducing the risk of deterministic and non-deterministic side effects.
Smart Images

Figure 2024541833000001_ABST
Abstract
Description
[Technical field]
[0001] Various exemplary embodiments relate to interventional radiology.
[0002] The invention applies in particular to the prevention of the risks associated with exposure due to this type of radiation, by enabling a near real-time prediction of the exposure map of the intervention environment. [Background technology]
[0003] Interventional radiology refers to all medical procedures performed by a radiologist under radiation control that allow the treatment or invasive diagnosis of a number of pathologies. The principle of interventional radiology is therefore to access lesions located inside the body in order to carry out diagnostic (e.g. sampling) or therapeutic actions (aimed at treatment, repair, closure, etc.).
[0004] Thus, in the context of therapeutic intervention, interventional radiology represents an alternative to traditional surgical procedures by making it possible, in certain cases, to intervene in the body through natural channels and without the creation of surgical openings.
[0005] Different techniques may be used, such as, for example, fluoroscopy, x-ray scanning, ultrasound scanning, MRI (magnetic resonance imaging), etc.
[0006] Thus, interventional radiology involves irradiating parts of the human body from within the body for the duration of the procedure. It therefore poses radiation risks to the patient. Thus, prolonged exposure to X-rays can cause deterministic side effects (skin bonding, hair loss, cataracts, etc.) and non-deterministic side effects (especially cancer).
[0007] International Commission on Radiological Protection (ICRP) CRP) and other organizations have emphasized the importance of patient dose control during interventional radiology.
[0008] The imaging system captures real-time measurements of metrics such as dose area product (DAP). Several systems have been commercialized with capabilities for time measurement. Examples of such systems are Toshiba's Dose Tracking System, GE Healthcare's DoseMap, and Siemens' CAREwatch.
[0009] However, these systems do not take into account the specific anatomy of the patient, or even the position and orientation of the interventional radiology device. Therefore, the information provided is necessarily approximate. Moreover, they estimate the dose only on the skin surface.
[0010] Another family of proposals is based on more complex simulations of the dose, in particular by Monte Carlo methods. The simulations may take into account different input parameters characterizing the patient and the X-ray device, but suffer from very high calculation times that are not suitable for real time. It is therefore not possible to use such an approach to allow the physician to control the dose that is applied to the patient during the procedure.
[0011] Dedicated computers such as Graphics Processing Units (GPUs) It has been proposed to deploy Monte Carlo simulation methods on computing processors.
[0012] Examples of published papers include Badal A, Badano A. <Accelerating Monte Carlo simulations of photon transport in a voxelized geometry using a massively parallel graphics processing unit> >in Med Phys.2009 Nov;36(11):4878-80.doi:10.1118 / 1.3231824.PMID:19994495, or J.Bert, H.Perez-Ponce, Z.El Bitar, S.Jan, Y.Boursier, et al..<<Geant4-based Monte Carlo simulations on GPU for medical applications> >in Physics in Medicine and Biology, IOP Publishing, 2013, 58, pp. 5593-5611.
[0013] These proposals indeed reduce the computation times, but remain unsuitable for real-time, being of the order of 10 seconds for the first proposal (and which does not take into account the specific anatomy of the patient) and 2 minutes for the second proposal.
[0014] Further proposals that take advantage of recent advances in the field of multi-layer neural networks are beginning to emerge. These include Roser, P., Zhong, X., Birkhold, A., Strobel, N., Kowarschik, M., Fahrig, R., & Maier, A. (2019). "Physics-Driven Learning of X-ray Skin Dose Distribution in Interventional Procedures" in Medical physics, 46, 4654-4665.
[0015] However, this approach is also not feasible for several reasons: it relies on ray tracing preprocessing, which is also expensive in terms of computation time and not suitable for real-time. The paper shows that the calculation of the irradiation map takes about 2 seconds. Moreover, the accuracy of the estimation is not very good, since it gives an average error of 8% with a peak of 22%, and furthermore, the map is concentrated in the most irradiated areas, thus reducing the information provided to the physician. The method also relies on a three-dimensional model of each exposed organ of the patient to proceed with the training of the predictive model. In practice, this type of information is not widely available and may only hinder the construction of a training set to build a valid predictive model. In fact, in the paper, the training is carried out only on three "phantoms".
[0016] Therefore, there is no solution in the state of the art that is both near real-time and sufficiently accurate. Summary of the Invention [Problem to be solved by the invention]
[0017] It is an object of the present invention to provide a method and system that at least partially alleviates the above mentioned drawbacks.
[0018] More specifically, according to an embodiment, the invention aims to provide an illumination map in real time or quasi-real time, i.e. in a time that allows the physician to control the patient's illumination during the operation, in particular so that the physician is not hindered in his operation by the time required to acquire the illumination map, and thus can react during the operation according to the dynamically provided information. [Means for solving the problem]
[0019] In a first exemplary embodiment, a method for acquiring an exposure map of a patient during interventional radiology, comprising: a learning phase consisting of submitting to the multi-layer neural network a learning set comprising associations between a first input tensor comprising data of a first radiographic image of an intervention area of a patient and of first acquisition parameters of an interventional radiology device, and labels corresponding to an exposure map obtained by simulation by a simulation module from said first radiographic image and said first acquisition parameters; A method is provided that includes a prediction step for the given patient that includes acquiring a second stream of acquisition parameters of the interventional radiology device, creating a second input tensor that includes data of a second radiological image and second acquisition parameters of the given patient, submitting the second input tensor to the neural network, and retrieving an exposure map prediction.
[0020] In preferred embodiments, the invention comprises one or more of the following features, which may be used separately or in part or in whole combination with each other: During a learning phase, a plurality of data of a learning set is generated by varying, for a first radiological image, said first acquisition parameters among the possible parameters. The simulation is carried out by means of a Monte Carlo method adapted to a graphics processor. The neural network in question is of the U-Net type. - a first sub-network having data of the first or second radiological image as input and including a first series of convolutional and pooling layers and a second series of deconvolutional and convolutional layers such that the output of the neural network is the same size as the input, the output of each deconvolutional layer being concatenated with a corresponding output in the first series, and a second sub-network having the first or second acquisition parameters as input, respectively, the output of the second sub-network being concatenated with the output of the last pooling layer of the first series. said input tensor being a concatenation of said data of a first radiological image and said first acquisition parameters. the radiological image is a computed tomography scan.
[0021] Another embodiment relates to a computer readable medium encoding a machine executable program of instructions for performing the methods described above.
[0022] According to another embodiment, the present invention provides a system for obtaining a prediction for an exposure map of a patient during interventional radiology, comprising at least one simulation module and a multi-layer neural network module including a multi-layer neural network, the system comprising: During a learning phase, a learning set is submitted to the multi-layer neural network, the learning set including associations between a first input tensor including data of a first radiographic image of an intervention area of a patient and first acquisition parameters of an interventional radiology device, and labels corresponding to an illumination map obtained from the simulation module based on the first radiographic image and the first acquisition parameters; In a prediction step for a given patient, a second stream of acquisition parameters is acquired from the interventional radiology device, a second input tensor is created that includes data of a second radiological image of the given patient and the second acquisition parameters, and the second input tensor is generated based on the second radiological image data. and submitting the second input vector to a neural network to derive an illumination map prediction.
[0023] Further features and advantages of the invention will become apparent from the following description of preferred embodiments of the invention, given by way of example with reference to the accompanying drawings, in which: [Brief description of the drawings]
[0024] Some embodiments will now be described, by way of example only, and with reference to the accompanying drawings. [Figure 1] 2 illustrates a schematic representation of an example of a functional architecture for the learning phase, according to an embodiment of the present invention; [Diagram 2] 2 illustrates generally an example of a functional architecture for the prediction stage according to an embodiment of the present invention. [Diagram 3] 1 illustrates a schematic diagram of the basic architecture of a classical neural network. [Figure 4] 1 illustrates generally an example of a multi-layer neural network, according to an embodiment of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0025] One objective of the present invention is to obtain an exposure map of a patient during a given patient's radiation exposure.
[0026] This irradiation map relates the amount or dose of irradiation to points in the space contained in the patient's environment affected by the interventional radiology. These points are more or less numerous according to the selected sampling of the area.
[0027] The irradiation maps are based on data acquired intraoperatively with interventional radiology.
[0028] According to the present invention, the illumination map remains relevant to the acquired data and is acquired fast enough to enable its generation for the physician during surgery, in other words, the illumination that occurs between the time of data acquisition and the time of generation of the illumination map can be considered negligible.
[0029] The generation of the exposure map may enable the physician to react to the information represented by this map during surgery, which may be referred to as real-time in the sense that a human physician can dynamically react to the exposure map calculated from the measured data.
[0030] According to the present invention, the generation of the illumination map is based on a predictive model built by a multi-layer neural network.
[0031] From a very high-level perspective, a multi-layer neural network can be viewed as a black box whose internal parameters must be adjusted during a learning (or training) phase by presenting it with both input data and a desired output (or "label"). The error between this desired output and the "natural" output of the network allows the parameters to be adjusted slightly to reduce the error. By presenting a large number of such "input data I desired output" pairs, the network learns to respond correctly and provide good outputs when presented with new unlabeled input data.
[0032] According to an embodiment of the present invention, the neural network used may be a multi-layer perceptron. Other neural network architectures may also be possible. In particular, a convolutional neural network may be used (ConvNet or CNN).
[0033] An example of a multi-layer perceptron is shown in FIG.
[0034] A multilayer perceptron (MLP) is a neural network in which information comes out of the input layer L1. force layer L k It is a type of artificial neural network that is organized into several layers in which the inputs flow only to the inputs, and is therefore a feedforward network. Each layer L1, L2, L3...L k are n1, n2, n3...n k The neurons in the last layer (called "output") are the output of the neural network and represent the predictions of the model in response to the inputs provided on the L1 layer.
[0035] In a multilayer perceptron, each neuron n i,j The output of the next layer L i+1Conversely, it is connected to all neurons in the previous layer L i-1 It receives as input the outputs of all neurons in. In Figure 3, for clarity, only some of the connections are represented by directed arrows.
[0036] Each connection is associated with a weight. The set of weights forms the internal parameters of the neural network. They must be determined during the learning (or training) phase and then allow the prediction, by generalization, of the output value from a new input vector presented on the L1 input layer.
[0037] Each neuron n i,j Classically, one performs a weighted sum of these inputs by the weights of the associated connections and then applies an activation function to this sum.
[0038] There are several techniques for determining the internal parameters and thresholds of a network through learning. One example is the Stochastic Gradient Descent (SGD) algorithm. backprop is a generalized algorithm, e.g. Le Cun, Yann A., et al. "Efficient backprop. Neural networks: Tricks of the trade", Springer Berlin Heidelberg, 2012.9-48. Another example is ADAM, first described in Diederik P. Kingma and Jimmy Lei Ba. "Adam: A method for stochastic optimization", 2014. arXiv:1412.6980v9, or Tijmen Tieleman and Geoffrey Hinton, "Lecture 6.5-rmsprop: Divide the gradient by a running average of its recent magnitude”,COURSERA:neural networks for RMSprop, specifically described in machine learning, 4(2):26:31, 2012.
[0039] The process of obtaining the illumination map also includes a learning phase that makes it possible to determine the internal parameters of the multi-layer neural network.
[0040] Once these are determined, the multi-layer neural network builds a predictive model that can be used in the predictive phase during the intervention (or surgery).
[0041] FIG. 1 illustrates such a learning stage, according to one embodiment of the present invention.
[0042] According to the invention, the learning phase is carried out by the simulation module 12 on the basis of the radiation image 31 of the intervention area of the patient and on the basis of the acquisition parameters 32 of the interventional radiology device, by the learning input tensor itself, which comprises the data of the radiation image 31 of the intervention area of the patient and the acquisition parameters 32 of the interventional radiology device, and on the basis of the radiation image 31 and these acquisition parameters 32, by the simulation module 12, which corresponds to the exposure map 33 obtained by simulation. It consists of submitting the multi-layer neural network 13 to a training set that includes associations between the labels. The training set may be stored in a database or data repository 20.
[0043] In the conventional manner, this learning phase is carried out externally prior to surgery involving interventional radiology.
[0044] The learning phase is based on a set of radiological images.
[0045] According to an embodiment, the radiological image is a three-dimensional image.
[0046] However, according to some other embodiments, other options may be contemplated: for example, 2D images may be considered, and the volume of the patient's intervention area captured by the radiology device is represented by a set of 2D images, each of which represents a layer of the volume.
[0047] Also 4D images may be considered where the fourth dimension is time (video streams).
[0048] Experiments show that predicting a 3D image all at once using a multi-layer neural network is faster than sequentially predicting a series of 2D images that make up the 3D image.
[0049] In the following, the expression "three-dimensional images" is preferred, but the patent scope of the present invention covers 2D and 4D images as well.
[0050] The radiological image (eg, a three-dimensional image) captures the patient's intervention area relevant to interventional radiology, ie the location of the patient's body where interventional radiology is intended to have an effect.
[0051] These radiological images (e.g. three-dimensional images) may be of different types, for example they may be CT scans or computerised tomography (CT). It is a medical imaging technique that consists in measuring the absorption of X-rays by tissues, which are then digitized by computer processing and finally reconstruct a 2D or 3D image of the anatomical structures. To obtain the data, a tomographic or "slice" analysis technique is used, by subjecting the patient to a scanning X-ray beam.
[0052] Typically, such 3D images are taken prior to any intervention (or surgery) in order to plan it, so that a large number of such 3D images can be easily acquired to form a training set.
[0053] Furthermore, for each of these 3D images there may be associated acquisition parameters 32 of the interventional radiology device used for the procedure, the interventional radiology device being designed to acquire approximately two-dimensional images of the patient's intervention area that are a function of these acquisition parameters.
[0054] In practice, the interventional radiology device generates a two-dimensional image stream for the physician. It should be noted that these images are different from the images 31 acquired before the intervention.
[0055] The acquisition parameters are primarily the three-axis coordinates x, y, z of the interventional radiology device, the device orientation angles α, β, and the device tube voltage KV, where α may represent the rotation around the patient's longitudinal axis, whereas β may represent the rotation around the horizontal axis.
[0056] Depending on the type of interventional radiology device, other parameters may be used.
[0057] In accordance with the present invention, the input tensor of the neural network 13 may be formed by concatenating data from a CT scan image with the acquisition parameters (x, y, z, α, β, KV) 32 .
[0058] According to one embodiment of the present invention, for each 3D image 31, multiple training set data are generated by varying the acquisition parameters (x, y, z, α, β, KV) among the possible parameters.
[0059] To do this, an excursion interval can be defined for each of the parameters x, y, z, α, β, KV and the respective sampling rate. Thus, for the same image for which the illumination map 33 can be simulated, a large number of possible parameterizations can be defined.
[0060] Each 3D image 31 / acquisition parameters 32 pair may be generated at the input of a simulator 12 that provides at its output a simulated illumination map 33. This illumination map 33 may be submitted to the neural network 13 as labels.
[0061] In other words, at each iteration of the training phase, the neural network 12: - a three-dimensional image 31 and acquisition parameters 32, - comprising an input tensor associated with a label consisting of this same three-dimensional image 31 and a simulated illumination map 33 based on these same acquisition parameters 32.
[0062] In other words, the neural network enables correlation between the three-dimensional image data, including the corresponding acquisition map, and the acquisition parameters (x, y, z, α, β, KV) 32.
[0063] According to the present invention, the input tensor of the neural network contains only data generated by the interventional radiology device, and no complex pre-processing needs to be performed to convert this acquired data into other data (apart from possible simple pre-processing such as rescaling, resampling, etc., which do not change the nature of the data).
[0064] As a result, the training set is cheap to build and a large number of labeled examples can be generated simply by extracting the 3D images. This training set can be further enhanced by varying the acquisition parameters according to embodiments of the present invention.
[0065] Unlike the method described in the above-mentioned Roser et al. paper, there is no need to deploy expensive pre-processing that would slow down the construction of a sufficiently large learning base, nor to acquire voxelized 3D models of each patient organ, the latter requirement being, in fact, detrimental to the construction of a large learning base.
[0066] However, it is important that the training base is large enough to allow the neural network to converge to a steady state that minimizes its cost function, so that an effective predictive model can be trained. A training base that is too small will give either erroneous or very inaccurate predictions, or correct predictions but with examples that are only slightly different from those in the training set (reduced generalization ability).
[0067] Thus, the present invention can also utilize available databases that collect patient data to perform this learning phase.
[0068] For example, a database of CT scan images of the pancreas exists and is documented in the paper HR Roth, L. Lu, A. Farag, H.-C. Shin, J. Liu, E. Turkbey, RM Summers, "Data from Pancreas-CT", 2016. This database contains 82 images of 512 x 512 pixels spaced at depths of 0.7 to 1 mm. From this database, a three-dimensional image with a voxel resolution of 4 arcmin x 4 arcmin x 2 arcmin can be redefined.
[0069] The simulation module 12 is capable of performing a simulation of the behavior of particles emitted by an interventional radiology device in an intervention area of interest as defined by a three-dimensional image 31 according to its acquisition parameters 32 .
[0070] The simulation may be a Monte Carlo simulation.
[0071] In particular, it may be a GPU-based Monte Carlo simulation. For example, Physics in Medicine and Biology, IOP "GPU GEant4-based Monte Carlo" as described in the paper "Geant4-based Monte Carlo simulations on GPU for medical applications" by J. Bert, H. Perez-Ponce, Z. El Bitar, S. Jan, Y. Boursier et al., in the journal Computer Vision, 2013, 58, pp. 5593-5611. The GGEMS platform for "Medical Simulations" can be used. The goal of GGEMS is to provide a flexible platform for the simulation of various medical applications. Simulation algorithms can also be modified to better fit the platform running on a GPU.
[0072] In a specific case of experimentation of the method of the present invention, the source constructed by the interventional radiology device can be defined as forming a cone of radiation of KV value. 7 The behavior of a particle can be simulated in about 6 seconds on an NVIDIA 1080 GTX Ti type GPU.
[0073] According to one embodiment of the present invention, acquisition parameters may be varied from image to image in order to increase the population of the resulting training set.
[0074] Typically, the angles α and β vary in the ranges [−50°, 90°] and [−30°, +30°], respectively, in increments of 2°.
[0075] The position of the interventional radiology device can be varied in the interval [-30cm, +30cm] for the three parameters x, y, z in 5cm increments.
[0076] Finally, the tube voltage KV can be varied in the range [80 kV, 120 kV] in 10 kV steps.
[0077] These values of intervals and steps have been determined by studying technical data of interventional radiology devices and by feedback from physicians (surgeons, radiologists, etc.).
[0078] The simulation module 12 provides an exposure map 33. The radiation map is a three-dimensional image that associates points in space with radiation doses in the simulated environment.
[0079] The simulated radiation map 33 forms the labels for the neural network 130 implemented by the neural network module 13. In a manner known per se, training involves calculating the output of the neural network 130 as a function of the input tensors and measuring a cost function that estimates the distance between that output and the associated label or illumination map, which is the desired output. The difference between these two values (in this case two tensors), evaluated by the cost function, is used to modify the parameters of the neural network, generally by gradient descent, as already explained.
[0080] According to one embodiment of the present invention, the cost function may be an evaluation of the mean squared error between the output tensor of the neural network 13 and the desired output (label) represented by the illumination map.
[0081] The learning mechanism may be optimized, for example, by the ADAM algorithm described in D.P. Kingma and J. Ba, "Adam: A Method for Stochastic Optimization", CoRR, abs / 1412.6980 (2014).
[0082] The neural network 13 may be implemented on the TensorFlow platform using the Keras API.
[0083] According to one embodiment of the present invention, the neural network is a convolutional auto-encoding multi-layer neural network.
[0084] More specifically, the neural network may be of the u-net type.
[0085] The u-net neural network is a convolutional network first described in Olaf Ronneberger, Philipp Fischer, Thomas Brox, "U-net: Convolutional networks for biomedical image segmentation", in International Conference on Medical Image Computing and Computer-Assisted Intervention, pages 234-241, Springer, 2015.
[0086] The 3D version was then developed using the 3D filter, O. Cicek, A. Abdulkadir, S. Lienkamp, T. Brox, and O. Ronneberger, "3D U-net: learning dense volumetric segmentation from sparse annotation," in Lect. Ronneberger, "3D U-net: learning dense volumetric segmentation from sparse annotation," in This was proposed in "Segmentation from sparse annotation" in Lect. Notes Computing Science, vol. 9901 LNCS, pages 424-432, 2016.
[0087] According to one embodiment of the present invention, the neural network is based on a 3D U-net architecture.
[0088] FIG. 4 illustrates an example of such an embodiment.
[0089] In particular, the neural network 130 of the neural network module 13 has a first subnetwork 131 corresponding to a 3D u-net, and an output is connected within the first subnetwork. and a second subnetwork 132 that is connected to the first subnetwork 132 .
[0090] The first sub-network 131 itself includes a first succession of layers that make up the encoding path (or downsampling, or subsampling) and a second succession of layers that make up the decoding path (or upsampling, or oversampling).
[0091] The encoding path is a sequence of convolutional and pooling layers.
[0092] More specifically, the encoding path may include a sequence of two successive 3x3x3 convolutional layers, each followed by a ReLU activation and a 2x2x2 maxPooling layer. At each subsampling (or encoding) stage, the number of filters (or features) is doubled, starting from 8 for the first convolution.
[0093] The first layer of this encoding path receives as input the tensor formed by the 3D image data 31.
[0094] The pooling operation reduces the dimensionality and captures the most relevant filters or features.
[0095] In FIG. 4, the horizontal arrows to the right correspond to these convolution operations, and the downward arrows correspond to the maxPooling operations.
[0096] Similarly, the decoding path (right in FIG. 4) comprises a succession of deconvolutional and convolutional layers.
[0097] A deconvolution (or transposed convolution, or oversampling), for example of 2x2x2, is represented by an upward arrow, while a convolution, for example of 3x3x3, is represented by a rightward arrow, as is the decoded channel.
[0098] In this way, an output (ie, illumination map 34) of the same size as the input 31 may be obtained.
[0099] This neural network is related to the family of encoder-decoder networks, but unlike conventional architectures, there is a connection between the encoding path and the decoding path. More precisely, the output of each deconvolution layer is concatenated with the corresponding output in the first series. This connection is represented by the thin arrows in Figure 4, and the empty blocks represent the output of the deconvolution layer that is concatenated with the decoding path.
[0100] This combination allows the transfer of detailed information from lower layers of the encoding path to higher layers of the decoding path, since this information allows the generation of reconstructions with fine details.
[0101] As illustrated in FIG. 4, multiple pooling (forming subsampling) and subsequent oversampling operations give the network a U-shaped (functional) form, giving the network its common name of "U-net."
[0102] The last layer of the encoding path and the first layer of the decoding path may also be considered to form a bottleneck.
[0103] According to this embodiment, the neural network originally comprises a second sub-network 132 coupled to the first sub-network 131 at the level of this bottleneck.
[0104] This second subnetwork 132 is connected to the acquisition path of the interventional radiology device. It takes parameter 32 as input.
[0105] This second sub-network can be architecturally a network of several fully connected layers (here three in Fig. 4). The output tensor 133 is concatenated with the output of the last pooling layer of the encoding path. This tensor has the same number of units as the output tensor of this last pooling layer (19 x 12 x 15 = 3420) to allow their concatenation.
[0106] According to one embodiment, the number of filters (or "features") for the convolution and deconvolution operations is empirically reduced by a factor of 8 compared to the original 3D u-net to enable faster prediction (at the prediction stage) that meets real-time requirements. However, the idea of doubling the number of filters before the pooling operation is retained.
[0107] According to one embodiment, another distinguishing feature compared to the u-net 3D scheme is the use of a padding mechanism to obtain an output of the same size as the input.
[0108] The padding aims to avoid edge effects of the recursive convolution process applied to the image.
[0109] More precisely, pixels around the image edges cannot be processed due to convolution.
[0110] For example, in the case of a 3x3x3 convolution filter, the edges around the processed image are cropped with a size of 1 pixel, so the image size is reduced by 2 pixels in each dimension after each convolution, and it is not possible to restore this dimension on the decoder path.
[0111] This can be avoided by applying zero padding to the input image before each convolution operation, which consists of adding pixels with zero values around the edges of the image.
[0112] To optimize the training time, different mechanisms can be implemented.
[0113] For example, training may be divided into two sub-phases.
[0114] In a first sub-stage, only a subset of the plurality of training set data acquired by varying the acquisition parameters among the possible parameters is used, the subset being substantially smaller than the plurality.
[0115] For example, module 11 generates only a smaller number of parameters in this sub-stage, but preferably uniformly distributed, for example using a larger step size.
[0116] For example, the orientation angle varies in 10° increments, the x, y, z position of the source varies in 15 cm increments, and the tube voltage KV varies in 40 kV increments.
[0117] This first sub-stage allows the state of the neural network to converge quickly to an acceptable state.
[0118] In the second sub-stage, the entire training set is used (ie, with smaller steps, as shown above).
[0119] Training sets may be specific to a particular area of intervention, e.g., specific training sets may be created for the head and neck, thorax, pelvis, etc. Training sets may be created for the head and neck, thorax, pelvis, etc. Training sets may also be created for the whole body.
[0120] Each training set leads to a particular state of the neural network 13. This state is embodied by the values of all its parameters (such as synaptic weights) and thus builds a particular predictive model.
[0121] Once constructed, these predictive models (states of the neural network 13) may be stored for use in the prediction stage.
[0122] Thus, during the prediction phase, the first step may be the selection of a suitable predictive model from a library of predictive models. This step may be manual, with the physician selecting the intervention area, which leads to the selection of the associated predictive model. According to other embodiments, this step may be automatic, by applying an image recognition algorithm that allows automatic recognition of the intervention area.
[0123] This prediction step consists of applying a prediction model to a given patient in order to predict a real-time radiation map to assist or guide the physician during interventional radiology.
[0124] This step is illustrated in FIG.
[0125] During the intervention, the interventional radiology device 21 generates an acquisition parameter stream 32. This stream is determined, for example, by the sampling rate at which it transmits the acquisition parameters to the radiation map generation system 10 radiology.
[0126] The parameters may be the 6-tuple (x, y, z, α, β, KV) mentioned above, or any set of parameters specific to the type of radiology device. In any case, these parameters must be the same as those used for the learning phase.
[0127] These parameters evolve over time during the intervention depending on the actions of the physician. Their values affect the patient irradiation, in particular the patient area for the parameters (x, y, z, α, β) and the instantaneous dose for the parameter KV.
[0128] Furthermore, the system 10 is provided for creating an input tensor containing a radiological image 31 of a given patient, e.g. three-dimensional data. This radiological image is of the same nature as that used in the learning phase. It may be a pre-operative image of the patient stored in the data repository 20, or an image acquired during an interventional radiology procedure. According to the invention, this image may be unique during the procedure (only the parameters change over time).
[0129] The input tensors also include parameters for acquiring the flow generated by the interventional radiology device 21 .
[0130] Sampling may be applied to receive only a sampled portion of the flow to construct this input tensor. Sampling may be determined based on the computation time per prediction.
[0131] This input tensor is created and submitted as input to the neural network 13. and a prediction of the illumination map 34 can be taken as output from the neural network 13.
[0132] This exposure map 34 is of the same type as the one provided as labels during the learning phase. In particular, it can provide an estimated radiation dose per unit volume of the intervention area of the patient during an interventional radiology procedure.
[0133] This exposure map can be viewed as a 3D image where each voxel is associated with an exposure dose (instead of colorimetric data as in classical 3D images).
[0134] Experimental measurements carried out by the inventors make it possible to estimate that, using a standard GPU graphics card, an illumination map takes less than 70 ms.
[0135] This exposure map can be used to assist the physician during his or her intervention on the patient.
[0136] Specifically, it can be displayed on a screen so that the physician can see it during the procedure.
[0137] The radiation dose may be represented by associating a different color with each interval of dose values.
[0138] Because the time to calculate a new map is very short, the display can be refreshed (approximately 15 maps per second) so that the displayed map corresponds to the physician's current time, allowing the physician to monitor the map in real time.
[0139] Also, an alert (visual, audible...) can be triggered if the radiation dose exceeds a certain predefined threshold for a particular sub-area of the intervention area.
[0140] For example, it may be possible to visually highlight areas that may be problematic because the associated radiation dose has exceeded a threshold value or because the increase in radiation dose is too great, suggesting that the threshold value may be exceeded in the future.
[0141] In contrast to some prior art proposals, the predicted radiation map is a 3D map, in other words, radiation dose estimates can be obtained not only for the skin surface, but also for the entire volume of irradiated organs and body parts.
[0142] Furthermore, the error inherent in the neural network predictions is very reasonable, less than 5% compared to the baseline Monte Carlo simulations.
[0143] Furthermore, as mentioned above, since the method uses only medical imaging data that is easy to acquire in the training phase, training sets can be constructed for different parts of the body that are large enough to allow the neural network to converge well and thus obtain good qualitative performance in the prediction phase.
[0144] In one embodiment, system 10 may be implemented by a set of circuitry co-located in a centralized server or distributed in or across a set of distributed servers, which may include a "server farm" or "cloud computing" configuration. The server may also be implemented on a single computer located in a surgical room.
[0145] The system 10 takes three-dimensional images of the patient during both the learning and prediction phases. The device may have means for communicating with a remote server to provide pre-operative images of the patient on which the procedure is being performed.
[0146] The system may also have means for communicating with an interventional radiology device 21, in particular to retrieve acquisition parameters.
[0147] These communication means may be according to the latest technology: in particular, they may be wired (Ethernet, USB, etc.) or wireless (Wi-Fi, Bluetooth, 3G / 4G / 5G cellular, etc.) communication means.
[0148] Naturally, the invention is not limited to the examples and embodiments described and represented, but is defined by the claims, in particular because of the numerous variants available to the skilled person.
Claims
1. 1. A method for acquiring an exposure map of a patient during interventional radiology, comprising: a learning phase consisting of submitting to the multi-layer neural network (130) a first input tensor comprising data of a first radiographic image (31) of an intervention area of a patient and of first acquisition parameters (32) of an interventional radiology device, and a learning set comprising associations between labels corresponding to an irradiation map (33) obtained by simulation by a simulation module (12) from said first radiographic image (31) and said first acquisition parameters (32); a prediction step for the given patient, comprising acquiring a stream of second acquisition parameters (32) of the interventional radiology device (21), creating a second input tensor comprising data of second radiological images (31) of the given patient and the second acquisition parameters, submitting the second input vector to the neural network (130), and deriving an exposure map prediction (34).
2. 2. The method of claim 1, wherein during the learning phase, the plurality of data of the learning set is generated (11) by varying the first acquisition parameters among possible parameters for a first radiological image (31).
3. The method according to claim 1 or 2, wherein the simulation is performed by a Monte Carlo method adapted for a graphics processor.
4. The method according to any one of claims 1 to 3, wherein said neural network is of the U-Net type.
5. 5. The method of claim 4, wherein the neural network (130) comprises: a first sub-network (131) having as input the data of the first or second radiological image and including a first series of convolutional and pooling layers and a second series of deconvolutional and convolutional layers such that the output of the neural network (130) is the same size as the input, the output of each deconvolutional layer being concatenated with a corresponding output in the first series; and a second sub-network (132) having as input the first or second acquisition parameters, respectively, the output of the second sub-network (132) being concatenated with the output of the last pooling layer of the first series.
6. The method according to any one of the preceding claims, wherein the input tensor is a concatenation of the data of a first radiological image (31) and the first acquisition parameters (32).
7. The method according to any one of the preceding claims, wherein the radiological image (31) is a computed tomography scan.
8. A computer readable medium encoding a machine executable program of instructions for performing the method of any one of claims 1 to 7.
9. A system (10) for obtaining a prediction for an exposure map (34) of a patient during interventional radiology, comprising at least one simulation module (12) and a multi-layer neural network module (13) including a multi-layer neural network (130), the system comprising: During the learning phase, a first radiological image (31) of the intervention area of the patient and an intervention submitting to the multi-layer neural network (130) a learning set including associations between a first input tensor including data of first acquisition parameters (32) of a functional radiology device and labels corresponding to an exposure map (33) obtained from the simulation module (12) based on the first radiation image (31) and the first acquisition parameters (32); - during a prediction phase for a given patient, acquiring a second stream of acquisition parameters (32) from the interventional radiology device (21), creating a second input tensor comprising data of a second radiological image (31) of the given patient and the second acquisition parameters, submitting the second input vector to the neural network (13), and deriving a prediction of an exposure map (34).
Citation Information
Patent Citations
Three-dimensional dose distribution prediction method based on adaptive correction adversarial network
CN111888665A
Dose distribution display apparatus and x-ray diagnostic apparatus
JP2016107060A
Tracking object recognition simulator or marker recognition simulator, moving body tracking device, and irradiation system
JP2019141109A
Systems and methods for learning models of radiation therapy treatment plans and predicting radiation therapy dose distributions - Patents.com
JP2019526380A
X-ray cone beam CT image reconstruction method
JP2021023761A
