Motion-compensated image reconstruction in an X-ray-based imaging procedure

Description

[0001] The present invention is directed to a method for generating a motion-compensated image reconstruction in an X-ray-based imaging method, wherein a plurality of projection images of an object generated during the performance of the imaging method are provided. The invention is further directed to a method for X-ray-based imaging, wherein a plurality of projection images of an object are generated and a method for generating a motion-compensated image reconstruction is performed. The invention is also directed to a data processing device, a device for X-ray-based imaging comprising an X-ray imaging modality, and a computer program product.

[0002] X-ray-based imaging techniques, particularly cone-beam computed tomography (CBCT), can be used, among other things, to assess brain structures during surgical procedures. This helps, for example, in the differential diagnosis of strokes, classifying them as ischemic or hemorrhagic. However, a major obstacle to high-quality image assessment is patient movement, which distorts the geometric information used for reconstruction. This leads to streaking or blurring artifacts.

[0003] Motion compensation techniques can be used to correct the motion retrospectively. This involves estimating a rigid motion trajectory of the patient. If the rigid motion trajectory is successfully estimated, a motion-compensated image reconstruction can be created.

[0004] In particular, the motion is compensated by optimizing an objective function. The input for the objective function contains a set of projection images and a set of parameters that define the approximate patient motion trajectory. If the projection images are denoted as I and the parameters that define the motion trajectory are denoted as X, then the problem of motion compensation can be solved by determining optimal parameters X' as follows: X ′ = argmin X c r I X , where r() denotes the image reconstruction based on the projection images and the current estimate of the motion trajectory, and c() denotes a cost function. The cost function is correlated with motion, for example, so low values ​​correspond to little motion.

[0005] As described in the publication J.Wang, R. Schaffert, A. Borsdorf, B. Heigl, X. Huang, J. Hornegger, A. Maier: "Dynamic 2-D / 3-D rigid registration framework using point-to-plane correspondence model", IEEE Transactions on Medical Imaging, Vol. 36, No. 9, pp. 1939-1954, 2017, the cost function can be based on a 2D / 3D image registration, for example.

[0006] The cost function can also be based on consistency constraints, as described in the publications A. Preuhs, M. Manhart, A. Maier: "Fast Epipolar Consistency without the Need for Pseudo Matrix Inverses", Proc. 15th International Conference on Image Formation in X-Ray Computed Tomography, pp. 202-205, 2018 and A. Preuhs, M. Manhart, E. Hoppe, M. Kowarschik, A. Maier: "Motion gradients for epipolar consistency", Proc. 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine, p. 110720D, 2019. In this case, the image reconstruction r() is not required in the above equation.

[0007] Furthermore, the cost function can be based on autofocus approaches, as described in the publication A. Preuhs, M. Manhart, P. Roser, E. Hoppe, H. Yixing, M. Psychogios, M. Kowarschik, A. Maier: "Appearance Learning for Image-based Motion Estimation in Tomography", IEEE Transactions on Medical Imaging, pp. 1-10, 2020.

[0008] One possible parameterization X is based on rotations and translations. In this case, three parameters for rotation and three for translation are used for each projection image. If the number of projection images is N, the number of required parameters is N*6. For a classic CBCT trajectory, for example, comprising 400-600 projection images, the parameter vector X would have 2400-3600 entries. This makes the optimization problem very computationally intensive.

[0009] To mitigate this problem, splines can be used for motion compensation, as described in the above-mentioned 2020 publication by Preuhs et al. In this case, the motion is parameterized by spline nodes. This greatly reduces the complexity of the motion compensation problem. For example, a single direction of motion can be approximated by 20 spline nodes, resulting in a parameter vector X with only 120 entries. Another advantage of splines is the inclusion of constraints. For example, the patient's motion is usually smooth, which is inherently taken into account by splines.

[0010] A problem with both of these parameterizations is that they allow for many motion trajectories that are physically unreasonable. Since the cost function is usually non-convex, these physically unreasonable motion trajectories can lead the optimization to local minima and prevent successful convergence. This partially prevents reliable motion compensation.

[0011] It is an object of the present invention to increase the reliability of motion compensation in image reconstruction in X-ray-based imaging methods.

[0012] This problem is solved by the respective subject matter of the independent claims. Advantageous further developments and preferred embodiments are the subject matter of the dependent claims. Regardless of the grammatical gender of a particular term, persons with male, female, or other gender identities are also included.

[0013] The invention is based on the idea of ​​choosing a parameterization of an object motion trajectory such that unrealistic or physically impossible motion configurations are not considered during optimization. To this end, the object motion trajectories are parameterized in an abstract latent space using corresponding latent vectors, and concrete object motion trajectories are generated by applying a correspondingly trained algorithm.

[0014] According to one aspect of the invention, a method for generating a motion-compensated image reconstruction in an X-ray-based imaging method is provided. The following steps i), ii), iii), and iv) are performed, but not necessarily in this order. According to step i), a plurality of projection images of an object generated during the implementation of the imaging method are provided. According to ii), a trained algorithm is provided, in particular stored on a storage device, which is trained to assign an object motion trajectory to a latent vector of a given latent space.According to step iii), an optimal object motion trajectory is determined by applying the trained algorithm to an optimal latent vector, in particular an optimal latent vector of the latent space, and the motion-compensated image reconstruction is generated depending on the plurality of projection images under the assumption that the object has moved according to the optimal object motion trajectory during the execution of the imaging process.

[0015] According to step iv), the optimal latent vector is determined by minimizing a given cost function using the latent vector as an optimization parameter. For each optimization step with a current latent vector, a current object motion trajectory is determined by applying the trained algorithm to the current latent vector. A current value of the cost function is calculated depending on the plurality of projection images, assuming that the object moved according to the current object motion trajectory during the imaging process.

[0016] Depending on the cost function used, for each of the optimization steps, for example, the current value of the cost function can be calculated depending on a given set of consistency conditions, in particular epipolar consistency conditions, concerning the plurality of projection images.

[0017] This is the case, for example, when using a cost function as described in the publications by Preuhs et al. from 2018 and 2019 mentioned above.

[0018] Alternatively, for each of the optimization steps, a current image reconstruction can be generated based on the plurality of projection images, assuming that the object moved according to the current object motion trajectory during the imaging process. The current value of the cost function is then calculated based on the current image reconstruction.

[0019] This is the case, for example, when using cost functions as described in the above-mentioned publication by Wang et al. or the above-mentioned publication by Preuhs et al. from 2020.

[0020] The method according to the invention for generating a motion-compensated image reconstruction can, in some embodiments, be a purely computer-implemented method, in which all steps of the computer-implemented method are carried out by a data processing device having at least one computing unit. In particular, the at least one computing unit is configured or adapted to carry out the steps of the computer-implemented method. For this purpose, the at least one computing unit or a storage device of the at least one computing unit can, for example, store a computer program containing instructions that, when executed by the at least one computing unit, cause the at least one computing unit to execute the computer-implemented method.

[0021] In other embodiments, the method is not purely computer-implemented, for example if the step of carrying out the imaging method, i.e. in particular of generating the projection images, is part of the method.

[0022] The X-ray-based imaging method can, for example, be a computed tomography method, in particular a cone-beam computed tomography method. The projection images correspond, in particular, to X-rays detected by an X-ray detector that have passed through the object and are correspondingly attenuated. Generating an image reconstruction based on such projection images then includes, in particular, determining corresponding attenuation values ​​in the corresponding spatial regions of the object.

[0023] Various known methods can be used for image reconstruction. In particular, a geometric operation can be assigned to each of the projection images for image reconstruction, by means of which the image reconstruction or a portion of the image reconstruction is generated from the respective projection image. It is also known to perform motion compensation in such image reconstructions in order to compensate for movements of the object, in particular rigid movements of the object, during the acquisition of the projection images. For this purpose, the geometric operations assigned to the individual projection images are corrected according to the respectively estimated or assumed object motion trajectory or the corresponding translation and rotation position of the object during the generation of the corresponding projection image.

[0024] This correction can be performed, for example, by appropriate rotations and / or translations that are present according to the object motion trajectory at the corresponding time of generation of the projection image. In other words, an object motion trajectory can be defined by a number of consecutive rotation and / or translation operations. The format in which an object motion trajectory is specified or stored can vary. Corresponding translation vectors and rotation matrices can be specified, or corresponding rotation angles and translation values, and so on. Accordingly, to specify a translation and rotation for a given projection image, i.e., a corresponding step of the object motion trajectory, three rotation parameters and three translation parameters are generally required.The object motion trajectory therefore includes these three rotation parameters and translation parameters for each projection image directly or indirectly.

[0025] The latent vector can, for example, be understood as a tuple with N entries, where the entries can correspond to real or complex numbers. The individual entries of the latent vector generally have no descriptive meaning. The trained algorithm is trained to calculate exactly one object motion trajectory from any latent vector in the latent space, i.e., a sequence of rotation and / or translation operations for each of the projection images, or a sequence of the corresponding rotation parameters and translation parameters for the individual projection images.

[0026] The size of N can be adjusted to the required accuracy and the available computational resources. Typically, latent spaces with 50 to 150 dimensions, and correspondingly latent vectors with 50 to 150 entries, can be used. This ensures that sufficient modification of the latent vectors is possible to encode complex objects such as the object's motion trajectory. The latent space can, for example, ℝ N or ℂ N be.

[0027] It is known to unambiguously encode complex inputs, such as text, images, and so on, into a latent space using algorithms trained through machine learning. For this purpose, it is particularly well-known to use artificial neural networks, such as so-called autoencoder networks or variational autoencoder networks. Unless otherwise stated, the term autoencoder can be understood in the following to include both autoencoders in the narrower sense and variational autoencoders. Instead of autoencoder networks, generative adversarial networks (GANs) can also be used.

[0028] Such algorithms can be trained in a self-supervised manner so that they can generate exactly one latent vector from a complex input and reconstruct the input from the same latent vector. Once the algorithm is trained, the reconstructing part of the algorithm, also called a generator module or decoder module depending on the design, can be isolated and used to generate the reconstructed input based on arbitrary latent vectors.

[0029] This concept is used according to the invention to generate object motion trajectories from latent vectors. The trained algorithm was trained, in particular, based exclusively on physically possible training object motion trajectories. After training, it can assign exactly one latent vector to each such training object motion trajectory and reliably and accurately reconstruct the training object motion trajectory from this latent vector. To ensure that only physically possible training object motion trajectories are used as a basis and, accordingly, that the reconstruction of any latent vector is also physically possible, the training object motion trajectories can be obtained, for example, from monitoring the actual movement of patients.

[0030] For the inventive method for generating motion-compensated image reconstruction, only the reconstructive component is required. The latent vectors, which are used as input to the trained algorithm during optimization as current latent vectors, are arbitrary in the given latent space. Accordingly, arbitrary, yet always physically possible, current object motion trajectories result.

[0031] By reconstructing and evaluating the cost function, the latent vector that minimizes the cost function is identified as the optimal latent vector. As described above, the cost function is correlated with the actual object motion, so that low values ​​of the cost function correspond to a low degree of motion. Accordingly, the object motion trajectory that corresponds to the most realistic—and therefore minimal—movement of the object during the X-ray-based imaging procedure is identified as the optimal object motion trajectory.

[0032] For clarification, it should be noted that using the latent vector as an optimization parameter can be understood as selecting different current latent vectors from the latent space for different optimization steps to perform the corresponding optimization step. Various well-known optimization methods can also be used as optimization methods.

[0033] In optimization problems, we generally speak of objective functions, regardless of whether they are to be minimized or maximized. If the objective function is to be minimized, it is a cost function. It should be noted that in this case, at least, minimization does not represent any restriction, since even in the case ei For an objective function to be maximized, a modified objective function can always be defined which must be minimized.

[0034] The inventive method for generating a motion-compensated image reconstruction exploits the realization that the motion compensation problem is a non-convex problem, in which local minima can become problematic in the optimization if they correspond to physically unreasonable motion configurations. Parameterization based on latent vectors prevents the optimization from becoming stuck in local minima, resulting instead in an optimal object motion trajectory with only physically possible or realistic motion configurations. This can significantly increase the reliability of the image reconstruction.

[0035] According to at least one embodiment of the method, the trained algorithm is designed to assign a time series of rotations and translations in three-dimensional space to a latent vector of the latent space, in particular to any arbitrary latent vector of the latent space, as an object movement trajectory.

[0036] Each rotation can be defined by three rotation parameters, in particular three rotation angles, and each translation by three translation values, i.e., displacements in three different spatial directions. The time series of rotations and translations can be a six-dimensional time series for the three rotation and three translation parameters, or it can also contain six individual time series for the individual degrees of freedom, or rotation and translation parameters. The time series of rotations and translations can also be a time series of rotation matrices and translation vectors.

[0037] In particular, the plurality of projection images may include K projection images, each projection image corresponding to one of K consecutive recording periods and the time series including K rotations and K translations.

[0038] K is an integer greater than 1 and can, for example, be in the range 100 to 1,000. Especially for CBCT, the number of projection images can be in such a range, for example, in the range 300 to 600.

[0039] During each recording period, one of the K projection images is generated. Each projection image is thus assigned a rotation and a translation in three dimensions. This assignment can be used, as described above, to correct the geometric image reconstruction operation for the corresponding projection image in order to implement motion compensation.

[0040] According to at least one embodiment, to generate the trained algorithm, a plurality of training object motion trajectories are generated, each describing an actual movement of a test object. For each of the training object motion trajectories, a training vector is generated in the latent space, in particular by applying a coding neural network to the respective training object motion trajectory. A reconstructing neural network is applied to the training vector to generate a reconstructed object motion trajectory.

[0041] At least one predefined loss function is evaluated based on the reconstructed object motion trajectory, optionally using one or more additional neural networks. Network parameters, in particular weighting factors and / or bias factors, of the reconstructing neural network, in particular of the reconstructing neural network and the coding neural network, and optionally of the one or more additional neural networks, are adjusted depending on a result of the evaluation of the at least one loss function.

[0042] For example, the encoding neural network can be an encoder module of an autoencoder, and the reconstructing neural network can be a decoder module of the autoencoder. In the case of a GAN, the reconstructing neural network can be a generator module of the GAN, and the encoding neural network can be a neural network provided in addition to the GAN. The at least one additional neural network can then be a discriminator module of the GAN.

[0043] According to at least one embodiment, the trained algorithm is designed as a generator module of a GAN network.

[0044] GAN networks have emerged as reliable tools for encoding complex input data in latent spaces and enabling reliable reconstructions of the input data.

[0045] A GAN network initially contains the generator module and a corresponding discriminator module. However, the discriminator module is no longer needed after training is complete.

[0046] According to at least one embodiment in which the trained algorithm is configured as a generator module of the GAN network, to generate the trained algorithm, i.e., in particular, to train the algorithm, a plurality of training object motion trajectories are generated, each describing an actual movement of a test object. For each of the training object motion trajectories, a training vector is generated in the latent space by applying an artificial neural network configured as an encoder to the respective training object motion trajectory. The generator module and the discriminator module of the GAN network are trained using the training vectors as input data.

[0047] The training of the GAN network, in particular the generator module and the discriminator module, can be performed in a known manner. The generator module generates a reconstructed object motion trajectory based on the respective training vector, and the discriminator module decides whether the reconstructed object motion trajectory is a reconstructed object motion trajectory, also referred to as a "fake sample," or a genuine training object motion trajectory, also referred to as a "real sample." Training is continued until the discriminator module can no longer distinguish genuine training object motion trajectories from reconstructed object motion trajectories. The generator module can then be used to reliably generate object motion trajectories based on any latent vector in the latent space.

[0048] In particular, a known generator loss function and a known discriminator loss function can be used to train the generator module and the discriminator module accordingly. The encoder module can be pre-trained, i.e., a known coding encoder module, or it can be trained before training the GAN network or together with the GAN network.

[0049] The encoder module, which generates the respective training vector, is not strictly part of the GAN network. Rather, GAN networks can be trained on any input, especially noise vectors. In this case, however, the trajectories are specifically generated based on actual training object motion trajectories that represent physically realistic motion constellations, so that, conversely, the object motion trajectories generated by the generator module also correspond to physically realistic motion constellations. The encoder module, like the discriminator module, is no longer required after training is complete.

[0050] According to at least one embodiment, the trained algorithm is designed as a decoder module of an autoencoder.

[0051] Autoencoders essentially consist of an encoder module, which can convert a corresponding input—in this case, a training object's motion trajectory—into a latent vector, and a decoder module, which can reconstruct the input based on the latent vector. The encoder module is no longer required after training is complete.

[0052] Autoencoders have proven to be reliable tools for encoding complex input data in latent spaces and enabling reliable reconstructions of the input data. Autoencoders also have the advantage of being less problematic for regularization during training compared to other networks.

[0053] Also for generating or training the autoencoder, a plurality of training object motion trajectories can be generated, each describing an actual movement of a test object. For each of the training object motion trajectories, a training vector is generated in the latent space by applying the encoder module of the autoencoder or variational autoencoder to the respective training object motion trajectory. Using the decoder module, a reconstructed object motion trajectory is generated based on the training vector, and a loss function is calculated based on a deviation of the reconstructed Ob The object motion trajectory is evaluated from the training object motion trajectory. Network parameters of the autoencoder, in particular the encoder module and the decoder module, are adjusted depending on the result of the loss function evaluation.

[0054] Known methods and known loss functions can be used to adapt the network parameters.

[0055] In this case, the encoder module is specifically trained based on actual training object motion trajectories, which represent physically realistic motion constellations, in order to generate the training vectors, so that conversely the object motion trajectories generated by the decoder module also correspond to physically realistic motion constellations.

[0056] According to at least one embodiment, for each of the training object movement trajectories to generate the respective training object movement trajectory, the test object is monitored by means of at least one camera during a corresponding test period, and the respective training object movement trajectory is determined based on camera images generated during the monitoring by means of the at least one camera.

[0057] In particular, a position and / or orientation of the test object in a reference coordinate system can be determined based on the camera images and a corresponding sequence of rotations and translations or corresponding rotation and translation parameters can be calculated.

[0058] According to at least one embodiment, at least one visual marker is attached to the test object and the respective training object movement trajectory is determined based on the corresponding positions of the at least one visual marker in the camera images.

[0059] In particular, the position and / or orientation in the reference coordinate system can be determined from the positions in the camera images, since the positions and orientations of the at least one camera in the reference coordinate system are known.

[0060] In this way, a reliable recording of only physically realistic movement constellations can be achieved and accordingly the reliability of the trained algorithm and accordingly of the entire method according to the invention can be increased.

[0061] The object could, for example, be a patient's head. With X-ray-based imaging of the head, the described motion compensation can be performed with particularly high accuracy, as the assumption of rigid movements, especially without deformation of the object, is particularly well met.

[0062] According to a further aspect of the invention, a method for X-ray-based imaging is also provided. A plurality of projection images of an object are generated, and a method for generating a motion-compensated image reconstruction according to the invention is performed based on the plurality of projection images.

[0063] In particular, the X-ray-based imaging technique is a cone beam computed tomography technique.

[0064] For use cases or application situations that may arise during the method and which are not explicitly described here, it may be provided that, in accordance with the method, an error message and / or a request to enter user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0065] According to a further aspect of the invention, a data processing device is provided that has at least one computing unit. The at least one computing unit is configured or adapted to carry out a method according to the invention for generating a motion-compensated image reconstruction.

[0066] According to a further aspect of the invention, a device for X-ray-based imaging is provided. The device comprises an X-ray imaging modality configured to generate a plurality of projection images of an object. The device comprises a storage device that stores a trained algorithm that is trained to assign an object motion trajectory to a latent vector of a predetermined latent space. The device comprises at least one computing unit configured to determine an optimal object motion trajectory by applying the trained algorithm to an optimal latent vector and to generate the motion-compensated image reconstruction depending on the plurality of projection images under the assumption that the object moved according to the optimal object motion trajectory during the implementation of the imaging method.

[0067] The at least one computing unit is configured to minimize a predetermined cost function using the latent vector as an optimization parameter in order to determine the optimal latent vector and, in doing so, to determine a current object motion trajectory for each optimization step with a current latent vector by applying the trained algorithm to the current latent vector, and to calculate a current value of the cost function depending on the plurality of projection images under the assumption that the object has moved according to the current object motion trajectory during the implementation of the imaging method.

[0068] A computing unit can be understood, in particular, as a data processing device that contains a processing circuit. The computing unit can therefore, in particular, process data to perform computing operations. This may also include operations for performing indexed access to a data structure, for example, a look-up table (LUT).

[0069] The computing unit may, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more single-chip systems (SoCs). The computing unit may also contain one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual network of computers or other of the aforementioned units.

[0070] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more memory units.

[0071] A memory unit can be a volatile data memory, for example a dynamic random access memory (DRAM) or a static random access memory (SRAM), or a non-volatile data memory, for example a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash memory or flash EEPROM, a ferroelectric random access memory (FRAM), a magnetoresistive random access memory,MRAM (magnetoresistive random access memory) or phase-change random access memory (PCRAM).

[0072] Further embodiments of the device according to the invention follow directly from the various embodiments of the method according to the invention, and vice versa. In particular, individual features and corresponding explanations, as well as advantages relating to the various embodiments of the method according to the invention, can be transferred analogously to corresponding embodiments of the device according to the invention. In particular, the device according to the invention is designed or programmed to carry out a method according to the invention. In particular, the device according to the invention carries out a method according to the invention.

[0073] According to a further aspect of the invention, a first computer program with first instructions is provided. Upon execution of the first instructions by a data processing device, in particular a data processing device according to the invention, the first instructions cause the data processing device to perform a method according to the invention for generating a motion-compensated image reconstruction.

[0074] According to a further aspect of the invention, a second computer program with second instructions is provided. Upon execution of the second instructions by an inventive device for X-ray-based imaging, in particular by the at least one computing unit of the device, the second instructions cause the device to perform an inventive method for generating a motion-compensated image reconstruction and / or an inventive method for X-ray-based imaging.

[0075] According to a further aspect of the invention, a computer-readable storage medium is provided which stores a first computer program and / or a second computer program according to the invention.

[0076] The first computer program, the second computer program and the computer-readable storage medium can be understood as respective computer program products comprising the first and / or the second instructions.

[0077] Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown in the figures can be encompassed by the invention not only in the respectively specified combination, but also in other combinations. In particular, the invention can also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention can encompass embodiments and combinations of features that go beyond the combinations of features set out in the backreferences to the claims or deviate from them.

[0078] The invention is explained in more detail below using specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be provided with the same reference numerals. The description of identical or functionally equivalent elements may not necessarily be repeated for different figures.

[0079] The figures show: FIG 1 shows a schematic representation of an exemplary embodiment of an apparatus according to the invention for X-ray-based imaging, FIG 2 shows a schematic representation of various parameter spaces for parameterizing object movement trajectories, FIG 3 shows a schematic representation of a training phase for training an algorithm for carrying out an exemplary embodiment of a method according to the invention for generating a motion-compensated image reconstruction, and FIG 4 shows a schematic flow diagram of a further exemplary embodiment of a method according to the invention for generating a motion-compensated image reconstruction.

[0080] In FIG 1 1 schematically shows a device 1 according to the invention for X-ray-based imaging comprising an X-ray imaging modality 2. The X-ray imaging modality 2 is shown as an example as a device for cone beam computed tomography, CBCT, and has, for example, a C-arm 3 with an X-ray source 4 and an X-ray detector 5 arranged opposite one another. An object 6, for example the head of a patient, is arranged between the X-ray source 4 and the X-ray detector 5. Furthermore, the device 1 has a computing unit 7. The device 1, in particular the computing unit 7, has a storage device (not shown) which stores a trained algorithm 12 (see FIG 3, FIG 4 ) that is trained to map a latent vector of a given latent space (see FIG 3, FIG 4 ) to assign an object motion trajectory.

[0081] The device 1 is particularly configured to perform a method according to the invention for X-ray-based imaging. For this purpose, the X-ray imaging modality 2 generates a plurality of projection images of the object 6, in particular according to a known CBCT protocol. For example, approximately 500 projection images of the object 6 can be generated. Furthermore, a method according to the invention for generating a motion-compensated image reconstruction 14 is performed. A schematic flow diagram of such a method is shown in FIG 4 shown.

[0082] In particular, an optimal object motion trajectory 10' for the object 6 is generated by applying the trained algorithm 12 to an optimal latent vector in the latent space 13, and the motion-compensated image reconstruction 14 is generated depending on the plurality of projection images under the assumption that the object 6 has moved according to the optimal object motion trajectory 10' during the execution of the imaging method, i.e., during the generation of the projection images.

[0083] The optimal latent vector is determined by minimizing a given cost function using the latent vector as an optimization parameter. For each optimization step with a current latent vector, a current object motion trajectory is determined by applying the trained algorithm 12 to the current latent vector. A current image reconstruction is generated depending on the plurality of projection images. This is done under the assumption that the object 6 has moved according to the current object motion trajectory during the imaging process. A current value of the cost function is calculated depending on the current image reconstruction.

[0084] Alternatively, the current value of the cost function can be calculated depending on a given set of consistency conditions concerning the plurality of projection images, so that no current image reconstruction needs to be generated during the individual optimization steps.

[0085] The trained algorithm 12 may, for example, be a decoder module 12 of an autoencoder 9, as shown schematically in FIG 3 During the training phase, the autoencoder 9 comprises an encoder module 11 that can encode a training object motion trajectory 10 into a corresponding latent vector of the latent space 13. The decoder module 12 reconstructs the training object motion trajectory 10 based on the respective latent vector.

[0086] The model of rigid motion without any constraints is based on rotations and translations, typically defined as the special Euclidean group in three dimensions SE(3). This motion model is very powerful, but most motion constellations that can be described by SE(3) are not realistic. The motions that can in principle be described by rotations and translations are in FIG 2 schematically shown as square 8. In particular, FIG 2 a manifold of possible movements is schematically represented as square 8.

[0087] Sampling the motion via spline nodes already reduces the number of possible unrealistic motion constellations. The movements that can in principle be described by splines are FIG 2 schematically represented as circle 8a, specifically as a submanifold of the manifold of all possible movements. Nevertheless, many unrealistic movement constellations are still possible. For patients lying on a table and constrained by human anatomy, only a subset of SE(3) is physically possible. This portion is in Fig. 2 schematically irregular area 8b is shown and represents the subset of SE(3) that contains only physically or anatomically possible motion constellations, in particular as a smaller submanifold of the motions that can be described by splines.

[0088] In various embodiments of the invention, in a first step, movement trajectories of real patients are sampled as test object movement trajectories 10. This can be done, for example, by tracking. For example, a patient wears markers on their head, and a camera system tracks these markers. Based on the markers and the camera system, a movement trajectory of the patient can be calculated. By recording, for example, several patients over several hours, a very large database of realistic test object movement trajectories 10 is created.

[0089] Based on a set of realistic test object motion trajectories 10, the autoencoder 9 can be used to encode and decode these test object motion trajectories 10. The autoencoder 9 can be configured as a regular autoencoder or as a variational autoencoder 9. An autoencoder 9 is particularly defined by reproducing the network's input in the output layer. By introducing a bottleneck in the middle of the network, i.e., after the encoder module 11, the autoencoder 9 is forced to learn the latent space 13 of the test object motion trajectories 10 instead of performing a simple identity mapping.

[0090] The encoder module 11 can be an implicit neural network, for example, according to a SIREN architecture, as described in the publication V. Sitzmann, J. Martel, A. Bergman, A. Lindell, G. Wetzstein: "Implicit neural representations with periodic activation functions", Advances in Neural Information Processing Systems. 2020;33:7462-73. One advantage of the SIREN architecture is the ability to compute higher-order derivatives. This is possible because the classic Re-Lu activation function is replaced by a sine function. As a result, Jacobian regularization or bending regularization are possible, which enforces a smooth and realistic mapping from the latent space to the output space, so that, for example, small changes in the latent space 13 lead to small changes in the output space. Note that this type of behavior is also enforced by the variational autoencoder 9.This property can be important when latent space 13 is used as a parameterization for motion compensation.

[0091] After successful training, the decoder module 12 of the autoencoder 9 can be extracted and used in generating the object motion trajectories.

[0092] A GAN can be used as an alternative to the autoencoder 9. In this case, the input to the GAN would again be a latent vector, the generator module of the GAN would map a latent vector to an object motion trajectory, and the loss would be based on distinguishing between true and so-called false object motion trajectories.

[0093] After training, the decoder module 12 can generate a current object motion trajectory based on a current latent vector. This object motion trajectory can then be translated into projection matrices and used in the reconstruction algorithm to create a current image reconstruction based on the current object motion trajectory. However, compared to parameterization with splines, only physically meaningful motion constellations result.

[0094] The latent space 13 now represents the parameterization for the object motion trajectory. That is, the parameterization X is the latent vector. The motion compensation can then be written in the regular notation by X ′ = argmin X c r I X , be defined or by X ′ = argmin X c I X , if the cost function c() is based on consistency constraints, in particular epipolar consistency constraints. Solving this minimization problem yields the optimal latent vector X', which solves the motion compensation.

Claims

1. Method for creation of a movement-compensated image reconstruction (14) for an x-ray based imaging method, wherein i) a plurality of projection images of an object (6), created when the imaging method is being carried out, is provided; ii) a trained algorithm is provided, which is trained to allocate an object movement trajectory to a latent vector of a predetermined latent space (13); iii) an optimal object movement trajectory (10') is determined by application of the trained algorithm (12) to an optimal latent vector and the movement-compensated image reconstruction (14) is created depending on the plurality of projection images, with the assumption that the object (6) would have moved in accordance with the optimal object movement trajectory (10') while the imaging method was being carried out; and iv) the optimal latent vector is determined by a predetermined cost function being minimised using the latent vector as the optimisation parameter, wherein, for each optimisation step with a current latent vector - a current object movement trajectory is determined by application of the trained algorithm (12) to the current latent vector; and - a current value of the cost function is computed depending on the plurality of projection images, with the assumption that the object (6) would have moved in accordance with the current object movement trajectory while the imaging method was being carried out.

2. Method according to claim 1, wherein the trained algorithm (12) is designed to allocate a time series of rotations and translations in three-dimensional space to a latent vector of the latent space (13) as an object movement trajectory.

3. Method according to claim 2, wherein - the plurality of projection images contains K projection images, wherein each projection image corresponds to one of K consecutive recording time spaces; and - the time series contains K rotations and K translations.

4. Method according to one of the preceding claims, wherein the trained algorithm (12) is embodied as a decoder module (12) of an autoencoder (9) or of a variational autoencoder (9).

5. Method according to claim 4 wherein, for creation of the trained algorithm (12), a plurality of training object movement trajectories (10) is created, which each describe an actual movement of a test object, and for each of the training object movement trajectories (10) - a training vector in the latent space (13) is created, by an encoder module (11) of the autoencoder (9) or the variational autoencoder (9) being applied to the respective training object movement trajectory (10); - a reconstructed object movement trajectory is created by means of the decoder module (12) based on the training vector; - a loss function based on a deviation of the reconstructed object movement trajectory from the training object movement trajectory (10) is evaluated; and - network parameters of the autoencoder (9) or variational autoencoder (9) are adapted depending on a result of the evaluation of the loss function.

6. Method according to one of claims 1 to 4, wherein the trained algorithm (12) is embodied as a generator module of a GAN network.

7. Method according to claim 6, wherein - for creation of the trained algorithm (12), a plurality of training object movement trajectories (10) is created, which each describe an actual movement of a test object; - for each of the training object movement trajectories (10) a training vector in the latent space (13) is created, by an artificial neural network embodied as an encoder being applied to the respective training object movement trajectory (10); - the generator module and a discriminator module of the GAN network are trained using the training vectors as input data.

8. Method according to one of claims 5 or 7, wherein for each of the training object movement trajectories (10), for creation of the respective training object movement trajectory (10), the test object is monitored during a corresponding test period by means of at least one camera and the respective training object movement trajectory (10) is determined based on camera images created by means of the at least one camera during the monitoring.

9. Method according to claim 8, wherein at least one visual marker is attached to the test object (6) and the respective training object movement trajectory (10) is determined based on corresponding positions of the at least one visual marker in the camera images.

10. Method according to one of the preceding claims, wherein, for each of the optimisation steps - a current image reconstruction is created depending on the plurality of projection images, with the assumption that the object (6) would have moved in accordance with the current object movement trajectory while the imaging method was being carried out, and the current value of the cost function is computed depending on the current image reconstruction; or - the current value of the cost function is computed depending on a predetermined set of consistency conditions relating to the plurality of projection images.

11. Method for x-ray based imaging, wherein a plurality of projection images of an object (6) is created and a method for creation of a movement-compensated image reconstruction (14) in accordance with one of the preceding claims is carried out based on the plurality of projection images.

12. Method according to claim 11, wherein the method is a cone-beam computed tomography method.

13. Data processing facility (7) with at least one processing unit (7), which is adapted to carry out a method for creation of a movement-compensated image reconstruction (14) according to one of the preceding claims 1 to 10.

14. Apparatus (1) for x-ray based imaging having an x-ray imaging modality (2), which is configured to create a plurality of projection images of an object (6), a memory device, which stores a trained algorithm (12), which is trained to allocate an object movement trajectory to a latent vector of a predetermined latent space (13), as well as at least one processing unit (7), which is configured, - to determine an optimal object movement trajectory (10') by application of the trained algorithm (12) to an optimal latent vector and to create the movement-compensated image reconstruction (14) depending on the plurality of projection images, with the assumption that the object (6) would have moved in accordance with the optimal object movement trajectory (10') while the imaging method was being carried out; and - to minimise a predetermined cost function using the latent vector as an optimisation parameter, in order to determine the optimal latent vector and, in doing so, to determine a current object movement trajectory for each optimisation step with a current latent vector by application of the trained algorithm (12) to the current latent vector, and to compute a current value of the cost function depending on the plurality of projection images, with the assumption that the object (6) would have moved in accordance with the current object movement trajectory while the imaging method was being carried out.

15. Computer program product having - first instructions that, when executed by a data processing facility (7), cause said data processing facility (7) to carry out a method according to one of claims 1 to 10; and / or - second instructions that, when executed by an apparatus (1) according to claim 14, cause said apparatus (1) to carry out a method according to one of claims 1 to 12.