Motion compensated image reconstruction in x-ray based imaging

Abstract

For generating a motion compensated image reconstruction (14) in an X-ray based imaging process, a best object motion trajectory (10') is determined by applying the trained algorithm (12) to a best latent vector, and the motion compensated image reconstruction (14) is generated from the plurality of projection images under the assumption that the object (6) has moved according to the best object motion trajectory (10') during the imaging process execution. The best latent vector is determined by minimizing a pre-set cost function using the latent vector as an optimization parameter, wherein for each optimization step using a current latent vector, a current object motion trajectory is determined by applying the trained algorithm (12) to the current latent vector, and a current value of the cost function is calculated from the plurality of projection images under the assumption that the object (6) has moved according to the current object motion trajectory during the imaging process execution.

Description

TECHNICAL FIELD

[0001] The present invention relates to a method for generating a motion compensated image reconstruction in an X-ray based imaging procedure, wherein a plurality of projection images of an object generated during execution of the imaging procedure is provided. The present invention further relates to a method for X-ray imaging, wherein a plurality of projection images of an object is generated and the method for generating a motion compensated image reconstruction is implemented. The present invention further relates to a data processing device, a device for X-ray imaging having an X-ray imaging modality and a computer program product. BACKGROUND

[0002] With the aid of X-ray based imaging procedures, in particular cone-beam computed tomography, i.e. CBCT (English: "cone-beam computed tomography"), brain structures can be assessed primarily during a surgical procedure. This for example facilitates the differential diagnosis of a stroke in order to classify it as an ischemic or hemorrhagic stroke. But one major obstacle for high quality image assessment is the movement of the patient, which distorts the geometric information used for the reconstruction. This leads to streak or blurring artifacts.

[0003] Motion can be corrected retrospectively using motion compensation techniques. For this the rigid motion trajectory of the patient is estimated. In case the rigid motion trajectory is successfully estimated a motion compensated image reconstruction can be created.

[0004] Here the motion is compensated, in particular by optimizing an objective function. The input of the objective function contains a set of projection images and a set of parameters defining an approximate motion trajectory of the patient. If the projection images are denoted by I and the parameters defining the motion trajectory by X, the motion compensation problem can be solved by determination of the optimal parameters X' as follows

[0005] X' = argmin X c(r(I, X)),

[0006] where r() denotes an image reconstruction based on the projection images and the current estimate of the motion trajectory and c() denotes a cost function. The cost function for example is associated with the motion, so that a low value corresponds to a small motion.

[0007] The cost function can be based on 2D / 3D image registration, for example, as described in the article by J. Wang, R. Schaffert, A. Borsdorf, B. Heigl, X. Huang, J. Hornegger, A. Maier, Dynamic 2-D / 3-D rigid registration framework using point-to-plane correspondence model, IEEE Transactions on Medical Imaging, Vol. 36, No. 9, pp. 1939-1954, 2017.

[0008] The cost function can also be based on consistency conditions, as described in the articles by A. Preuhs, M. Manhart, A. Maier, Fast Epipolar Consistency without the Need for Pseudo Matrix Inverses, Proc. 15th International Conference on Image Formation in X-Ray Computed Tomography, pp. 202-205, 2018, and A. Preuhs, M. Manhart, E. Hoppe, M. Kowarschik, A. Maier, Motion gradients for epipolar consistency, Proc. 15th International Meeting on Fully Three-Dimensional Image Reconstruction in Radiology and Nuclear Medicine, p. 110720D, 2019. In this case, no image reconstruction r() is needed in the above formula.

[0009] Furthermore, the cost function can be based on an autofocus method as described in the article "Appearance Learning for Image-based Motion Estimation in Tomography" by A. Preuhs, M. Manhart, P. Roser, E. Hoppe, H. Yixing, M. Psychogios, M. Kowarschik, A. Maier, IEEE Transactions on Medical Imaging, pp. 1-10, 2020.

[0010] One possible parameterization of X is based on rotations and translations. In this case, three parameters for rotations and three parameters for translations are used for each projection image. If the number of projection images is N, the number of required parameters is N*6. For a regular CBCT trajectory comprising for example 400-600 projection images, the parameter vector X will have 2400-3600 entries. This makes the computational load of the optimization problem very large.

[0011] To alleviate this problem, a spline can be used for the motion compensation as described in the aforementioned article by Preuhs et al. published in 2020. In this case, the motion is parameterized by the spline node parameters. This greatly reduces the complexity of the motion compensation problem. For example, a single motion direction can be approximated by 20 spline nodes, resulting in a parameter vector X with only 120 entries. Another advantage of splines is that constraints can be incorporated. The motion of a patient is for example usually uniform, which is already taken into account by the spline itself.

[0012] A problem of the two parameterizations is that they can produce many physically unreasonable motion trajectories. Since the cost function is usually not convex, these physically unreasonable motion trajectories cause the optimization to reach a local minimum and hinder successful convergence. This partly hinders reliable motion compensation.

[0013] Invention problem

[0014] The technical problem addressed by the present invention is to improve the reliability of motion compensation in image reconstruction in X-ray imaging processes.

[0015] The technical problem is solved by the technical solution according to the aspects of the present invention. Advantageous developments and preferred embodiments result from the description. Irrespective of the grammatical gender of the specific terms, people of male, female or other gender identity are included.

[0016] The present application is based on the idea that the parameterization of the object motion trajectory is chosen such that unrealistic or physically impossible motion configurations are not considered in the optimization. To this end, the object motion trajectory is parameterized by a respective latent vector in an abstract latent space and a concrete object motion trajectory is generated by applying a correspondingly trained algorithm.

[0017] According to one aspect of the present application, a method for generating a motion compensated image reconstruction in an X-ray based imaging process is presented. Herein, the following steps i), ii), iii) and iv) are carried out, but not necessarily in this order. According to step i), a plurality of projection images of an object generated during the execution of the imaging process is provided. According to step ii), a trained algorithm is provided, which is stored, inter alia, on a storage device and trained for assigning an object motion trajectory to a latent vector of a pre-set latent space. According to step iii), an optimal object motion trajectory is determined by applying the trained algorithm to an optimal latent vector, in particular an optimal latent vector of the latent space, and a motion compensated image reconstruction (or in other words: a motion compensated image reconstruction) is generated from the plurality of projection images under the assumption that the object has moved according to the optimal object motion trajectory during the execution of the imaging process.

[0018] According to step iv), the optimal latent vector is determined in such a way that a pre-set cost function is minimized using the latent vector as an optimization parameter. Herein, for each optimization step using a current latent vector, a current object motion trajectory is determined by applying the trained algorithm to the current latent vector. A current value of the cost function is calculated from the plurality of projection images under the assumption that the object has moved according to the current object motion trajectory during the execution of the imaging process.

[0019] Depending on the cost function used, here a current value of the cost function can be calculated for each optimization step, for example, in accordance with a pre-set consistency condition related to the plurality of projection images, in particular epipolar en Konsistenzbedingungen.

[0020] This is the case, for example, when using the cost function described in the articles by Preuhs et al. published in 2018 and 2019 mentioned at the beginning of this text.

[0021] Alternatively, for each optimization step, for example, a current image reconstruction can be generated from the plurality of projection images under the assumption that the object has moved according to the current object motion trajectory during the execution of the imaging process. Subsequently, a current value of the cost function is calculated from the current image reconstruction.

[0022] This is the case, for example, when using the cost function described in the article by Wang et al. published in 2020 mentioned at the beginning of this text or in the article by Preuhs et al. published in 2020 mentioned at the beginning of this text.

[0023] In some embodiments, the method for generating a motion compensated image reconstruction according to the present application can be a purely computer-implemented method, wherein all steps of the computer-implemented method are performed by a data processing device having at least one computing unit. In particular, the at least one computing unit is configured or adapted to carry out the steps of the computer-implemented method. To this end, the at least one computing unit or a storage device of the at least one computing unit may, for example, store a computer program which contains instructions which, when executed by the at least one computing unit, cause the at least one computing unit to carry out the computer-implemented method.

[0024] In other embodiments, for example when the steps of performing an imaging procedure, i.e. in particular generating projection images, are part of the method, the method is not purely computer-implemented.

[0025] The X-ray-based imaging procedure may, for example, be a computed tomography procedure, in particular a cone beam computed tomography procedure. The projection images in particular correspond to X-ray radiation which is detected by an X-ray detector, which X-ray radiation has penetrated an object and is thus attenuated. Generating an image reconstruction on the basis of the projection images in particular comprises determining the respective attenuation values of the object in the respective spatial region.

[0026] Various known methods can be used for image reconstruction. In particular, each projection image can be assigned a geometric operation by means of which an image reconstruction or a part of an image reconstruction is generated from the respective projection image. It is likewise known that, in such image reconstruction, motion compensation is performed in order to compensate for motion of the object, in particular rigid motion of the object, during recording of the projection images. To this end, the geometric operation assigned to (or configured for) the individual projection images is corrected in accordance with the respectively estimated or assumed object motion trajectory or the respective object translation and rotation position when generating the respective projection image.

[0027] The correction can be implemented, for example, by corresponding rotations and / or translations which exist at the time points of the respective generated projection images in accordance with the object motion trajectory. In other words, the object motion trajectory can be defined by a number of successive rotation operations and / or translation operations. The format in which the object motion trajectory is given or stored can be different here. It is possible to give corresponding translation vectors and rotation matrices, but also corresponding rotation angles and translation values, etc. Thus, generally three rotation parameters and three translation parameters are required in order to determine the translation and rotation for a given projection image, i.e. the respective step of the object motion trajectory. The object motion trajectory thus contains in direct or indirect form these three rotation parameters and translation parameters for each projection image.

[0028] The latent vector can be understood, for example, as a tuple with N entries, wherein the entries can correspond to real or complex numbers. The individual entries of the latent vector generally have no explanatory meaning. The trained algorithm is trained to calculate exactly one object motion trajectory from each arbitrary latent vector of the latent space, i.e. a sequence of rotation and / or translation operations for each projection image or a sequence of corresponding rotation parameters and translation parameters for the individual projection images.

[0029] The size of N can be adapted depending on the required accuracy and available computing resources. It is generally possible to use a latent space with 50 to 150 dimensions, i.e. thus a latent vector with 50 to 150 entries. In this way it is possible to achieve a sufficient modification of the latent vector in order to be able to encode complex objects, such as object motion trajectories. The latent space can be, for example, or is

[0030] It is known that complex inputs such as texts, images, etc. are explicitly or uniquely encoded in a latent space using an algorithm trained by machine learning. For this purpose, it is known in particular to use artificial neural networks, for example so-called autoencoder networks or variational autoencoder networks. Unless stated otherwise, the term autoencoder can be understood below to include both the narrow sense of autoencoder and variational autoencoder. Instead of an autoencoder network, it is also possible to use a generative adversarial network, i.e. a GAN (English: "Generative Adversarial Networks") for example.

[0031] Such an algorithm can be trained in a self-supervised manner, so that the algorithm can generate exactly one latent vector from a complex input and can reconstruct the input from the same latent vector. Once the algorithm has been trained, the reconstruction part of the algorithm, also referred to as generator module or decoder module depending on the design, can be separated and used to generate a reconstructed input on the basis of an arbitrary latent vector.

[0032] According to the application, the approach is designed to generate an object motion trajectory from a latent vector. Thus, the trained algorithm has been trained, inter alia, on the basis of only physically possible training object motion trajectories, so that the algorithm, after training, is able to assign exactly one latent vector to each such training object motion trajectory and to reliably and accurately reconstruct the training object motion trajectory from the latent vector. In order to ensure that only physically possible training object motion trajectories are used as a basis and, correspondingly, that the reconstruction of an arbitrary latent vector is also physically possible, the training object motion trajectories can be obtained, for example, from the monitoring of actual movements of a patient.

[0033] The method for generating a motion-compensated image reconstruction according to the application only requires the partial reconstruction. The latent vectors that are used as input during the optimization as the current latent vectors that are the basis of the trained algorithm are arbitrary in the pre-defined latent space. Thus, an arbitrary, but always physically possible, current object motion trajectory is also generated.

[0034] The latent vector that minimizes the cost function is identified as the best latent vector by reconstructing and analyzing the cost function. Here, as mentioned at the beginning, the cost function is associated with the actual object motion, so that a low value of the cost function corresponds to a small amount of motion. Correspondingly, the object motion trajectory that corresponds to the object motion that is as small as possible and as realistic as possible during the execution of the x-ray-based imaging process is identified as the best object motion trajectory.

[0035] For the sake of clarity only, it should be mentioned that the use of the latent vectors as optimization parameters can be understood in such a way that, for different optimization steps, different current latent vectors are selected from the latent space in order to carry out the respective optimization step. Various known optimization methods can also be used as optimization methods.

[0036] In optimization problems, the function to be minimized or maximized is generally referred to as the objective function, regardless of whether it is to be minimized or maximized. If the objective function is to be minimized, it is the cost function. It should be noted here that minimizing does not represent any restriction at least in the present case, since even in the case of an objective function to be maximized, a modified objective function can always be defined which is suitable for being minimized.

[0037] In the method for generating a motion-compensated image reconstruction according to the application, use is made of the insight that the motion compensation problem is a non-convex problem, since a local minimum can become problematic in the optimization when it does not correspond to a physically plausible motion configuration. By means of the parameterization according to the latent vector, it is possible to prevent the optimization from becoming stuck in a local minimum to some extent and, conversely, to generate a best object motion trajectory that only has physically possible or realistic motion configurations. As a result, the reliability of the image reconstruction can be greatly improved.

[0038] According to at least one embodiment of the method, the trained algorithm is designed to assign a time series of rotations and translations in three-dimensional space as an object motion trajectory to a latent vector of the latent space, in particular to each arbitrary latent vector of the latent space.

[0039] Each rotation can in particular be defined by three rotation parameters, in particular three rotation angles, and each translation can be defined by three translation values, i.e. movements along three different spatial directions. The time series of rotations and translations can be a six-dimensional time series for the three rotation parameters and the three translation parameters, or can also comprise six individual time series for the respective individual degrees of freedom or rotation and translation parameters. The time series of rotations and translations can also be a time series of rotation matrices and translation vectors.

[0040] In particular, the plurality of projection images comprises K projection images, wherein each projection image corresponds to one recording time period of K sequentially successive recording time periods, and the time series comprises K rotations and K translations.

[0041] Here, K is an integer greater than 1 and can for example lie in the range from 100 to 1000. In particular for CBCT, the number of projection images can lie in such a range, for example in the range from 300 to 600.

[0042] Thus, during each recording time period, in particular one of the K projection images is generated. Each projection image is thus assigned a three-dimensional rotation and a three-dimensional translation. As described above, by means of this assignment it is possible to correct the geometric operation of the image reconstruction for the respective projection image in order to achieve motion compensation.

[0043] According to at least one embodiment, in order to generate the trained algorithm, a plurality of training object motion trajectories is generated, which respectively describe an actual motion of a test object. For each training object motion trajectory, a training vector is generated in the latent space, in particular by applying an encoding neural network to the respective training object motion trajectory. A reconstruction neural network is applied to the training vector in order to generate a reconstructed object motion trajectory.

[0044] On the basis of the reconstructed object motion trajectory, if necessary using one or more further neural networks, at least one predefined loss function is analyzed. On the basis of the result of the analysis of the at least one loss function, the network parameters of the reconstruction neural network, in particular of the reconstruction neural network and of the encoding neural network and, if necessary, of the one or more further neural networks, are adjusted, in particular weighting factors and / or bias factors.

[0045] The encoding neural network can be, for example, an encoder module of an autoencoder, and the reconstruction neural network can be a decoder module of the autoencoder. In the case of a GAN, the reconstruction neural network can be a generator module of the GAN, and the encoding neural network can be a neural network provided in addition to the GAN. In this case, the at least one further neural network can be a discriminator module of the GAN.

[0046] According to at least one embodiment, the trained algorithm is designed as a generator module of a GAN network.

[0047] GAN networks have proven to be reliable tools for encoding complex input data in a latent space and enable reliable reconstruction of the input data.

[0048] GAN networks initially comprise a generator module and a corresponding discriminator module. However, the discriminator module is no longer required after the training has been completed.

[0049] In at least one embodiment, the trained algorithm is designed as a generator module of a GAN network, according to which a plurality of training object trajectories are generated for generating the trained algorithm, i.e. in particular for training the algorithm, the training object trajectories each describing an actual movement of a test object. For each training object trajectory, a training vector is generated in a latent space by applying an artificial neural network designed as an encoder to the respective training object trajectory. The generator module and the discriminator module of the GAN network are trained using the training vectors as input data.

[0050] The training of the GAN network, in particular of the generator module and the discriminator module, can take place in a known manner. Here, the generator module generates a reconstructed object trajectory on the basis of the respective training vector, and the discriminator module decides whether the reconstructed object trajectory is a reconstructed object trajectory, also referred to as a "fake sample", or a real training object trajectory, also referred to as a "real sample". The training is carried out until the discriminator module can no longer distinguish between real training object trajectories and reconstructed object trajectories. At this point, the generator module can be used to reliably generate object trajectories on the basis of arbitrary latent vectors in the latent space.

[0051] In particular, a known generator loss function and a known discriminator loss function can be used to train the generator module and the discriminator module, respectively. The encoder module can be a pre-trained, i.e. known, encoder module for encoding, or can also be trained before the GAN network or jointly with the GAN network.

[0052] The encoder module, which generates the respective training vectors, is not part of the GAN network in the narrow sense. Rather, the GAN network can be trained on the basis of arbitrary inputs, in particular noise vectors. In the present case, however, the generation is carried out purposefully on the basis of actual training object movement trajectories, which represent physically realistic movement situations, so that, conversely, the object movement trajectories generated by the generator module also correspond to physically realistic movement situations. After the training has been completed, the encoder module is no longer required, as is also the case for the discriminator module.

[0053] According to at least one embodiment, the trained algorithm is designed as a decoder module of an autoencoder.

[0054] An autoencoder essentially consists of an encoder module, which is able to convert a respective input, in this case for example a training object movement trajectory, into a latent vector, and a decoder module, which is able to reconstruct the input on the basis of the latent vector. The encoder module is no longer required after the training has been completed.

[0055] Autoencoders have proven to be reliable tools in order to encode complex input data in a latent space and to enable a reliable reconstruction of the input data. Autoencoders also have the advantage that regularization during training is not problematic compared to other networks.

[0056] In order to generate or train the autoencoder, a plurality of training object movement trajectories can likewise be generated, which each describe an actual movement of a test object. For each training object movement trajectory, a training vector is generated in the latent space by applying the encoder module of the autoencoder or variational autoencoder to the respective training object movement trajectory. A reconstructed object movement trajectory is generated on the basis of the training vector by means of the decoder module, and a loss function is analyzed on the basis of the deviation of the reconstructed object movement trajectory from the training object movement trajectory. The network parameters of the autoencoder, in particular of the encoder module and of the decoder module, are adjusted on the basis of the result of the analysis of the loss function.

[0057] The network parameters can be adjusted using known methods and known loss functions.

[0058] In the present case, the encoder module is trained purposefully on the basis of actual training object movement trajectories in order to generate training vectors, which actual training object movement trajectories represent physically realistic movement situations, so that, conversely, the object movement trajectories generated by the decoder module also correspond to physically realistic movement situations.

[0059] According to at least one embodiment, for each training object motion trajectory, the respective training object motion trajectory is generated by monitoring the test object during a respective test time period by means of the at least one camera and determining the respective training object motion trajectory on the basis of camera images generated by the at least one camera during the monitoring.

[0060] In particular, the position and / or orientation of the test object can be determined in the reference coordinate system on the basis of the camera images, and the respective rotation and translation sequence or the respective rotation and translation parameters can be calculated.

[0061] According to at least one embodiment, at least one visual marker is mounted on the test object, and the respective training object motion trajectory is determined on the basis of the respective position of the at least one visual marker in the camera images.

[0062] In particular, the position and / or orientation in the reference coordinate system can be determined from the position in the camera images, since the position and orientation of the at least one camera in the reference coordinate system are known.

[0063] In this way, it is possible to reliably record only physically realistic motion situations, thereby increasing the reliability of the trained algorithm and thus the overall reliability of the method according to the application.

[0064] The object can be, for example, the head of a patient. The motion compensation can be carried out with particularly high precision in X-ray-based imaging of the head, since the assumption of rigid motion is particularly well fulfilled, in particular in the case of an object that is not deformed.

[0065] According to another aspect of the application, a method for X-ray-based imaging is also presented. Here, a plurality of projection images of an object are generated, and the method for generating a motion-compensated image reconstruction according to the application is carried out on the basis of the plurality of projection images.

[0066] In particular, the method for X-ray-based imaging is a cone beam computed tomography method.

[0067] For application scenarios or application cases that can be derived in the method and are not described in detail here, it can be provided that a fault report and / or a request for inputting user feedback and / or an adjustment of the standard setting and / or a predetermined initial state is output according to the method.

[0068] According to another aspect of the application, a data processing device is presented, which has at least one computing unit. The at least one computing unit is set up or adjusted to carry out the method for generating a motion-compensated image reconstruction according to the application.

[0069] According to another aspect of the present application a device for X-ray imaging based is given. The device has an X-ray imaging modality which is arranged for generating a plurality of projection images of an object. The device has a storage device which stores a trained algorithm which is trained for assigning an object motion trajectory to a latent vector of a pre-set latent space. The device has at least one computing unit which is arranged for determining an optimal object motion trajectory by applying the trained algorithm to an optimal latent vector and generating a motion compensated image reconstruction from the plurality of projection images under the assumption that the object has moved according to the optimal object motion trajectory during execution of the imaging process.

[0070] The at least one computing unit is arranged for minimizing a pre-set cost function using the latent vector as an optimization parameter in order to determine the optimal latent vector and, in this regard, for each optimization step using a current latent vector, determining a current object motion trajectory by applying the trained algorithm to the current latent vector and calculating a current value of the cost function from the plurality of projection images under the assumption that the object has moved according to the current object motion trajectory during execution of the imaging process.

[0071] A computing unit can in particular be understood as a data processing device comprising a processing circuit. A computing unit can in particular process data for performing a computing operation. The computing operation can also include, if necessary, an operation for performing a retrieval access to a data structure, for example a look-up table LUT (English: "look-up table").

[0072] A computing unit can in particular comprise one or more computers, one or more microcontrollers and / or one or more integrated circuits, for example one or more application-specific integrated circuits ASIC (English: "application-specific integrated circuit"), one or more field-programmable gate arrays FPGA and / or one or more system on a chip SoC (English: "system on a chip"). A computing unit can also include one or more processors, for example one or more microprocessors, one or more central processor units CPU (English: "central processing unit"), one or more graphics processor units GPU (English: "graphics processing unit") and / or one or more signal processors, in particular one or more digital signal processors DSP. A computing unit can also include a physical or virtual complex of computers or other mentioned units.

[0073] In different embodiments, a computing unit includes one or more hardware and / or software interfaces and / or one or more storage units.

[0074] The storage unit can be designed as a volatile data memory, for example a dynamic random access memory, DRAM (English: "dynamic random access memory"), or a static random access memory, SRAM (English: "static random access memory"), or as a non-volatile data memory, for example a read-only memory, ROM (English: "read-only memory"), a programmable read-only memory, PROM (English: "programmable read-only memory"), an erasable programmable read-only memory, EPROM (English: "erasable programmable read-only memory"), an electrically erasable programmable read-only memory, EEPROM (English: "electrically erasable programmable read-only memory"), a flash memory or flash EEPROM, a ferroelectric random access memory, FRAM (English: "ferroelectric random access memory"), a magnetoresistive random access memory, MRAM (English: "magnetoresistive random access memory"), or a phase-change random access memory, PCRAM (English: "phase-change random access memory").

[0075] Further embodiments of the apparatus according to the application directly result from different design options of the method according to the application and vice versa. In particular, individual features and the respective explanations and advantages related to different embodiments of the method according to the application can be similarly transferred to the respective embodiments of the apparatus according to the application. In particular, the apparatus according to the application is designed or programmed for carrying out the method according to the application. In particular, the apparatus according to the application carries out the method according to the application.

[0076] According to a further aspect of the application, a first computer program having first instructions is provided. The first instructions, when executed by a data processing apparatus, in particular a data processing apparatus according to the application, cause the data processing apparatus to carry out the method according to the application for generating a motion compensated image reconstruction.

[0077] According to a further aspect of the application, a second computer program having second instructions is provided. The second instructions, when executed by an apparatus for X-ray imaging according to the application, in particular by at least one computing unit of the apparatus, cause the apparatus to carry out the method according to the application for generating a motion compensated image reconstruction and / or to carry out the method according to the application for X-ray imaging.

[0078] According to another aspect of the present application a computer readable storage medium is provided, storing the first computer program and / or the second computer program according to the present application.

[0079] The first computer program, the second computer program and the computer readable storage medium can be understood as respective computer program products having the first and / or second instructions.

[0080] Further features of the present application are derived from the figures and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and / or shown in the figures are not only able to be included in the present application in the respective combination given, but also in other different combinations. BRIEF DESCRIPTION OF DRAWINGS

[0081] The present application is explained below in more detail according to specific embodiments and the associated drawings. In the drawings, identical or functionally identical elements can be provided with the same reference signs. Identical or functionally identical elements need not necessarily be described repeatedly with respect to different figures.

[0082] In the drawings:

[0083] Figure 1 shows a schematic diagram illustrating an exemplary embodiment of a method for generating a motion compensated image reconstruction according to the present application;

[0084] Figure 2 shows a schematic diagram illustrating different parameter spaces for parameterization of an object motion trajectory;

[0085] Figure 3 shows a schematic diagram illustrating a training phase for training an algorithm for performing an exemplary embodiment of a method for generating a motion compensated image reconstruction according to the present application; and

[0086] Figure 4 shows a schematic flow chart illustrating another exemplary embodiment of a method for generating a motion compensated image reconstruction according to the present application. DETAILED DESCRIPTION

[0087] In Figure 1Fig. 1 shows a device 1 for X-ray imaging based on a trained algorithm 12 according to the present application. The device 1 has an X-ray imaging modality 2. The X-ray imaging modality 2 is exemplarily shown as a cone beam computed tomography (CBCT) apparatus and has e.g. a C-arm 3 with an X-ray source 4 and an oppositely arranged X-ray detector 5. Between the X-ray source 4 and the X-ray detector 5 an object 6, e.g. a head of a patient, is arranged. Further, the device 1 has a computing unit 7. The device 1, in particular the computing unit 7, has a storage device (not shown) storing a trained algorithm 12 (cf. Fig. 2) trained for assigning an object motion trajectory to a latent vector of a pre-defined latent space (cf. Fig. 3). Figure 3 , Figure 4 Figure 3 , Figure 4 .

[0088] The device 1 is in particular configured for performing a method for X-ray imaging based on a trained algorithm 12 according to the present application. To this end, the X-ray imaging modality 2 generates a plurality of projection images of the object 6 in particular according to known protocols for CBCT. For example, about 500 projection images of the object 6 can be generated. Further, the method for generating a motion compensated image reconstruction 14 according to the present application is performed. A flowchart of this method is shown in Fig. 4. Figure 4

[0089] In particular, a best object motion trajectory 10' for the object 6 is generated here by applying the trained algorithm 12 to the best latent vector in the latent space 13, and a motion compensated image reconstruction 14 is generated from the plurality of projection images under the assumption that the object 6 has moved according to the best object motion trajectory 10' during the execution of the imaging process, i.e. during the generation of the projection images.

[0090] The best latent vector is determined here in such a way that a pre-defined cost function is minimized using the latent vector as an optimization parameter. For each optimization step using a current latent vector, a current object motion trajectory is determined by applying the trained algorithm 12 to the current latent vector. A current image reconstruction is generated from the plurality of projection images. This is done under the assumption that the object 6 has moved according to the current object motion trajectory during the execution of the imaging process. A current value of the cost function is calculated from the current image reconstruction.

[0091] Alternatively, the current value of the cost function can be calculated from a pre-defined set of consistency conditions related to the plurality of projection images, so that no current image reconstruction needs to be generated in each individual optimization step.

[0092] The trained algorithm 12 can e.g. be a decoder module 12 of an autoencoder 9 as in Fig. 2. Figure 3 ​​In the training phase, the autoencoder 9 has an encoder module 11 that can encode the training subject motion trajectories 10 into respective latent vectors of a latent space 13. A decoder module 12 reconstructs the training subject motion trajectories 10 based on the respective latent vectors.

[0093] The model of rigid motion without any constraints is based on rotations and translations, usually defined as the special three-dimensional Euclidean group SE(3). This model of motion is very powerful, however most of the motion cases that can be described by SE(3) are unrealistic. In principle, the motions that can be described by rotations and translations are Figure 2 In the training phase, the autoencoder 9 has an encoder module 11 that can encode the training subject motion trajectories 10 into respective latent vectors of a latent space 13. A decoder module 12 reconstructs the training subject motion trajectories 10 based on the respective latent vectors. Figure 2 In the training phase, the autoencoder 9 has an encoder module 11 that can encode the training subject motion trajectories 10 into respective latent vectors of a latent space 13. A decoder module 12 reconstructs the training subject motion trajectories 10 based on the respective latent vectors.

[0094] Scanning the motion by spline knots has reduced the possible unrealistic motion cases. In principle, the motions that can be described by splines are Figure 2 In the training phase, the autoencoder 9 has an encoder module 11 that can encode the training subject motion trajectories 10 into respective latent vectors of a latent space 13. A decoder module 12 reconstructs the training subject motion trajectories 10 based on the respective latent vectors. Figure 2 In the training phase, the autoencoder 9 has an encoder module 11 that can encode the training subject motion trajectories 10 into respective latent vectors of a latent space 13. A decoder module 12 reconstructs the training subject motion trajectories 10 based on the respective latent vectors.

[0095] In various embodiments of the present invention, in a first step motion trajectories of real patients are scanned as test subject motion trajectories 10. This can be done for example by tracking. For example, the patient's head is marked and a camera system tracks these markers. With the markers and the camera system the motion trajectory of the patient can be calculated. By recording for example multiple patients over several hours a large database with realistic test subject motion trajectories 10 can be created.

[0096] The group of realistic test subject motion trajectories 10 can be encoded and decoded using an autoencoder 9. The autoencoder 9 can be designed as a usual autoencoder or as a variational autoencoder 9. The autoencoder 9 is defined in particular by the way that it replicates the input of the network in the output layer. By introducing a bottleneck in the middle of the network, i.e. after the encoder module 11, the autoencoder 9 is forced to learn a latent space 13 of the test subject motion trajectories 10 instead of performing a simple identity mapping.

[0097] The encoder module 11 can be an implicit neural network, e.g. in line with the SIREN architecture, as described in the article “Implicit neural representations with periodic activation functions” by V. Sitzmann, J. Martel, A. Bergman, A. Lindell, G. Wetzstein, Advances in Neural Information Processing Systems. 2020; 33:7462-73. One advantage of the SIREN architecture is the ability to compute high-order derivatives. This is because the classical ReLu activation function is replaced by a sine function. As a result, Jacobian regularization or curvature regularization is possible, which enforces a smooth and realistic mapping from the latent space to the output space, such that e.g. small changes in the latent space 13 lead to small changes in the output space. Note that this behavior is also enforced by the variational autoencoder 9. This property can be important when the latent space 13 is used as a parameterization for the motion compensation.

[0098] After the training has been successful, the decoder module 12 of the autoencoder 9 can be extracted and used to generate object motion trajectories.

[0099] GANs can be used as an alternative to the autoencoder 9. In this case, the input to the GAN is also a latent vector, the generator module of the GAN maps the latent vector to an object motion trajectory, and the loss will be based on the difference between the real object motion trajectory and a so-called fake object motion trajectory.

[0100] After the training, the decoder module 12 can generate a current object motion trajectory based on a current latent vector. This object motion trajectory can then be translated into a projection matrix and used in a reconstruction algorithm in order to create a current image reconstruction based on the current object motion trajectory. But only physically plausible cases of motion are formed compared to the parameterization using splines.

[0101] Now, the latent space 13 represents a parameterization of the object motion trajectories. This means that the parameterization X is a latent vector. The motion compensation can then be written in conventional notation by

[0102] X’ = argmin X c(r(I, X))

[0103] or, if the cost function c() is based on consistency conditions, in particular disparity consistency conditions, by

[0104] X’ = argmin X c(I, X)

[0105] Definition. The solution to this minimization problem is the optimal latent vector X' that solves the motion compensation.

Claims

1. A method for generating motion-compensated image reconstruction during X-ray-based imaging, wherein, i) Provides multiple projected images of the object generated during the imaging process; ii) Provide a trained algorithm that is trained to assign object motion trajectories to latent vectors in a predefined latent space; iii) Determine the optimal object trajectory by applying a trained algorithm to the optimal latent vector, and generate motion-compensated image reconstruction from the plurality of projected images, assuming that the object has moved along the optimal object trajectory during the imaging process; and iv) Determine the optimal latent vector by using the latent vector as an optimization parameter to minimize a preset cost function. For each optimization step using the current latent vector - By applying a trained algorithm to the current latent vector, the trajectory of the current object is determined; and - Assuming that the object has moved along its current trajectory during the imaging process, calculate the current value of the cost function based on the plurality of projected images.

2. The method according to claim 1, wherein, The trained algorithm is designed to assign time series of rotations and translations in three-dimensional space as object motion trajectories to latent vectors in the latent space.

3. The method according to claim 2, wherein, - The plurality of projected images comprises K projected images, wherein each projected image corresponds to one of K consecutively recorded time periods; and - The time series includes K rotations and K translations.

4. The method according to any one of the preceding claims, wherein, The trained algorithm is designed as a decoder module of an autoencoder or variational autoencoder.

5. The method according to claim 4, wherein, To generate the trained algorithm, multiple training object motion trajectories are generated. These training object motion trajectories describe the actual motion of the test object, and for each training object motion trajectory... - Generate training vectors in the latent space by applying the encoder module of an autoencoder or variational autoencoder to the corresponding training object's motion trajectory. - The decoder module generates the reconstructed object motion trajectory based on the training vectors; - A loss function based on the deviation analysis between the reconstructed object's motion trajectory and the training object's motion trajectory; and - Adjust the network parameters of the autoencoder or variational autoencoder based on the analysis results of the loss function.

6. The method according to claim 1, wherein, The trained algorithm is designed as the generator module of the GAN network.

7. The method according to claim 6, wherein, - In order to generate the trained algorithm, multiple training object motion trajectories are generated, which respectively describe the actual motion of the test object; - For each training object's motion trajectory, training vectors are generated in the latent space by applying an artificial neural network designed as an encoder to the corresponding training object's motion trajectory. - Use training vectors as input data to train the generator and discriminator modules of the GAN network.

8. The method according to any one of claims 5 or 7, wherein, For each training object's motion trajectory, in order to generate the corresponding training object's motion trajectory, the test object is monitored by at least one camera during the corresponding test time period, and the corresponding training object's motion trajectory is determined based on the camera images generated by at least one camera during the monitoring period.

9. The method according to claim 8, wherein, At least one visual marker is installed on the test object, and the corresponding motion trajectory of the training object is determined based on the corresponding position of the at least one visual marker in the camera image.

10. The method according to claim 1, wherein, For each optimization step - Assuming that the object has moved along its current trajectory during the imaging process, generate a current image reconstruction based on multiple projected images, and calculate the current value of the cost function based on the current image reconstruction; or - Calculate the current value of the cost function based on a pre-defined set of consistency conditions related to multiple projected images.

11. A method for X-ray-based imaging, wherein, Generate multiple projected images of an object, and implement, based on the multiple projected images, a method for generating motion-compensated image reconstruction during X-ray-based imaging according to any one of the preceding claims.

12. The method according to claim 11, wherein, This method is cone-beam computed tomography (CBCT).

13. A data processing apparatus having at least one computing unit configured to implement a method for generating motion-compensated image reconstruction in an X-ray-based imaging process according to any one of claims 1 to 10.

14. An apparatus for X-ray-based imaging, comprising: an X-ray imaging mode configured to generate multiple projection images of an object; a storage device for storing a trained algorithm trained to assign object motion trajectories to latent vectors in a preset latent space; and at least one computing unit configured to, - By applying a trained algorithm to the optimal latent vector to determine the optimal object motion trajectory, and assuming that the object has moved along the optimal object motion trajectory during the imaging process, motion-compensated image reconstruction is generated from the plurality of projected images; and - The pre-defined cost function is minimized using the latent vector as an optimization parameter in order to determine the optimal latent vector. Here, for each optimization step using the current latent vector, the current object trajectory is determined by applying the trained algorithm to the current latent vector, and the current value of the cost function is calculated based on the plurality of projected images, assuming that the object has moved along the current object trajectory during the imaging process.

15. A computer program product having - A first instruction, which, when executed by the data processing apparatus, causes the data processing apparatus to perform the method according to any one of claims 1 to 10; and / or - A second instruction, which, when executed by the apparatus according to claim 14, causes the apparatus to perform the method according to any one of claims 1 to 12.

Images