Mutual registration of 2D image data with 3D image data

Abstract

For the mutual registration of 2D image data with 3D image data, particularly concerning a patient, a 3D reference representation of an area of ​​interest of the patient (3) and a 2D representation of the area of ​​interest are obtained. A pose (19) is determined in a predefined three-dimensional coordinate system, and based on the 3D reference representation (11), a 2D reference representation (12) corresponding to the determined pose (19) is generated. Display information (13) for display on a display device (5) is generated depending on the 2D representation (10) and the 2D reference representation (12). At least one feature map is generated depending on the 2D image (10) using an encoder module (15) of a trained masked autoencoder and the pose (19) is determined by applying a pose estimation module (16) of a trained artificial neural network to the at least one feature map.

Description

[0001] The present invention relates to a computer-implemented method for the mutual registration of 2D image data with 3D image data. The image data can, in particular, relate to a patient. A 3D reference representation of an area of ​​interest of a patient and a 2D representation of the area of ​​interest are obtained. A pose in a predetermined three-dimensional coordinate system is determined, and based on the 3D reference representation, a 2D reference representation corresponding to the determined pose is generated. A 2D representation based on a pose of a 3D representation is thus considered to be registered to the 3D representation. The invention further relates to a computer-implemented training method for training an algorithm to carry out a computer-implemented method according to the invention.The invention further relates to a data processing device for carrying out a computer-implemented method or computer-implemented training method according to the invention, to a system for carrying out a computer-implemented method according to the invention, and to a corresponding computer program product.

[0002] The image data can relate specifically to a patient and be intended for visual support during a medical procedure. In image-guided procedures, such as X-ray-guided interventions, it is advantageous to combine information from a pre-operative 3D reference image with information from an intra-operative 2D image, such as an X-ray, to improve the information immediately available to medical personnel during the procedure. The spatial alignment of the 3D reference image with the current patient position is sometimes crucial for accurate combination. This can be achieved through a process called 2D / 3D registration, in which 2D image data is registered alongside 3D image data.

[0003] A conventional workflow for performing 2D / 3D registration relies on a manual initialization step followed by optimization-based registration. As explained in the document S. Jaganathan et al.: “Deep Iterative 2D / 3D Registration.” (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 383-392, Springer, 2021), machine learning-based 2D / 3D registration techniques can also be successfully employed. However, the data annotation requirements and the comparatively high complexity of the process for training the various components of the machine learning-based 2D / 3D registration algorithms represent a significant disadvantage, particularly for clinical applications.

[0004] Furthermore, even learning-based approaches still require an initialization step, usually manual, to bring both the 3D reference representation and the 2D mapping into the capture range of the registration algorithms.

[0005] In the publication by A. Kanazawa et al.: “End-to-end Recovery of Human Shape and Pose.” (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7122-7131, 2018), Human Mesh Recovery (HMR) is described, an end-to-end approach for reconstructing a complete 3D mesh of a human body from a single RGB image using a neural network. Specifically, a set of pose parameters and a set of shape parameters are generated from the image, and the 3D mesh is then constructed based on these parameters.

[0006] In the publication K. He et al.: “Masked autoencoders are scalable vision learners.”, (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022), masked autoencoders are presented as scalable self-supervised learners for computer vision tasks. During their training, random areas of the input image are masked, and the missing pixels are reconstructed.

[0007] In the publication J. Wang et al.: “Dynamic 2-d / 3-d rigid registration framework using point-to-plane correspondence model.” (IEEE transactions on medical imaging 36, 9, 1939-1954, 2017) a method for rigid 2D / 3D registration is described that uses a point-to-plane correspondence model.

[0008] It is an object of the present invention to further automate the mutual registration of 2D image data with 3D image data, in particular so that no manual initialization is required for registration.

[0009] This problem is solved by the subject matter of the independent claim. Advantageous further developments and preferred embodiments are the subject matter of the dependent claims.

[0010] The invention is based on the idea of ​​extracting features from a 2D image using a masked authoring encoder, determining a pose in a predefined three-dimensional coordinate system based on these features, and generating a corresponding 2D reference representation based on the 3D reference representation according to the pose thus determined. The 2D image is registered with the 3D reference representation from the outset, and determining the pose is therefore equivalent to registering the 2D image with the 3D reference representation. Display information for visual support is generated and, in particular, displayed depending on the 2D image and the 2D reference representation.

[0011] According to one aspect of the invention, a computer-implemented method for the mutual registration of 2D image data with 3D image data is provided. The image data can, in particular, relate to a patient. A 3D reference representation, especially a pre-operative 3D reference representation, of an area of ​​interest to the patient is obtained. A 2D image of the area of ​​interest, especially an intra-operative 2D image, is also obtained. A pose is determined in a predetermined three-dimensional coordinate system, and based on the 3D reference representation, a 2D reference representation corresponding to the determined pose is generated. Depending on the 2D image, display information is generated for display on a display device and, in particular, displayed on the display device.

[0012] To determine the pose, at least one feature map is generated based on the 2D image using an encoder module of a trained masked authoring encoder. The pose is then determined by applying a pose estimation module of a trained artificial neural network to this feature map.

[0013] Unless otherwise specified, all steps of the computer-implemented method can be performed by a data processing device comprising at least one processing unit. In particular, the at least one processing unit is configured or adapted to perform the steps of the computer-implemented method. For this purpose, the at least one processing unit can, for example, store a computer program containing instructions which, when executed by the at least one processing unit, cause the at least one processing unit to execute the computer-implemented method.

[0014] For each embodiment of a computer-implemented method according to the invention, an embodiment of a corresponding method that is not purely computer-implemented is directly obtained by including a respective method step for generating the 3D reference representation and / or a method step for generating the 2D image.

[0015] The 3D reference representation and / or the 2D image can be obtained in particular by storing them on a storage medium of the at least one computing unit and reading them from the at least one computing unit.

[0016] The image data can be used, in particular, for visual support during a medical procedure. The medical procedure may be, for example, a diagnostic procedure and / or a therapeutic procedure, or the like. However, the medical procedure or other surgical measures, or the like, are not part of the computer-implemented method according to the invention.

[0017] A 3D reference image is a 3D representation of body parts, organs, or other anatomical objects within the area of ​​interest and, optionally, outside the area of ​​interest, determined by an imaging procedure prior to a medical intervention, i.e., pre-operatively. The 3D reference image can, for example, include a 3D computed tomography (CT) scan, a reconstruction, or a 3D thomosynthesis volume. In addition to the visual representation of anatomical objects, the 3D reference image can also contain planning information.

[0018] The 2D image can be a 2D image generated during or immediately before a procedure, i.e., intraoperatively, for example, an X-ray projection image. The 3D reference image and the 2D image are not necessarily generated using the same imaging system. In the case of X-ray imaging, the 3D reference image can be generated using a CT system, for example, and the imaging system used to generate the 2D image can be a C-arm X-ray system, such as a C-arm X-ray angiography system. It is also possible that different imaging modalities are used, so that the 3D reference image is, for example, a CT reconstruction and the 2D image is a magnetic resonance image or the like.

[0019] While the 3D reference representation depicts the area of ​​interest in 3D, the 2D image depicts the area of ​​interest in 2D, particularly according to a specific viewing direction, for example, the X-ray projection direction.

[0020] The pose can be understood as the pose of the image content of the 2D image. The pose can be understood as including a corresponding position, in particular a three-dimensional position, and a corresponding orientation, in particular a three-dimensional orientation, especially with respect to the given three-dimensional coordinate system. The pose can thus be described as a 3D translation vector that connects the origin of the coordinate system with the position, and as a rotation in three-dimensional space by means of which a reference orientation is transformed into the orientation of the 2D image. The pose is therefore defined in particular by six parameters, where the position defines three position or translation parameters and the orientation three rotation angles, for example, Euler angles.

[0021] The 3D reference representation is not necessarily given in the same coordinate system, but its position or pose in the coordinate system is known or predetermined. Accordingly, if the pose of the 2D image is known, the 2D reference representation can be generated based on the 3D reference representation such that the pose of the resulting 2D reference representation corresponds to, or in particular approximately corresponds to, the pose of the 2D image.

[0022] The pose can also be interpreted in terms of camera parameters, i.e., camera position and camera orientation, of a virtual camera. Accordingly, the virtual camera used to generate the 2D image is the same as the virtual camera used to generate the 2D reference representation.

[0023] The pose estimation module itself can be considered an artificial neural network and can, for example, be designed and trained according to the HMR approach by Kanazawa et al. mentioned earlier. The potentially high-dimensional set of pose parameters is reduced here to the 6D pose as described above. The set of shape parameters from Kanazawa et al. is not required in this case.

[0024] The display information depends on the 2D image and the 2D reference representation. For example, the display information can include a combination, fusion, or overlay of the 2D image with the 2D reference representation, such as a registered overlay of the 2D image with the 2D reference representation. Since the 2D image and the 3D reference representation are each in the same pose, a consistent display of the information can be achieved, thus providing particularly reliable support during a medical procedure. If the display information includes, for example, the registered overlay of the 2D image and the 2D reference representation, then providing the 2D reference representation can be considered a pre-registration or initialization step for the actual registration.

[0025] Instead of or in addition to registration, the 2D reference representation and the 2D image can also be used for other purposes or for other image processing algorithms, for example for patch matching or for multi-level approaches.

[0026] According to at least one embodiment, at least one further feature map is generated using the encoder module or a trained further encoder module, depending on the 2D reference representation. Image registration information is generated depending on the at least one feature map and the at least one further feature map.

[0027] The display information is generated by registering and superimposing the 2D image and the 2D reference representation according to the image registration information. For example, a modified 2D image is created from the 2D image according to the image registration information and superimposed on the 2D reference representation. Alternatively, or additionally, the display information is generated by registering and superimposing another 2D image of the patient and another 2D reference representation of the patient according to the image registration information.

[0028] The encoder module, or the additional encoder module, is applied, for example, to the 2D reference representation to generate at least one further feature map, or to input data that depends on the 2D reference representation. The input data can, for example, correspond to a preprocessed version of the 2D reference representation.

[0029] Specifically, based on the 2D image, the 2D reference representation, and the image registration information, a superimposed image is generated, and the display information includes this superimposed image. The superimposed image can then be displayed using the display device.

[0030] Preferably, the encoder module is applied to both the 2D representation to generate at least one feature map and the 2D reference representation to generate at least one additional feature map. Thus, the same encoder module is used for both initialization and registration. Firstly, the advantages of the masked autoencoder also apply to the generation of the at least one additional feature map. Secondly, the overall training is simplified, since only one encoder module, rather than two, needs to be trained.

[0031] According to at least one embodiment, the image registration information includes three-dimensional rotation information, in particular three rotation angles or equivalent rotation parameters, and / or three-dimensional translation information, in particular three position coordinates or equivalent translation parameters. The transformation parameters are determined using a trained correspondence estimation module on further input data. The further input data includes at least one feature map and at least one further feature map. The display information is generated by registering and superimposing the 2D image and the 2D reference representation relative to each other according to the transformation parameters, and / or by registering and superimposing the further 2D image of the patient and the further 2D reference representation of the patient relative to each other according to the transformation parameters.

[0032] In other words, image registration is performed according to the image registration information based on the translation parameters and / or the rotation parameters, preferably based on the translation parameters and the rotation parameters.

[0033] The fact that the transformation parameters are determined using the correspondence estimator can be understood, in particular, to mean that determining the transformation parameters involves applying the correspondence estimator to the input data. However, further steps may be necessary to obtain the transformation parameters from the input data.

[0034] The correspondence estimation module itself can be considered an artificial neural network trained to solve, or approximate, the corresponding correspondence problem involving a 2D mapping and a 2D reference representation. Specifically, the correspondence estimation module is trained to receive at least one feature map and at least one other feature map as input data and to generate correspondence information based on this data. The correspondence information describes, for each pixel of the 2D mapping, how it can be mapped to a corresponding pixel of the 2D reference representation, or vice versa. For example, the correspondence information can define a 2D displacement vector for each pixel, which can be calculated, for instance, according to an optical flow or similar method.The transformation parameters can then be calculated based on the correspondence information.

[0035] The correspondence estimation module can, for example, be designed based on the RAFT architecture, as explained in the publication by Jaganathan et al. mentioned at the beginning.

[0036] According to at least one embodiment, by applying the correspondence estimation module to at least one further feature map, a data set relating to an optical flow between the 2D reference representation and the 2D mapping is determined, and the transformation parameters are determined depending on the data set relating to the optical flow.

[0037] The data set concerning optical flow includes, in particular, an optical flow—for example, a 2D displacement vector corresponding to the optical flow—for each pixel of the 2D image or the 2D reference representation. If the number of pixel rows and / or pixel columns of the 2D image does not match that of the 2D reference representation, appropriate scaling can be performed to adjust the data set concerning optical flow.

[0038] Optical flux has proven to be a reliable measure for defining registration.

[0039] According to at least one embodiment, a point-to-plane correspondence model (PPC model) is used to determine the transformation parameters depending on the data set relating to the optical flow.

[0040] In other words, a so-called solver module is applied to the dataset concerning the optical flow in order to calculate the transformation parameters according to the PPC model. This allows for a precise determination of the transformation parameters, i.e., information relating to three-dimensional space, from the 2D information of the optical flow.

[0041] According to at least one embodiment, the 3D reference image is a computed tomography (CT) reconstruction. The 3D reference image therefore contains particularly comprehensive and detailed information that can be used in the form of a 2D reference image to optimally support a surgical procedure.

[0042] According to at least one embodiment, the 2D reference representation is generated as a digitally reconstructed X-ray image, DRR.

[0043] This has the advantage that the DRR can be superimposed particularly well onto a real X-ray projection image as a 2D image. To generate the DRR, a rendering module is applied to the 3D reference image, especially the CT reconstruction. This module uses a known rendering algorithm for DRR rendering to create the 2D representation as a projection of the 3D reference image rendered according to the pose.

[0044] According to at least one embodiment, the 2D image is an X-ray image, also known as an X-ray projection image.

[0045] To generate the 2D image, X-rays are produced using an X-ray source, and the portions of the X-rays passing through the patient are detected by an X-ray detector. The X-ray projection image is then generated based on the corresponding detector signals produced by the X-ray detector.

[0046] In particular, the 3D reference image could be the CT reconstruction, and the 2D image the X-ray projection image. Alternatively, the 2D image could be, for example, an MRI image or an image generated based on ultrasound imaging.

[0047] For example, in some embodiments the 3D reference representation and the 2D image can be generated using different imaging modalities.

[0048] For example, the 2D image could be an X-ray projection image, and the 3D reference image could be generated using a non-X-ray-based imaging modality. Conversely, the 3D reference image could also be the CT reconstruction, whereas the 2D image might not be an X-ray projection image.

[0049] According to at least one embodiment, the pose estimation module is designed as an iterative regression module.

[0050] The iterative regression module can, for example, be designed according to HMR, as described in the publication by Kanazawa et al. mentioned at the beginning. This allows the patient-specific set of pose parameters and the patient-specific set of shape parameters to be determined easily and with high accuracy.

[0051] According to a further aspect of the invention, a computer-implemented training method for training an algorithm to perform a computer-implemented method according to the invention for the mutual registration of 2D image data with 3D image data is provided. The encoder module of the untrained or partially trained masked autoencoder, a decoder module of the untrained or partially trained masked autoencoder, and the pose estimation module are trained together using a common loss function.

[0052] In particular, in some embodiments of the computer-implemented method according to the invention for the mutual registration of 2D image data with 3D image data, the encoder module and the pose estimation module are trained according to a computer-implemented training method according to the invention. The decoder module is, for example, only required during the training, and not afterwards.

[0053] This enables end-to-end training of the algorithm used for the computer-implemented method according to the invention.

[0054] The display information can be generated in a computer-implemented method according to the invention, for example by means of at least one further trained artificial neural network, for example the correspondence estimation module, based on the 2D image and the 2D reference representation.

[0055] In corresponding embodiments, the encoder module, the decoder module, the pose estimator module and at least one other artificial neural network, in particular the correspondence estimator module, are trained together using the common loss function.

[0056] In principle, the individual modules, such as the pose estimation module and / or the correspondence estimation module, can also be trained independently of the masked autoencoder, i.e., the encoder module and the decoder module. However, joint training allows the features extracted by the encoder module to be optimized for multiple applications without significantly increased effort.

[0057] According to a further aspect, a data processing device with at least one computing unit is specified. The at least one computing unit is configured to carry out a computer-implemented method and / or a computer-implemented training method according to the invention.

[0058] A computing unit can be understood, in particular, as a data processing device containing a processing circuit. The computing unit can therefore process data to perform arithmetic operations. This may also include operations to perform indexed access to a data structure, such as a lookup table (LUT).

[0059] The computing unit may, in particular, contain one or more computers, one or more microcontrollers, and / or one or more integrated circuits, for example, one or more application-specific integrated circuits (ASICs), one or more field-programmable gate arrays (FPGAs), and / or one or more systems on a chip (SoCs). The computing unit may also contain one or more processors, for example, one or more microprocessors, one or more central processing units (CPUs), one or more graphics processing units (GPUs), and / or one or more signal processors, in particular one or more digital signal processors (DSPs). The computing unit may also include a physical or virtual array of computers or other units of the aforementioned type.

[0060] In various embodiments, the computing unit includes one or more hardware and / or software interfaces and / or one or more storage units.

[0061] A storage unit can be volatile data storage, for example as dynamic random access memory (DRAM) or static random access memory (SRAM), or as non-volatile data storage, for example as read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or flash EEPROM, ferroelectric random access memory (FRAM), or magnetoresistive random access memory.It can be designed as MRAM (magnetoresistive random access memory) or as phase-change random access memory, PCR_AM (phase-change random access memory).

[0062] According to a further aspect of the invention, a system for the mutual registration of 2D image data with 3D image data is provided. The system comprises a data processing device according to the invention, as well as an imaging modality. The imaging modality is configured to generate the 2D image, in particular controlled by at least one computing unit.

[0063] According to at least one embodiment, the imaging modality is configured as an X-ray imaging modality. The X-ray imaging modality comprises, in particular, an X-ray source and an X-ray detector. The at least one processing unit is configured to control the X-ray source to generate X-rays. The X-ray detector is configured to generate detector signals based on the proportion of X-rays passing through the patient and to transmit these signals to the at least one processing unit. The at least one processing unit is configured to generate the 2D image as an X-ray projection image based on the detector signals.

[0064] The X-ray imaging modality can be designed in particular as a C-arm X-ray angiography system.

[0065] According to at least one embodiment, the system includes a display device and the at least one computing unit is configured to control the display device to display the display information.

[0066] Further embodiments of the system according to the invention follow directly from the various configurations of the computer-implemented method according to the invention, and vice versa. In particular, individual features and corresponding explanations as well as advantages relating to the various configurations of the computer-implemented method according to the invention can be transferred analogously to corresponding configurations of the system according to the invention. In particular, the system according to the invention is configured or programmed to carry out a computer-implemented method according to the invention. In particular, the system according to the invention carries out the computer-implemented method according to the invention.

[0067] According to a further aspect of the invention, a computer program with instructions is specified. When the instructions are executed by a data processing device, in particular a data processing device according to the invention, the instructions cause the data processing device to carry out a computer-implemented method and / or a computer-implemented training method according to the invention.

[0068] The instructions can be provided, for example, as program code. This program code can be provided, for example, as binary code or assembly language, and / or as source code in a programming language such as C, and / or as a program script, such as Python.

[0069] According to another aspect of the invention, a computer-readable storage medium is specified which stores a computer program according to the invention.

[0070] The computer program and the computer-readable storage medium according to the invention can each be considered a computer program product with the commands.

[0071] Further features of the invention emerge from the claims, the figures and the description of the figures.

[0072] The features and combinations of features mentioned above in the description, as well as those subsequently mentioned in the figure description and / or shown in the figures, may be encompassed by the invention not only in the combinations specified, but also in other combinations. In particular, the invention may also encompass embodiments and combinations of features that do not have all the features of an originally formulated claim. Furthermore, the invention may encompass embodiments and combinations of features that go beyond or deviate from the combinations of features set out in the cross-references to the claims.

[0073] The invention is explained in more detail below using specific embodiments and associated schematic drawings. In the figures, identical or functionally equivalent elements may be provided with the same reference numerals. The description of identical or functionally equivalent elements may not necessarily be repeated for different figures.

[0074] The figures show Fig. 1 a schematic representation of a system according to the invention for visual support during a medical procedure; Fig. 2 a schematic block diagram of an algorithm for carrying out an exemplary embodiment of a computer-implemented method according to the invention for visual support during a medical procedure; and Fig. 3 A schematic block diagram of an exemplary embodiment of a computer-implemented training method.

[0075] Fig. Figure 1 shows a schematic representation of a system 1 according to the invention for the mutual registration of 2D image data with 3D image data. The system 1 comprises an imaging modality 2 and a data processing device 6 with at least one computing unit.

[0076] The at least one computing unit is configured to perform a computer-implemented method according to the invention for the mutual registration of 2D image data with 3D image data. The image data can, in particular, relate to a patient. For this purpose, the at least one computing unit receives a 3D reference representation 11, for example, a pre-operative CT reconstruction, of an area of ​​interest of a patient 3, and a 2D Fig. , in particular intra-operative 2D imaging, of the area of ​​interest.

[0077] For example, imaging modality 2 can be an X-ray imaging modality, such as a C-arm system, with an X-ray source 7 and an X-ray detector 8. In the 2D- Fig. This is in particular an X-ray projection image. System 1, in particular the imaging modality 2, may also include a patient table 4. System 1 may also include a display device 5, which in particular is controllable by at least one computing unit.

[0078] The at least one computing unit determines a pose 19 in a given three-dimensional coordinate system and generates a 2D reference representation 12 corresponding to the determined pose 19 based on the 3D reference representation 11. For this purpose, the at least one computing unit generates at least one feature map using an encoder module 15 of a trained masked autoencoder, depending on the 2D reference representation 11. Fig. The at least one computing unit determines the pose 19 by applying a pose estimation module 16 of a trained artificial neural network for pose estimation to the at least one feature map.

[0079] If the 3D reference image is the CT reconstruction, the 2D reference image 12 can, for example, be a digitally reconstructed X-ray image, DRR. The at least one processing unit generates display information 13 for display on the display device 5 depending on the 2D Fig. and the 2D reference representation 12 is generated.

[0080] Fig. Figure 2 shows a schematic block diagram of an algorithm 9 for carrying out a further exemplary embodiment of a computer-implemented method according to the invention for visual support during a medical procedure, which, for example, uses a system 1 consisting of Fig. 1 can be carried out, in particular by means of at least one computing unit.

[0081] As mentioned above, the encoder module 15 is applied to the 2D- Fig. applied to generate at least one feature map. Optionally, the 2D- Fig. The 2D image is randomly masked using a random masking module 21, and the masked 2D image can be fed to the encoder module 15 to generate at least one feature map. The pose 19 is then generated as described by applying the pose estimation module 16 to the at least one feature map. A rendering module 14 receives the pose 19 and, based on this, generates the 2D reference representation 12 from the 3D reference representation 11.

[0082] The 2D reference representation 12 is then fed back to the encoder module 15, optionally after masking by the random masking module 21, to generate at least one further feature map. The at least one processing unit generates image registration information, in particular transformation parameters 20 for three-dimensional rotation and / or translation, depending on the at least one feature map and the at least one further feature map. For this purpose, the at least one processing unit applies, in particular, a trained correspondence estimation module 17 to the at least one feature map and the at least one further feature map. The correspondence estimation module 17 outputs, for example, a data set relating to an optical flow between the 2D reference representation 12 and the 2D- Fig. . For example, a PPC solver 18 can be applied to the data set concerning the optical flow to determine the transformation parameters 20.

[0083] The display information 13 is generated, for example, by the 2D- Fig. and the 2D reference representation 12 are registered and superimposed according to the transformation parameters 20.

[0084] In some embodiments, multiple masked 2D images can be generated to achieve a more robust pose estimation using the pose estimation module 16. Pose 19 can be considered the initial starting point for registration. The coded representation of the 2D- Fig. , i.e., at least one feature map, is used together with the coded representation of the 2D reference representation 12, i.e., at least one further feature map, to determine the correspondence between the 2D- with the correspondence estimation module 17. Fig. -Reference representation 11 to be estimated. The matching points can be used as input for the PPC solver 18 to estimate the exact three-dimensional transformation parameters 20.

[0085] The proposed pipeline can, for example, be applied iteratively until convergence is achieved by predicting the relative poses in successive iterations. Furthermore, test-time optimization using differentiable rendering can be easily applied to Algorithm 9, allowing predictions to be updated with gradient-based methods by backpropagating image similarity loss between the predicted image and the input image.

[0086] The example shown is a sample application. The use of the Posen Estimation Module 16, the Correspondence Estimation Mode 17, and, for example, Patch Matching, makes it possible to combine these components into various workflows, such as for a multi-level approach.

[0087] Fig. Figure 3 schematically shows a block representation of an exemplary embodiment of a computer-implemented training procedure with which, for example, algorithm 9 can be trained.

[0088] This provides a simple approach for training the various components of Algorithm 9. The approach utilizes masked image modeling using the masked autoencoder from the aforementioned publication by He et al.

[0089] 3D CT reconstructions, or static DRRs 25 and / or moving DRRs 26 extracted therefrom, as well as real X-ray projection images 24 can be used for training in various embodiments without the need for annotation.

[0090] The trained components can be combined in different configurations during inference, enabling individual pipelines for different workflows, including the one mentioned above. Fig. 2 described example for fully automatic 2D / 3D registration including initialization.

[0091] The masked autoencoder contains the encoder module 15 and the decoder module 22 for reconstructing the input image. Therefore, the encoder module 15 can be directly trained for both X-ray projection images 24 and DRRs 25, 26.

[0092] In the training procedure according to Fig.3. Several other tasks can be trained together with the masked autoencoder, particularly by combining the individual task-specific loss functions with the loss function for masked image reconstruction. Furthermore, surgical tools and occlusions are common in interventional imaging. The use of masked images for training makes algorithm 9 robust against such occlusions and the like during the application phase.

[0093] The pose estimation module 16 can be trained, for example, using the DRRs 25. Pose estimation module 16 can be based, for example, on the iterative regression module for pose estimation by Kanazawa et al., and predicts the 6D pose relative to a canonical position and orientation based on the feature maps from the respective masked image. With the predicted 6D pose as input for a differentiable renderer 27, it is possible to generate the predicted image from the CT reconstruction. A simple reconstruction-based loss can be used to train the network directly based on the input image and the image rendered with the predicted pose. Additionally, direct monitoring with pose regression loss can be used, for example, to stabilize the training.

[0094] The correspondence estimation module 17 can be trained, for example, by matching the features extracted from the static DRRs 25 and the moving DRRs 26, respectively. The predicted correspondence is used to predict the relative 3D transformation between the static and moving DRRs 25 and 26. The relative 3D transformation can be used to render a DRR using the differentiable renderer 28, and a simple reconstruction loss can be used to match the input with the rendered image.

[0095] Furthermore, image fields of the input images can be shuffled or swapped, and a jigsaw puzzle loss can be determined using the Patch Matching Module 23. This can be done for all input images, allowing the Encoder Module 15 to learn spatial features that are easier to match. The predicted correct position can then be used to reconstruct the input image, and the Patch Matching Module 23 can also be trained directly on the reconstruction loss.

[0096] The pose estimation module 16, the correspondence estimation module 17, and, if applicable, the patch-matching module 23 can be trained separately from the basic architecture of the masked autoencoder, i.e., the encoder module 15 and the decoder module 22, or together with it. This allows for flexible configuration of the entire network.

[0097] For example, the encoder module 15, the decoder module 22 of the masked autoencoder, the pose estimator module 16 and / or the correspondence estimator module 17 and / or the patch-matching module 23 can be trained together using a common loss function.

[0098] It should be noted that the use of the differentiable renderers 27, 28 is not mandatory, network predictions can also be monitored directly, since annotations are usually available for DRRs 25, 26.

[0099] As described, the invention enables largely automated visual support during a medical procedure on a patient to be further automated, whereby in particular no manual initialization is required in the case of 2D / 3D image registration.

[0100] In various implementations, the use of a masked image modeling approach allows for effective scaling of the training data. Scaling the data used to train the learning-based approach has been shown to significantly improve performance. The proposed approach enables a simple formulation of the training goal and thus comparatively simple training. The approach can be easily extended, for example, with regard to the number of tasks to be solved.

[0101] In various configurations, representations learned by the encoder module can be directly applied to multiple components in the 2D / 3D registration workflow. The flexible arrangement of the various trained components allows for the creation of complex workflows tailored to specific scenarios. Furthermore, because the algorithm can be divided into multiple steps, debugging and error correction are simplified.

[0102] Regardless of the grammatical gender of a particular term, persons with male, female or other gender identities are included. QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited non-patent literature

[0000] S. Jaganathan et al.: “Deep Iterative 2D / 3D Registration.” (International Conference on Medical Image Computing and Computer-Assisted Intervention, pp. 383-392, Springer, 2021

[0003] A. Kanazawa et al.: “End-to-end Recovery of Human Shape and Pose.” (Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7122-7131, 2018

[0005] K. He et al.: “Masked autoencoders are scalable vision learners.”, (Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, pp. 16000-16009, 2022

[0006] J. Wang et al.: „Dynamic 2-d / 3-d rigid registration framework using point-to-plane correspondence model.“ (IEEE transactions on medical imaging 36, 9, 1939-1954, 2017

[0007]

Claims

[1] Computer-implemented method for mutual registration of 2D image data with 3D image data concerning a patient (3), wherein - a 3D reference representation (11) of an area of ​​interest of the patient (3) and a 2D image (10) of the area of ​​interest are obtained; - a pose (19) is determined in a predetermined three-dimensional coordinate system and, based on the 3D reference representation (11), a 2D reference representation (12) corresponding to the determined pose (19) is generated; - display information (13) for display on a display device (5) is generated depending on the 2D image (10) and the 2D reference representation (12); characterized by , that - at least one feature map is generated using an encoder module (15) of a trained masked autoencoder depending on the 2D image (10); and - the pose (19) is determined by applying a pose estimation module (16) of a trained artificial neural network for pose estimation to the at least one feature map. [2] A computer-implemented method according to claim 1, wherein - at least one further feature map is generated using the encoder module (15) or a trained further encoder module (15) depending on the 2D reference representation (12); - image registration information is generated depending on the at least one feature map and the at least one further feature map; and - the display information (13) is generated by registering and superimposing the 2D image (10) and the 2D reference representation (12) with each other according to the image registration information and / or by registering and superimposing a further 2D image (10) of the patient (3) and a further 2D reference representation (12) of the patient (3) with each other according to the image registration information. [3] A computer-implemented method according to claim 2, wherein - the image registration information includes transformation parameters (20) for three-dimensional rotation and / or translation, which are determined using a trained correspondence estimation module (17) on input data containing the at least one feature map and the at least one further feature map; and - the display information (13) is generated by registering and superimposing the 2D image (10) and the 2D reference representation (12) according to the transformation parameters (20) and / or by registering and superimposing the further 2D image (10) of the patient (3) and the further 2D reference representation (12) of the patient (3) according to the transformation parameters (20). [4] Computer-implemented method according to claim 3, wherein by applying the correspondence estimation module (17) to the at least one further feature map, a data set relating to an optical flow between the 2D reference representation (12) and the 2D image (10) is determined and the transformation parameters (20) are determined as a function of the data set relating to the optical flow. [5] A computer-implemented method according to claim 4, wherein a point-to-plane correspondence model (18) is used to determine the transformation parameters (20) depending on the optical flow data set. [6] Computer-implemented method according to one of the preceding claims, wherein the 3D reference representation (11) is a computed tomography reconstruction. [7] Computer-implemented method according to claim 6, wherein the 2D reference representation (12) is generated as a digitally reconstructed x-ray image. [8] Computer-implemented method according to one of the preceding claims, wherein the 2D image (10) is an X-ray image. [9] Computer-implemented method according to one of claims 2 to 7, wherein the 2D image (10) and the 3D reference representation (11) are generated by means of different imaging modalities. [10] Computer-implemented method according to one of the preceding claims, wherein the pose estimation module (16) is designed as an iterative regression module. [11] A computer-implemented training method for training an algorithm (9) for performing a computer-implemented method according to any one of the preceding claims, wherein the encoder module (15) of the untrained or partially trained masked autoencoder, a decoder module (22) of the untrained or partially trained masked autoencoder and the pose estimation module (16) are trained jointly using a common loss function. [12] Computer-implemented training method according to claim 11, wherein the computer-implemented method is designed according to claim 3 and the encoder module (15), the decoder module, the pose estimation module (16) and the correspondence estimation module (17) are trained jointly using the joint loss function. [13] Data processing device (6) with at least one computing unit which is configured to carry out a computer-implemented method according to one of claims 1 to 10 and / or a computer-implemented training method according to one of claims 11 or 12. [14] System (1) for mutual registration of 2D image data with 3D image data relating to a patient (3), the system (1) comprising a data processing device (6) according to claim 13 and an imaging modality (2) which is configured to generate the 2D image (10). [15] Computer program product comprising instructions which, when executed by a data processing device (6), cause the data processing device (6) to carry out a computer-implemented method according to one of claims 1 to 10 and / or a computer-implemented training method according to one of claims 11 or 12.

Images