Adjustments for augmented reality and / or virtual reality

Abstract

The invention relates to a display device for displaying augmented reality and / or virtual reality, wherein the display device has a sensor unit, which is designed to detect user input, wherein the display device is designed to receive a medical image data set with at least one projection mapping and a medical planning data set, wherein the display device is designed to generate and display augmented reality and / or virtual reality based on the planning data set and at least one projection mapping, wherein the display device is also designed to adjust the virtual spatial positioning of the planning data set relative to the at least one projection mapping in the augmented reality and / or virtual reality based on the user input. The invention also relates to a system, a method for registering a planning data set with an image data set, and a computer program product.

Description

Technical Field

[0001] The invention relates to a display device for displaying augmented reality and / or virtual reality, a system, a method for registering a planning data set with an image data set, and a computer program product. Background Art

[0002] In order to support medical operators (e.g., doctors) during the examination and / or treatment of the examination object, in particular during the process, pre-procedural medical image data of the examination object are often used and displayed to the medical operators. Here, the pre-procedural image data may have pre-processed information, which is used, for example, for planning the process.

[0003] A particularly realistic display of medical information, such as medical image data of an examination object, can be achieved by means of an augmented reality and / or virtual reality (AR and / or VR for short). In this case, real objects, such as medical objects and / or examination objects, are usually superimposed on virtual data, in particular medical image data and / or virtual objects, and displayed in a display. In order to have a realistic display with a high degree of immersion, an accurate registration is required within the virtual data and / or between the virtual data and the real object. For this purpose, automatic registration algorithms based on geometric and / or anatomical landmarks of the image data are usually applied. However, a disadvantage here is that the known registration algorithms require costly and simultaneously consistent annotations of landmarks in different image data. In this case, insufficient annotations may disadvantageously lead to poor registration results. Summary of the invention

[0004] The object of the present invention is therefore to enable a particularly intuitive adjustment of augmented reality and / or virtual reality.

[0005] According to the invention, the above-mentioned technical problem is solved by the corresponding content of the invention. Advantageous embodiments with expedient developments are the content of the invention.

[0006] In a first aspect, the invention relates to a display device for displaying an augmented reality and / or a virtual reality. In this case, the display device has a sensor unit, which is configured to capture user inputs. In addition, the display device is configured to receive a medical image data set having at least one projection mapping, and to receive a medical planning data set. In addition, the display device is configured to generate and display an augmented reality and / or a virtual reality based on a planning data set and at least one projection mapping. In addition, the display device is configured to adjust the virtual spatial positioning of the planning data set relative to at least one projection mapping in the augmented reality and / or the virtual reality based on the user input.

[0007] The display device can advantageously include a portable, in particular wearable, first display unit that is configured to display augmented reality and / or virtual reality (AR and / or VR for short). Here, the first display unit can be configured to be at least partially transparent and / or capable of being transparent. Advantageously, the first display unit can be configured so that it can be worn at least partially by the user within the user's field of vision. To this end, the first display unit can advantageously be configured as glasses (in particular data glasses) and / or a helmet (in particular a data helmet) and / or a screen.

[0008] Furthermore, the first display unit may be constructed to superimpose real objects (e.g. physical objects, in particular medical objects and / or examination objects) with virtual data (in particular measured and / or simulated and / or processed medical image data and / or virtual objects) and display them in a display.

[0009] The sensor unit may, for example, comprise an optical and / or acoustic and / or tactile and / or electromagnetic sensor. In addition, the sensor unit may be arranged to be at least partially integrated into the first display unit. Alternatively or additionally, the sensor unit may, for example, be arranged at least partially on a body part of a user as a glove and / or a pointing device, and / or be arranged on a medical device, in particular a medical imaging device, and / or be arranged in a fixed position in space. Here, the sensor unit is advantageously constructed to collect user input, in particular user input according to an input device. The input device may, for example, comprise a pointing device and / or an input unit and / or a body part of a user and / or an optical and / or acoustic signal. In particular, the user input may comprise a transformation instruction, which is used to adjust the virtual space positioning of the planning data set relative to at least one projection mapping in augmented reality and / or virtual reality. For this purpose, the display device may be constructed to scale the user input collected by means of the sensor unit.

[0010] Advantageously, the sensor unit can be constructed for two-dimensional and / or three-dimensional acquisition of user input, in particular user input according to an input device. This can be achieved in particular when acquiring user input by means of optical and / or tactile and / or electromagnetic sensors. Here, the user input can have a trajectory, in particular a trajectory between a spatial starting point and a spatial end point. The display device can be constructed for associating the spatial starting point with a gripping point of the planning data set, in particular at a corner and / or edge and / or contour and / or image point. Alternatively or additionally, the display device can be constructed for generating and displaying augmented reality and / or virtual reality, so that the planning data set has at least one virtual, in particular selectable operating element. Here, the display device can also be constructed for selecting at least one operating element by user input and associating it with the starting point of the trajectory.

[0011] The display device can advantageously have an interface constructed to receive a medical image data set and / or a medical planning data set. The interface can be particularly constructed to collect and / or read a computer-readable data memory and / or to receive from a data storage unit, such as a database. In addition, the display device can have a providing unit, which is constructed to generate and / or adjust augmented reality and / or virtual reality. In addition, the providing unit can be constructed to process user input, in particular a signal of a sensor unit, and / or a planning data set and / or an image data set. In addition, the medical image data set and / or the medical planning data set can be provided by another providing unit of the medical imaging device. With respect to the medical image data set, the medical planning data set can be recorded and / or provided by means of the same or different medical imaging devices as the medical image data set. The medical imaging device can be constructed, for example, as a medical X-ray device, in particular as a medical C-arm X-ray device, and / or a computer tomography device (CT) and / or a magnetic resonance device (MRI) and / or a positron emission tomography device (PET) and / or an ultrasound examination device.

[0012] Advantageously, the planning data set and the image data set can at least partially map a common examination area of ​​the examination object. Here, the examination object can be, for example, a human patient and / or an animal patient and / or a phantom and / or a workpiece. The examination area can be, for example, a spatial segment of the examination object, in particular a body area. Here, the planning data set can advantageously map the examination area of ​​the examination object before the process. In addition, the image data set can advantageously map the examination area within the process. The image data set can also have at least one (in particular two-dimensional) projection mapping, in particular an X-ray projection mapping and / or an ultrasound projection mapping, of the examination area. Advantageously, the image data set, in particular at least one projection mapping, can have an intra-process mapping of the examination area, wherein the medical object is arranged in the examination area. The medical object can be, for example, a surgical instrument and / or a diagnostic instrument, such as a catheter and / or a guidewire and / or an endoscope. Advantageously, at least one projection mapping can be recorded along a projection direction, in particular at an angle, with the aid of a medical imaging device. In addition, the image data set can have a plurality of projection mappings of the examination area, which are advantageously recorded along different projection directions with the aid of a medical imaging device. Furthermore, the image data sets, in particular the projection mapping, can be time-resolved.

[0013] The planning data set can advantageously have a two-dimensional and / or three-dimensional image of the examination region. Furthermore, the planning data set can be time-resolved.

[0014] The display device can advantageously be constructed to generate and display augmented reality and / or virtual reality based on the planning data set and at least one projection mapping. In particular, the display device can be constructed to generate augmented reality and / or virtual reality so that the planning data set is displayed in particular two-dimensionally and / or three-dimensionally as a graphic display relative to at least one projection mapping is spaced apart. In addition, the display device can be constructed to display at least one segmented part of the planning data set, such as a bone structure and / or tissue area, in augmented reality and / or virtual reality, in particular three-dimensionally, for example, as a volume mesh model. In addition, the display device, in particular the first display unit, can be constructed to display the planning data set and / or at least one projection mapping at least partially transparently in augmented reality and / or virtual reality. In addition, the display device can be constructed to generate and display augmented reality and / or virtual reality in a fixed position relative to the user's real space environment. For this purpose, the sensor unit can be constructed to collect the positioning change of the first display unit relative to the user's real space environment. For this purpose, the sensor unit can, for example, include a gyroscope and / or an optical and / or electromagnetic sensor. Furthermore, the display device can be designed to generate augmented reality and / or virtual reality, so that both the planning data set and the at least one projection mapping can be optically recorded by a user at least partially, in particular simultaneously.

[0015] The sensor unit can advantageously be designed to capture the user input, in particular the initial spatial positioning of the input device at the starting point. In this case, the initial spatial positioning can have information about the spatial position, in particular relative to the sensor unit and / or relative to a reference point, and / or about the spatial alignment of the input device captured at the starting point. Furthermore, the captured initial spatial positioning of the user input can be registered with the planning data set, in particular the virtual spatial positioning of the gripping point. The planning data set, in particular the virtual spatial positioning of the gripping point, can have information about the virtual spatial position and / or alignment of the planning data set in the coordinate system of the augmented reality and / or virtual reality.

[0016] Analogously thereto, the sensor unit can be designed to detect a user input at an end point, in particular a final spatial position of the input device.

[0017] The display device can be designed to adjust the virtual space positioning of the planning data set, in particular the gripping point, relative to at least one projection mapping according to the user input. In this case, the virtual space positioning of the planning data set can have information about the virtual space position and / or alignment of the planning data set, in particular the gripping point, in the augmented reality and / or virtual reality. In addition, the virtual space positioning can be determined relative to a coordinate system of the display device, in particular the first display unit. In addition, the virtual space positioning can have a relative positioning relative to at least one projection mapping. The display device can be designed to reposition the gripping point from a starting point to an end point in the augmented reality and / or virtual reality. In addition, the display device can be designed to adjust and display the augmented reality and / or virtual reality during and / or after the user input.

[0018] The proposed display device enables a particularly intuitive adjustment of the augmented reality and / or virtual reality. In addition, by adjusting the virtual spatial positioning of the planning data set relative to the at least one projection mapping in the augmented reality and / or virtual reality, a simple and simultaneously precise alignment can be achieved. In particular, incorrect alignments between the planning data set and the at least one projection mapping in the augmented reality and / or virtual reality can advantageously be reduced. In this case, the user can advantageously adjust the virtual spatial positioning of the planning data set in the augmented reality and / or virtual reality in a simple and precise manner by means of user input, in particular simultaneously along a plurality of spatial degrees of freedom.

[0019] In a further advantageous embodiment of the proposed display device, the display device can also be configured to generate at least one planning projection mapping of the planning data set. In addition, the display device can be configured to additionally generate and display augmented reality and / or virtual reality based on at least one planning projection mapping. In addition, the display device can be configured to adjust at least one planning projection mapping when the virtual spatial positioning of the planning data set relative to the at least one projection mapping changes.

[0020] Advantageously, the image data set may have metadata, wherein the metadata may include information about a recording geometry and / or operating parameters of a medical imaging device used to record the image data set. Furthermore, the display device may be designed to arrange virtual sources, such as virtual X-ray sources and / or ultrasound examination sources, in an augmented reality and / or virtual reality in accordance with the corresponding recording geometry relative to at least one projection mapping based on the metadata.

[0021] The at least one planning projection mapping can in particular be a virtual projection mapping of the planning data set along a virtual projection direction from a virtual source. The at least one planning projection mapping can in particular have all features and properties already described with respect to the at least one projection mapping and vice versa.

[0022] In addition, the display device can be constructed to generate and display an augmented reality and / or virtual reality with at least one virtual auxiliary element, which is used for at least one projection mapping and / or for at least one planning projection mapping. The virtual auxiliary element can, for example, have a projection cone and / or a graphic display of a center ray of a virtual source along a corresponding (particularly virtual) projection direction. Here, the sensor unit can be constructed to collect other user inputs. In addition, the display device can be constructed to adjust the virtual projection direction to generate at least one planning projection mapping based on other user inputs. Here, the collection of other user inputs can be carried out in a manner similar to the collection of user inputs. In addition, the adjustment of the virtual projection direction for generating at least one planning projection mapping can be carried out in a manner similar to the adjustment of the virtual space positioning of the planning data set in augmented reality and / or virtual reality.

[0023] The display device can be designed to generate an augmented reality and / or virtual reality so that the planning data set is displayed in particular two-dimensionally and / or three-dimensionally as being spaced apart from the graphic display of the at least one projection mapping and the graphic display of the at least one planning projection mapping. Furthermore, the display device can be designed to generate and display the at least one projection mapping and the at least one planning projection mapping in the augmented reality and / or virtual reality at least partially transparently and / or in particular superimposed on each other.

[0024] In the event of a change in the virtual spatial positioning of the planning data set relative to at least one projection mapping, in particular relative to the virtual source, the display device can be designed to adjust, in particular regenerate, the at least one planning projection mapping. The adjustment of the at least one planning projection mapping can in particular include a virtual projection mapping of the planning data set in the changed virtual spatial positioning along a virtual projection direction from the virtual source.

[0025] By generating an augmented reality and / or a virtual reality, additionally based on at least one planning projection mapping, a particularly intuitive and at the same time precise repositioning of the planning data set relative to a particularly virtual projection direction can advantageously be achieved. Furthermore, an easily understandable comparison between at least one planning projection mapping and at least one projection mapping can be achieved.

[0026] In a further advantageous embodiment of the proposed display device, the display device can be designed to generate at least one planned projection map along a projection direction of the at least one projection map.

[0027] Here, each projection direction of at least one projection mapping can be pre-given as each virtual projection direction for generating at least one planning projection mapping. It can be achieved that the mapping geometry of the virtual projection mapping for the planning data set along at least one virtual projection direction starting from the virtual source advantageously corresponds to the recording geometry for recording at least one projection mapping. Advantageously, it is possible to achieve an intuitive repositioning of the planning data set relative to at least one projection mapping in augmented reality and / or virtual reality, so that at least one planning projection mapping can be consistent with one of the at least one projection mappings, respectively. Thus, the inspection area of ​​the inspection object mapped in the planning data set can be virtually positioned relative to the virtual source, the virtual projection direction and / or the at least one planning projection mapping so that the deviation between the mapping geometry for generating at least one planning projection mapping and the recording geometry for recording at least one projection mapping of the inspection area is minimized.

[0028] In a further advantageous embodiment of the proposed display device, the display device can be configured to generate a back projection of at least one projection mapping. Furthermore, the display device can be configured to additionally generate and display an augmented reality and / or virtual reality based on at least one back projection. Furthermore, the display device can be configured to adjust at least one back projection in the event of a change in the virtual spatial positioning of the planning data set relative to at least one projection mapping.

[0029] Advantageously, the image data set can have metadata, wherein the metadata has information about a recording geometry and / or operating parameters of a medical imaging device used to record the image data set. Furthermore, the display device can be designed to arrange virtual sources, such as virtual X-ray sources and / or ultrasound examination sources, in an augmented reality and / or virtual reality in accordance with the corresponding recording geometry relative to at least one projection mapping based on the metadata.

[0030] The at least one backprojection may include at least one virtual, in particular filtered, backprojection of a projection map onto the planning data set. In this case, the display device may be designed to generate the at least one backprojection along a projection direction of the respective projection map toward the virtual source.

[0031] The display device can be designed in particular for generating and displaying at least one backprojection and the planning data set in augmented reality and / or virtual reality at least partially transparently and / or in particular superimposed on one another. In particular, the display device can be designed for displaying at least one backprojection in augmented reality and / or virtual reality at least partially superimposed on a surface of at least a part of the planning data set, for example a bone structure and / or a tissue region, for example a segmented volume mesh model.

[0032] In the event of a change in the virtual spatial positioning of the planning data set relative to at least one projection mapping, in particular relative to the corresponding projection direction, the display device can be designed to adjust, in particular regenerate, at least one backprojection. The adjustment of at least one backprojection can include a virtual, in particular filtered, backprojection of at least one projection mapping onto the planning data set in the changed virtual spatial positioning.

[0033] By generating an augmented reality and / or a virtual reality, a particularly intuitive and simultaneously precise repositioning of the planning data set relative to the projection direction of the at least one projection mapping and / or the backprojection can advantageously be achieved, additionally based on the at least one backprojection. Furthermore, an easily understandable comparison between the at least one backprojection and the planning data set can be achieved in the augmented reality and / or the virtual reality. Thus, an examination region of the examination object mapped in the planning data set can be virtually positioned relative to the virtual source, the at least one projection direction and / or the at least one projection mapping in such a way that deviations, in particular spatial deviations, between the at least one backprojection and the planning data set are minimized.

[0034] In a further advantageous embodiment of the proposed display device, the display device can also be designed to determine, based on a user input, a transformation rule for adjusting the virtual spatial positioning of the planning data set relative to the at least one projection mapping.

[0035] The display device can be constructed to determine a transformation rule based on the user input, so that when the transformation rule is applied to the planning data set, the planning data set, in particular the gripping point, is repositioned from the starting point to the end point in the augmented reality and / or virtual reality. As long as the user input at the starting point has information about the spatial positioning of the input device, the display device can be constructed to associate the virtual spatial positioning of the planning data set, in particular the gripping point, in particular the initial virtual spatial positioning with the spatial positioning of the input device at the starting point. In addition, the display device can be constructed to determine the transformation rule based on the change of the spatial positioning of the input device acquired from the starting point to the end point of the user input. Here, the transformation rule can be constructed to adjust the virtual spatial positioning of the planning data set, in particular the gripping point, according to the acquired changes, in particular along the trajectory in the augmented reality and / or virtual reality. Here, the transformation rule can include a rigid and / or non-rigid transformation of the planning data set, in particular with respect to at least one projection mapping in the augmented reality and / or virtual reality. The display device can be particularly constructed to repeatedly determine the transformation rule during the acquisition of the user input.

[0036] As a result, a particularly precise control of the repositioning of the planning data set in augmented reality and / or virtual reality can be achieved by the user using user input.

[0037] In a further advantageous embodiment of the proposed display device, the display device can also be designed to determine at least one spatial degree of freedom of the transformation rule based on geometrical properties of the planning data set and / or the image data set.

[0038] The transformation rule can be constructed to perform rigid and / or non-rigid transformation of the planning data set in augmented reality and / or virtual reality along at least one spatial degree of freedom, in particular along multiple spatial degrees of freedom. Here, at least one spatial degree of freedom can be, for example, the virtual rotational motion and / or virtual translational motion and / or the degree of freedom of zooming of the planning data set. Advantageously, the display device can be constructed to determine at least one spatial degree of freedom of the transformation rule based on the geometric properties of the planning data set and / or the image data set. Here, the geometric properties can, for example, have information about the recorded geometry of the planning data set and / or the image data set. In addition, the geometric properties can include information about the positioning of the inspection area of ​​the inspection object relative to the medical imaging device for recording the planning data set and / or the image data, in particular relative to the source and / or the detector. In addition, the geometric information can include information about the spatial construction of the planning data set and / or the image data, such as extension information and / or alignment information and / or positioning information and / or depth information. Advantageously, by determining at least one spatial degree of freedom of the transformation rule based on geometrical properties of the planning data set and / or the image data set, in particular undesirable and / or non-physical transformations of the planning data set in the augmented reality and / or virtual reality can be prevented. For example, a virtual rotational movement of the planning data set can be allowed, in particular arbitrarily, while a virtual translational movement of the planning data set is allowed only in a predefined spatial area and / or along a predefined spatial direction within the augmented reality and / or virtual reality. In particular, a virtual movement of the planning data set relative to the at least one projection mapping can be limited by means of at least one degree of freedom of the transformation rule, so that a mutual movement (English: out-of-plane) of the planning data set and the at least one projection mapping is prevented.

[0039] In a further advantageous embodiment of the proposed display device, the sensor unit may include an optical and / or tactile and / or electromagnetic and / or acoustic sensor and / or a gaze direction acquisition unit (English: eye-tracker). In this case, the sensor unit may be designed to acquire user inputs, to acquire the position and / or movement of at least one body part of the user and / or the user's voice input and / or the user's gaze direction.

[0040] Here, the optical sensor may, for example, include a camera, in particular a monocular camera and / or a stereo camera and / or a depth camera. Advantageously, the optical sensor may be constructed to capture the spatial positioning and / or movement of at least one body part of the user, in particular relative to the display device. In addition, the sensor unit may be constructed to capture the user's, in particular the input device's, in particular spatially resolved and / or time-resolved input gestures as user input. In addition, the tactile sensor may, for example, include a control element, such as a joystick and / or a pointing device and / or a tactile controller and / or a tactile glove, which can be operated with the help of the user's body part. The electromagnetic sensor may, for example, be constructed to capture the movement of the user's body part and / or the input device according to the change of the electromagnetic field around the display device. The acoustic sensor may, for example, include at least one microphone, which is constructed to capture the user's voice input, such as a voice command and / or an acoustic signal. In addition, the sensor unit may have a sight direction acquisition unit, which is constructed to capture the user's sight direction relative to the display of augmented reality and / or virtual reality. In particular, the sensor unit may be at least partially integrated into the display device, in particular integrated into the first display unit.

[0041] The sensor unit can in particular have a plurality of different sensors for capturing user inputs. For example, at least one of the sensors can be designed for capturing user inputs for determining and / or selecting a gripping point. In addition, at least one further sensor can be designed for capturing user inputs for adjusting the virtual spatial positioning of the planning data set. This allows a particularly intuitive and flexible operation of the display device by the user. In particular, in a surgical environment, a high degree of flexibility in the way in which the user inputs are captured by means of the sensor unit can be advantageous.

[0042] In a further advantageous embodiment of the proposed display device, the planning data set and / or the image data set can have predetermined image information. Here, the display device can be designed to additionally generate and display augmented reality and / or virtual reality based on the predetermined image information.

[0043] The display device can be designed in particular for receiving and / or generating predetermined image information, in particular based on a planning data set and / or an image data set. The predetermined image information can include in particular preprocessed image information, such as image gradients and / or edge images; and / or probability maps with probability information about the presence of specific structures. Alternatively or additionally, the predetermined image information can include a segmentation (in particular a two-dimensional segmentation) of landmarks (in particular anatomical landmarks) and / or of structures (for example blood vessels and / or bone structures and / or organs and / or tissue regions) and / or of medical objects (for example catheters and / or guidewires and / or endoscopes).

[0044] As long as the planning data set has predetermined image information, the display device can be constructed to generate and display augmented reality and / or virtual reality based on the predetermined image information of the planning data set, so that the predetermined image information can be collected for the user. For this reason, the display device can be constructed to display the predetermined image information of the planning data set (for example, as an overlay) on a particularly two-dimensional and / or three-dimensional surface of the planning data set in an augmented reality and / or virtual reality, for example, in a graphical manner. Alternatively or additionally, the display device can be constructed to map the predetermined image information of the planning data set similarly to the planning projection mapping. In particular, the display device can be constructed to display the predetermined image information of the planning data set as a part of the planning projection mapping.

[0045] As long as the image data set has predetermined image information, the display device can be designed to generate and display an augmented reality and / or virtual reality additionally based on the predetermined image information of the image data set, so that the predetermined image information of the image data set can be acquired by the user. To this end, the display device can be designed to graphically display the predetermined image information of the image data set in the augmented reality and / or virtual reality, for example as an overlay with at least one projection mapping. Alternatively or additionally, the display device can be designed to map the predetermined image information of the image data set similarly to a backprojection. In particular, the display device can be designed to display the predetermined image information of the image data set in the augmented reality and / or virtual reality as part of a backprojection on the planning data set.

[0046] Incorporating predetermined image information of the planning data set and / or image data set into the generation and / or display of augmented reality and / or virtual reality can provide a visual aid for the user to adjust the virtual spatial positioning of the planning data set. This allows for a fast and simultaneously precise detection of deviations by the user in augmented reality and / or virtual reality, for example between the planning data set and the backprojection and / or between at least one planning projection mapping and at least one projection mapping.

[0047] In a further advantageous embodiment of the proposed display device, the display device can also be designed to select predetermined image information based on a user input, in particular from a plurality of received predetermined image information, for example by displaying a preview and / or a menu in augmented reality and / or virtual reality. Alternatively or additionally, the display device can be designed to generate predetermined image information based on user input. For this purpose, the user can, for example, determine image points and / or mappings of anatomical structures and / or geometric structures and / or medical objects in augmented reality and / or virtual reality, in particular in the planning data set and / or at least one projection mapping, by means of user input.

[0048] The proposed specific embodiment enables a particularly user-friendly display of aids for adjusting the virtual spatial positioning of planning data sets in augmented reality and / or virtual reality.

[0049] In a further advantageous embodiment of the proposed display device, the display device may have a first display unit and a second display unit. In this case, the first display unit may be designed to display an augmented reality and / or a virtual reality. In addition, the second display unit may be designed to display at least one projection mapping and / or at least one planned projection mapping. In addition, the sensor unit may be designed to acquire a spatial relative positioning of the first display unit and the second display unit relative to each other. In this case, the display device may be designed to adjust the augmented reality and / or the virtual reality based on the acquired relative positioning.

[0050] The spatial relative positioning of the first display unit and the second display unit to each other may include information about the spatial position and / or alignment of the first display unit and the second display unit relative to each other.

[0051] The second display unit may, for example, include a monitor and / or a display. Here, the sensor unit may be constructed to at least partially capture the spatial relative positioning of the second display unit, in particular the second display unit relative to the first display unit. As long as the sensor unit is arranged to be at least partially integrated into the first display unit, the acquisition of the second display unit may be sufficient to determine the spatial relative positioning. Alternatively, the sensor unit may be constructed to capture the first display unit and the second display unit, in particular relative to each other and / or absolutely relative to a spatial reference point. The first display unit and / or the second display unit may, for example, have a defined shape and / or marking structure. Alternatively or additionally, the sensor unit may be constructed to capture the display of at least one projection mapping and / or at least one planned projection mapping on the second display unit. In addition, the first display unit and / or the second display unit may be constructed to provide a signal to the sensor unit, the signal including information about spatial positioning. To this end, the first display unit and / or the second display unit may have a positioning sensor, such as a gyroscope and / or an optical and / or electromagnetic sensor.

[0052] Advantageously, the display device can be configured to adjust the augmented reality and / or virtual reality based on the acquired relative positioning, so that the display of the at least one projection mapping and / or the at least one planned projection mapping on the second display unit is at least partially embedded in and / or integrated into the augmented reality and / or virtual reality. The display device, in particular the first display unit, can be configured to display the augmented reality and / or virtual reality at least partially superimposed on the display on the second display unit.

[0053] This makes it possible to integrate augmented reality and / or virtual reality particularly easily into an existing working environment of a user with a second display unit. The proposed display device can thereby comprehensibly display additional information to the user in the form of an at least partially superimposed display of the first display unit and the second display unit.

[0054] In a second aspect, the invention relates to a system comprising a display device according to the invention and a medical imaging device. The medical imaging device is designed to record and / or to provide and / or to receive planning data sets and / or image data sets.

[0055] The advantages of the proposed system substantially correspond to the advantages of the proposed display device for displaying augmented reality and / or virtual reality. The features, advantages or alternative embodiments mentioned here can also be transferred to other claimed contents, and vice versa.

[0056] The medical imaging device can be designed, for example, as a medical X-ray device, in particular a medical C-arm X-ray device and / or a computed tomography device (CT) and / or a magnetic resonance device (MRI) and / or a positron emission tomography device (PET) and / or an ultrasound examination device. In this case, the medical imaging device can also be designed for recording and / or for receiving and / or for providing planning data sets and / or image data sets.

[0057] The medical imaging device can in particular have an interface which is designed to provide and / or receive medical image data sets and / or medical planning data sets, in particular to a display device. The interface can in particular be designed to acquire and / or read a computer-readable data memory and / or to receive from a data storage unit, for example a database.

[0058] In a third aspect, the invention relates to a method for registering a planning data set with an image data set. In this case, in a first step a), a planning data set of an object to be examined is received. Furthermore, in a second step b), an image data set with at least one projection mapping of the object to be examined is received. In a third step c), an augmented reality and / or virtual reality is generated based on the planning data set and at least one projection mapping. In a fourth step d), a graphical display of the augmented reality and / or virtual reality is displayed with the aid of a display device having a sensor unit. In a fifth step e), user inputs are acquired with the aid of the sensor unit. Furthermore, in a sixth step f), the virtual spatial positioning of the planning data set relative to at least one projection mapping is adjusted in the augmented reality and / or virtual reality based on the user input.

[0059] The advantages of the proposed method correspond substantially to the advantages of the proposed display device and / or the proposed system for displaying augmented reality and / or virtual reality. The features, advantages or alternative embodiments mentioned here can also be transferred to other claimed contents, and vice versa.

[0060] The previously described steps a) to f) of the proposed method can advantageously be carried out sequentially and / or at least partially simultaneously. Advantageously, the steps a) to f) of the proposed method can be carried out with the aid of the proposed display device. In particular, the display of the graphical display of the augmented reality and / or virtual reality can be carried out in step d) with the aid of the proposed display device, in particular the first display unit.

[0061] Advantageously, by adjusting the virtual spatial positioning of the planning data set relative to the at least one projection mapping in the augmented reality and / or virtual reality in step f), the registration between the planning data set and the image data set, in particular the at least one projection mapping, can be improved. Advantageously, the planning data set in the augmented reality and / or virtual reality can be positioned based on the user input so that the virtual spatial positioning of the examination region mapped therein relative to the at least one projection mapping is highly consistent with the spatial positioning of the examination region relative to the at least one projection mapping at the time of recording the at least one projection mapping. Thereby, a particularly coordinated and precise display of the planning data set and the at least one projection mapping in the augmented reality and / or virtual reality can be advantageously achieved.

[0062] In a further advantageous embodiment of the proposed method, steps d) to f) can be repeatedly performed until a termination condition occurs. The termination condition can, for example, comprise a maximum number of repetitions. Alternatively or additionally, the termination condition can be triggered by a predefined user input, for example a predefined gesture of the user.

[0063] This makes it possible to adjust, in particular iteratively adjust, the virtual relative positioning of the planning data set with respect to the at least one projection mapping in augmented reality and / or virtual reality.

[0064] In a further advantageous embodiment of the proposed method, step c) can furthermore comprise the application of a transformation rule for registering the planning data set with the image data set.

[0065] Here, the transformation rule can be determined based on the user input so that when the transformation rule is applied to the planning data set, the planning data set, in particular the gripping point, is repositioned from the starting point to the end point in the augmented reality and / or virtual reality. As long as the user input at the starting point has information about the spatial positioning of the input device, the virtual spatial positioning of the planning data set, in particular the initial virtual spatial positioning of the gripping point, can be associated with the spatial positioning of the input device at the starting point. In addition, the transformation rule can be determined based on the changes in the spatial positioning of the input device acquired from the starting point to the end point of the user input. Here, the transformation rule can be constructed to adjust the virtual spatial positioning of the planning data set, in particular the gripping point, in the augmented reality and / or virtual reality according to the acquired changes, in particular along the trajectory. Here, the transformation rule can include a rigid and / or non-rigid transformation of the planning data set, in particular with respect to at least one projection mapping in the augmented reality and / or virtual reality. The transformation rule can be determined during the acquisition of the user input in step e), in particular repeatedly.

[0066] This makes it possible to achieve particularly precise control of the repositioning of the planning data set in augmented reality and / or virtual reality by the user using user input.

[0067] In a further advantageous embodiment of the proposed method, step c) can also include generating at least one planning projection mapping of the planning data set along the projection direction of the at least one projection mapping. In this case, an augmented reality and / or virtual reality can be generated additionally based on the at least one planning projection mapping. Furthermore, step f) can include adjusting the at least one planning projection mapping in the event of a change in the virtual spatial positioning of the planning data set relative to the at least one projection mapping.

[0068] Analogously to one of the previously described embodiments of the proposed method, the image data set can have metadata, wherein based on the metadata, a virtual source can be arranged in augmented reality and / or virtual reality relative to the at least one projection mapping according to the corresponding recorded geometry. The at least one planning projection mapping can in particular be a virtual projection mapping of the planning data set along a virtual projection direction from the virtual source. The at least one planning projection mapping can in particular have all the features and characteristics already described with respect to the at least one projection mapping, and vice versa.

[0069] Augmented reality and / or virtual reality can advantageously be generated and displayed in step c) and d) so that it has at least one virtual auxiliary element for at least one projection mapping and / or for at least one planning projection mapping. The virtual auxiliary element can, for example, have a projection cone and / or a graphic display of a center ray of a virtual source along a corresponding (particularly virtual and / or real) projection direction. Here, the sensor unit can be constructed to be used to collect other user inputs. In addition, the virtual projection direction for generating at least one planning projection mapping can be adjusted based on other user inputs. Here, the collection of other user inputs can be carried out in a manner similar to the collection of user inputs. In addition, the adjustment of the virtual projection direction for generating at least one planning projection mapping can be carried out in a manner similar to the adjustment of the virtual spatial positioning of the planning data set in augmented reality and / or virtual reality.

[0070] The augmented reality and / or virtual reality can advantageously be generated such that the planning data set is displayed in particular two-dimensionally and / or three-dimensionally as being spaced apart from the graphic display of the at least one projection mapping and the graphic display of the at least one planning projection mapping. Furthermore, the at least one projection mapping and the at least one planning projection mapping can be displayed in the augmented reality and / or virtual reality at least partially transparently and / or in particular superimposed on one another.

[0071] In the event of a change in the virtual spatial positioning of the planning data set relative to at least one projection mapping, in particular relative to the virtual source, the at least one planning projection mapping can be adjusted, in particular regenerated. The adjustment of the at least one planning projection mapping can in particular include a virtual projection mapping of the planning data set in the changed virtual spatial positioning along a virtual projection direction from the virtual source.

[0072] By generating an augmented reality and / or a virtual reality, additionally based on at least one planning projection mapping, a particularly intuitive and at the same time precise repositioning of the planning data set relative to a particularly virtual projection direction can advantageously be achieved. In addition, an easily understandable comparison between at least one planning projection mapping and at least one projection mapping can be achieved.

[0073] In a further advantageous embodiment of the proposed method, the termination condition can include a comparison between the degree of agreement and a predefined threshold value. In this case, the degree of agreement can evaluate the agreement between the at least one planned projection mapping and the at least one projection mapping.

[0074] The threshold value may be input and / or predetermined by a user, for example.

[0075] The degree of consistency can advantageously include information about the consistency between at least one planning projection mapping and at least a part of the planning data set, in particular along the real and virtual projection directions corresponding to each other, in particular the consistency of each image point. The degree of consistency can be determined in particular based on a consistency measure. In addition, the degree of consistency can be determined, for example, based on a comparison of image values ​​and / or geometric features and / or anatomical features between at least one planning projection mapping and at least one projection mapping. Here, if the planning data set is virtually rearranged relative to at least one projection mapping and / or relative to the corresponding projection direction, substantially similar to the mapping geometry of the examination area, in particular at the time point of recording at least one projection mapping, a high degree of consistency can be achieved between at least one planning projection mapping and at least one projection mapping. Here, the degree of consistency can also be determined so that the degree of consistency has in particular different weightings and / or scaling along the degrees of freedom of the transformation rule. Thus, a particularly accurate registration of planning data with an image data set, in particular with at least one projection mapping, can be achieved.

[0076] In a further advantageous embodiment of the proposed method, step c) can also include generating a back projection of the at least one projection mapping. In this case, an augmented reality and / or a virtual reality can be generated additionally based on the at least one back projection. Furthermore, step f) can include adjusting the at least one back projection in the event of a change in the virtual spatial positioning of the planning data set relative to the at least one projection mapping.

[0077] Advantageously, the image data set can have metadata, wherein the metadata has information about the recording geometry and / or operating parameters of the medical imaging device used to record the image data set. In addition, based on the metadata, a virtual source, for example a virtual X-ray source and / or an ultrasound examination source, can be arranged in augmented reality and / or virtual reality in accordance with the corresponding recording geometry relative to at least one projection mapping.

[0078] The at least one backprojection can advantageously include at least one virtual, in particular filtered, backprojection of the projection mapping onto the planning data set. In this case, the at least one backprojection can be generated along the projection direction of the corresponding projection mapping toward the virtual source. The augmented reality and / or virtual reality can advantageously be generated such that the at least one backprojection and the planning data set are displayed at least partially transparently and / or in particular superimposed on one another in the augmented reality and / or virtual reality. The at least one backprojection can be displayed at least partially superimposed on the surface of at least a portion of the planning data set, for example a bone structure and / or a tissue region, for example on a segmented volume mesh model in the augmented reality and / or virtual reality.

[0079] In the event of a change in the virtual spatial positioning of the planning data set relative to at least one projection mapping, in particular relative to the corresponding projection direction, at least one backprojection can be adjusted, in particular regenerated. The adjustment of at least one backprojection can advantageously include a virtual, in particular filtered, backprojection of at least one projection mapping onto the planning data set in the changed virtual spatial positioning.

[0080] By generating an augmented reality and / or a virtual reality, a particularly intuitive and simultaneously precise repositioning of the planning data set relative to the projection direction of the at least one projection mapping and / or the backprojection can advantageously be achieved, additionally based on the at least one backprojection. Furthermore, an easily understandable comparison between the at least one backprojection and the planning data set can be achieved in the augmented reality and / or the virtual reality. Thus, an examination region of the examination object mapped in the planning data set can be virtually positioned relative to the virtual source, the at least one projection direction and / or the at least one projection mapping in such a way that deviations, in particular spatial deviations, between the at least one backprojection and the planning data set are minimized.

[0081] In a further advantageous embodiment of the proposed method, the termination condition can include a comparison between the further degree of agreement and a further predetermined threshold value. The further degree of agreement can evaluate the agreement between the at least one backprojection and the planning data set.

[0082] Further threshold values ​​may be input and / or predefined, for example, by a user.

[0083] The further degree of consistency can advantageously include information about the consistency between at least one backprojection and at least a portion of the planning data set, in particular the portion of the planning data set on which the backprojection is mapped. The further degree of consistency can be determined in particular based on a consistency measure, in particular image point by image point. In addition, the further degree of consistency can be determined, for example, based on a comparison of image values ​​and / or geometrical features and / or anatomical features between the planning data set and at least one backprojection. Here, if the planning data set is virtually rearranged relative to at least one backprojection, in particular relative to at least one projection mapping and / or relative to the corresponding projection direction, substantially similar to the mapping geometry of the examination region, in particular at the time point of recording at least one projection mapping, a high degree of consistency can be achieved between the planning data set and at least one backprojection. Here, the further degree of consistency can also be determined so that the further degree of consistency has in particular different weightings and / or scalings along the degrees of freedom of the transformation rule. Thus, a particularly precise registration of the planning data set with the image data set, in particular with at least one projection mapping, can be achieved.

[0084] In a fourth aspect, the invention relates to a computer program product having a computer program that can be directly loaded into a memory of a provision unit, the computer program product having a program section in order to carry out all the steps of the proposed method for registering a planning data set with an image data set when the program section is executed by the provision unit. In this case, the computer program product can include software with source code, which still has to be compiled and linked or has to be interpreted, or can include executable software code, which is only loaded into the provision unit for execution. With the computer program product, the method for registering a planning data set with an image data set can be carried out quickly, reproducibly and robustly with the aid of the provision unit. The computer program product is configured such that it can carry out the method steps according to the invention with the aid of the provision unit.

[0085] The computer program product is stored, for example, on a computer-readable storage medium, or on a network or server, from which the computer program product can be loaded into a processor of a providing unit, which can be directly connected to the providing unit or constructed as a part of the providing unit. In addition, the control information of the computer program product can be stored on an electronically readable data carrier. The control information of the electronically readable data carrier can be designed so that when the data carrier is used in the providing unit, the control information executes the method according to the present invention. Examples of electronically readable data carriers are DVDs, magnetic tapes or USB sticks, on which electronically readable control information, in particular software, is stored. If these control information are read from the data carrier and stored in the providing unit, all embodiments of the method described above according to the present invention can be executed.

[0086] Furthermore, the present invention may also relate to a computer-readable storage medium and / or an electronically readable data carrier on which program sections are stored which can be read and executed by a provision unit in order to carry out all steps of the method for registering a planning data set with an image data set when the program sections are executed by the provision unit.

[0087] A largely software-based implementation has the advantage that previously already used provision units can also be easily retrofitted by software updates to operate in the manner according to the invention. In addition to the computer program, such a computer program product may include additional components, such as documentation and / or additional components, as well as hardware components, such as a hardware key (dongle, etc.) for using the software. BRIEF DESCRIPTION OF THE DRAWINGS

[0088] Embodiments of the invention are shown in the drawings and are described in more detail below. In the different drawings, the same reference numerals are used for the same features. In the drawings:

[0089] Figure 1 A schematic diagram of the proposed system is shown, which has the proposed display device and a medical C-arm X-ray device;

[0090] Figures 2 to 5 Schematic diagrams showing different implementations of augmented reality and / or virtual reality;

[0091] Figures 6 to 8 Schematic diagrams of different specific embodiments of the proposed method for registering a planning data set with an image data set are shown. DETAILED DESCRIPTION

[0092] Figure 1 The proposed system is schematically shown in . Here, the system may include the proposed display device D and a medical imaging device. In addition, the medical imaging device may be designed as a medical C-arm X-ray device 37, for example. Here, the display device D may have a provision unit PRVS.D, which is designed to generate and / or adjust augmented reality and / or virtual reality. In addition, the medical C-arm X-ray device 37 may include a further provision unit PRVS.37. Here, the display device D, in particular the provision unit PRVS.D, may be designed to implement the proposed method for registering a planning data set with an image data set.

[0093] Furthermore, the medical C-arm X-ray device 37 can have a detector unit 34 and an X-ray source 33, which can be movably mounted on an arm 38 of the C-arm X-ray device 37 around one or more axes. Furthermore, the medical C-arm X-ray device 37 can include a movement device 39, which enables the C-arm X-ray device 37 to be moved in space. In order to record a medical image data set and / or a planning data set with at least one projection mapping from an examination area UB of an examination object 31 arranged on a patient positioning device 32, a further provision unit PRVS.37 can send a signal 24 to the X-ray source 33. Accordingly, the X-ray source 33 can emit an X-ray beam, in particular a cone beam and / or a fan beam and / or a parallel beam. After interacting with the examination area to be imaged of the examination object 31, when the X-ray beam impinges on the surface of the detector unit 34, the detector unit 34 can send a signal 21 to the further provision unit PRVS.37. The further provision unit PRVS.37 can, for example, receive the image data set and / or the planning data set based on the signal 21. Furthermore, the further providing unit PRVS.37 can provide image data sets and / or planning data sets for generating augmented reality and / or virtual reality to the display device D, in particular the providing unit PRVS.D. For this purpose, the further providing unit PRVS.37 can send corresponding signals 27, in particular wirelessly, to the display device D, in particular the providing unit PRVS.D.

[0094] Advantageously, the display device D can have a sensor unit S, which is constructed to collect user inputs, in particular user inputs of an input device IM. The input device IM can, for example, include a pointing device and / or an input unit and / or a body part of a user and / or an optical and / or acoustic signal. In addition, the display device D can be constructed to receive a medical image data set and a medical planning data set with at least one projection mapping, in particular by means of a signal 27. Here, the providing unit PRVS.D can be constructed to process user inputs, in particular signals from the sensor unit S, and / or a planning data set and / or an image data set. In addition, the display device D can be constructed to generate and display augmented reality and / or virtual reality. In addition, the display device D is constructed to adjust the virtual space positioning of the planning data set relative to at least one projection mapping in augmented reality and / or virtual reality based on user input.

[0095] The display device D can advantageously include a portable, in particular wearable, first display unit by the user U, which is constructed to display augmented reality and / or virtual reality (English: AR and / or VR for short). Here, the first display unit can be constructed to be at least partially transparent and / or capable of being transparent. Advantageously, the first display unit can be constructed so that it can be at least partially worn by the user U within the field of vision of the user U. To this end, the first display unit can advantageously be constructed as glasses (in particular data glasses) and / or a helmet (in particular a data helmet) and / or a screen.

[0096] Furthermore, the first display unit may be constructed to superimpose real objects (e.g. physical objects, in particular medical objects and / or examination objects 31) with virtual data (in particular measured and / or simulated and / or processed medical image data and / or virtual objects) and display them in a display.

[0097] In addition, the sensor unit S may include an optical and / or tactile and / or electromagnetic and / or acoustic sensor and / or a sight direction acquisition unit. Here, the sensor unit S may be configured to acquire user input to acquire the position and / or movement of at least one body part of the user U and / or the voice input of the user U and / or the sight direction of the user U.

[0098] In addition, the medical C-arm X-ray device 37 may include an input unit 42 (e.g., a keyboard) and / or a second display unit 41 (e.g., a monitor and / or a display). For example, in the case of a capacitive input display, the input unit 42 may preferably be integrated in the second display unit 41. Here, the control of the medical C-arm X-ray device 37 can be achieved through the input of the user U to the input unit 42. To this end, the input unit 42 may, for example, send a signal 26 to the other providing unit PRVS.37.

[0099] Furthermore, the second display unit 41 can be designed to display information and / or graphic displays of the medical C-arm X-ray device 37 and / or the provision unit PRVS.D and / or the further provision unit PRVS.37 and / or the further components. Furthermore, the second display unit 41 can be designed to display at least one projection mapping. For this purpose, the provision unit PRVS.37 can, for example, send a signal 25 to the second display unit 41.

[0100] Furthermore, the sensor unit S may be configured to detect a spatial relative position of the first display unit and the second display unit 41. Here, the display device D may be configured to adjust the augmented reality and / or virtual reality based on the detected relative position.

[0101] Figure 2Schematically shows an exemplary construction of augmented reality and / or virtual reality, in particular from the perspective of user U. Here, the hand of user U can form input device IM. In addition, the user input can have a trajectory, in particular a trajectory between a spatial starting point and a spatial end point. The display device D can be constructed to associate the spatial starting point with a gripping point GP of the planning data set DS, in particular at a corner and / or edge and / or contour and / or image point. In addition, the display device D can be constructed to generate and display augmented reality and / or virtual reality based on the planning data set DS and the projection mappings PI1 and PI2. In particular, the display device D can be constructed to generate augmented reality and / or virtual reality so that the planning data set DS is displayed as being spaced apart from the graphic display of the projection mappings PI1 and PI2. In addition, the display device D can be constructed to display at least one segmented part of the planning data set DS, such as a bone structure and / or tissue area, in augmented reality and / or virtual reality, in particular three-dimensionally, for example as a volume mesh model. Furthermore, the display device D can be constructed for generating an augmented reality and / or a virtual reality, so that both the planning data set DS and the projection maps PI1 and PI2 can be optically acquired by the user U at least partially, in particular simultaneously. In this case, the planning data set DS can map the examination area UB before the procedure. Furthermore, the image data set, in particular the projection maps PI1 and PI2, can map the examination area UB during the procedure, wherein the medical object MD is arranged in the examination area UB. Thus, the projection maps PI1 and PI2 can each have a projection mapping of the examination area UB along different projection directions PD1 and PD2, in particular with the medical object MD arranged in the examination area during the procedure.

[0102] Furthermore, the planning data set DS and / or the image data set may have predetermined image information. In this case, the display device D may be constructed for generating and displaying augmented reality and / or virtual reality additionally based on the predetermined image information. The predetermined image information may include, in particular, preprocessed image information, such as image gradients and / or edge images, and / or probability maps with probability information about the presence of specific structures. Alternatively or additionally, the predetermined image information may include a segmentation (in particular, a two-dimensional segmentation) of landmarks (in particular anatomical landmarks) and / or of structures (for example blood vessels and / or bone structures and / or organs and / or tissue regions) and / or of medical objects (for example catheters and / or guidewires and / or endoscopes). By way of example, for the predetermined image information, Figure 2 Schematically depicts a segmented bone structure, in particular a skull, in augmented reality and / or virtual reality based on a planning data set DS.

[0103] The display device D may also be configured to select and / or generate predetermined image information based on user input.

[0104] Advantageously, the image data set can have metadata, wherein the metadata has information about the recording geometry and / or operating parameters of the medical imaging device used to record the image data set. In addition, the display device D can be designed to arrange the virtual source Q1 or Q2, for example a virtual X-ray source, in each case in an augmented reality and / or virtual reality in relation to the projection mappings PI1 and PI2 according to the corresponding recording geometry based on the metadata.

[0105] Furthermore, the display device D can be designed for generating and displaying an augmented reality and / or virtual reality with at least one virtual auxiliary element for the projection mappings PI1 and PI2. The virtual auxiliary element can, for example, have a graphic display of individual projection cones and / or individual center rays of the virtual sources Q1 and Q2 along the corresponding (in particular virtual) projection directions PD1 and PD2.

[0106] exist Figure 3 and Figure 4 Schematically shown in the figure are exemplary virtual space arrangements of a planning data set DS and projection mappings PI1 and PI2 in augmented reality and / or virtual reality, respectively. Here, a display device D may be constructed to generate planning projection mappings PPI1 and PPI2 of a planning data set DS, in particular planning projection mappings PPI1 and PPI2 of a planning data set DS along projection directions PD1 and PD2 of projection mappings PI1 and PI2. Furthermore, the display device D may be constructed to additionally generate and display augmented reality and / or virtual reality based on the planning projection mappings PPI1 and PPI2. Furthermore, the display device D may be constructed to adjust the planning projection mappings PPI1 and PPI2 when the virtual space positioning of the planning data set DS relative to the projection mappings PI1 and PI2 changes.

[0107] In other words, one of the projection directions PD1 and PD2 of the projection mappings PI1 and PI2 can be pre-given as a virtual projection direction for generating the planning projection mappings PPI1 and PPI2, respectively. It can thereby be achieved that the mapping geometry of the virtual projection mappings for the planning data set DS along the virtual projection directions starting from the virtual sources Q1 and Q2 advantageously corresponds to the recording geometry for recording the projection mappings PI1 and PI2. It is thereby advantageously possible to achieve an intuitive repositioning of the planning data set DS relative to the projection mappings PI1 and PI2 in augmented reality and / or virtual reality, so that the planning projection mappings PPI1 and PPI2 can be respectively consistent with one of the projection mappings PI1 and PI2, in particular with the reference segments RI1 and RI2 within the projection mappings PI1 and PI2. Thus, the examination area UB of the examination object 31 mapped in the planning data set DS can be virtually positioned relative to the virtual sources Q1 and Q2, the virtual projection directions and / or the planning projection mappings PPI1 and PPI2 so that the deviation between the mapping geometry used to generate the planning projection mappings PPI1 and PPI2 and the recording geometry used to record the projection mappings PI1 and PI2 is minimized.

[0108] Figure 4 An exemplary virtual repositioning of a planning data set in augmented reality and / or virtual reality is schematically shown in FIG. Here, the sensor unit S can advantageously be constructed to capture an initial spatial positioning of a user input UI, in particular an input device IM, at a starting point IP, in particular a starting point of a trajectory T. Similarly, the sensor unit S can be constructed to capture a final spatial positioning of a user input UI, in particular an input device IM, at an end point TP.

[0109] The display device D can be constructed to determine a transformation rule for adjusting the virtual spatial positioning of the planning data set DS relative to the projection mappings PI1 and PI2 based on the user input UI. In addition, the display device D can be constructed to determine at least one spatial degree of freedom of the transformation rule based on geometrical properties of the planning data set DS and / or the image data set.

[0110] The display device D can be configured to adjust the planning data set DS, in particular the virtual space positioning of the gripping point GP relative to the projection mappings PI1 and PI2, according to the user input UI. The display device D can be configured to reposition the gripping point GP from the starting point IP to the end point TP in augmented reality and / or virtual reality. In addition, the display device D can be configured to adjust and display augmented reality and / or virtual reality during and / or after the user input UI.

[0111] Figure 5A further exemplary configuration of an augmented reality and / or virtual reality is shown, in particular from the perspective of a user U. In this case, the display device D can be configured to generate a back projection RP1 of the projection mapping PI1. Furthermore, the display device D can be configured to additionally generate and display an augmented reality and / or virtual reality based on the back projection RP1.

[0112] The backprojection RP1 may, for example, comprise a virtual, particularly filtered backprojection of the projection mapping PI1 onto the planning data set DS. In this case, the display device D may be designed to generate the backprojection RP1 along the projection direction of the corresponding projection mapping toward the virtual source. The display device D may be designed in particular to generate and display the backprojection RP1 and the planning data set DS at least partially transparently and / or in particular superimposed on one another in augmented reality and / or virtual reality. Figure 5 In the embodiment schematically shown in FIG. 1 , for example, the medical object MD mapped in the projection map PI1 can be back-projected onto a map of the blood vessel structure (mapped in the planning data set DS and / or segmented) and displayed.

[0113] Furthermore, the display device D can be designed to adapt the back projection RP1 in the event of a change in the virtual spatial positioning of the planning data set DS relative to the projection mapping PI1 .

[0114] Figure 6 An advantageous implementation of the proposed method for registering a planning data set DS with an image data set ID is schematically shown in FIG. Here, in a first step a), a REC-DS planning data set DS can be received. In addition, in a second step b), an image data set ID having at least one projection mapping PI in REC-ID can be received. Thereafter, in a third step c), a GEN-AR augmented reality and / or virtual reality AR can be generated based on the planning data set DS and at least one projection mapping PI. In a fourth step d), a graphical display of the VISU-AR augmented reality and / or virtual reality AR can be displayed with the aid of a display device D, in particular with the aid of a first display unit. In a fifth step e), a CAP-UI user input UI can be acquired with the aid of a sensor unit S. In addition, in a sixth step f), the virtual spatial positioning of the ADJ-AR planning data set DS relative to at least one projection mapping PI can be adjusted in the augmented reality and / or virtual reality AR based on the user input UI. Here, steps d) to f) can be repeatedly implemented until a termination condition A occurs.

[0115] Furthermore, step c) can comprise the application of a transformation rule in order to register the planning data set DS with the image data set ID.

[0116] Figure 7A further advantageous embodiment of the proposed method for registering a planning data set DS with an image data set ID is schematically shown in . Here, step c) can also include generating a back projection of at least one projection mapping PI of GEN-RP. In addition, augmented reality and / or virtual reality AR can be additionally generated based on at least one back projection. In addition, step f) can include adjusting at least one back projection of ADJ-RP in the event of a change in the virtual spatial positioning of the planning data set DS relative to at least one projection mapping PI. Advantageously, the termination condition A can include a comparison between a further degree of consistency and a predefined further threshold value, wherein the further degree of consistency evaluates the consistency between at least one back projection and the planning data set DS.

[0117] Figure 8 A further advantageous embodiment of the proposed method for registering a planning data set DS with an image data set ID is shown. Here, step c) can also include generating GEN-PPI at least one planning projection mapping. In addition, augmented reality and / or virtual reality AR can be additionally generated based on at least one planning projection mapping. In addition, step f) can include adjusting ADJ-PPI at least one planning projection mapping in the event of a change in the virtual spatial positioning of the planning data set DS relative to at least one projection mapping PI. Advantageously, the termination condition A can include a comparison between a degree of consistency and a predetermined threshold value, wherein the degree of consistency evaluates the consistency between at least one planning projection mapping and at least one projection mapping DS.

[0118] The schematic illustrations contained in the described figures do not depict any scale or size ratios.

[0119] Finally, it is also pointed out again that the method described in detail above and the device shown are only embodiments, which can be modified by a person skilled in the art in very different ways without departing from the scope of the invention. In addition, the use of the indefinite article "a" or "an" does not exclude that the features involved can also be present multiple times. Similarly, the terms "unit" and "element" do not exclude that the components involved are composed of multiple cooperating sub-components, which sub-components can also be spatially distributed if necessary.

Claims

1. A display device (D) for displaying augmented reality and / or virtual reality (AR), in, The display device (D) has a sensor unit (S) which is designed to detect (CAP-UI) user inputs (UI), wherein the display device (D) is designed to receive (REC-ID, REC-DS) a medical image data set (ID) and a medical planning data set (DS), the medical image data set having at least one projection mapping (PI), wherein the display device (D) is designed to generate (GEN-AR) and display (VISU-AR) an augmented reality and / or a virtual reality (AR) based on the planning data set (DS) and the at least one projection mapping (PI), wherein the display device (D) is further configured to adjust (ADJ-AR) a virtual spatial positioning of the planning data set (DS) relative to at least one projection mapping (PI) in augmented reality and / or virtual reality (AR) based on a user input (UI), In which, the display device (D) is also constructed to determine, based on user input (UI), a transformation rule for adjusting the virtual space positioning of the planning data group (DS) relative to at least one projection mapping (PI), so that when the transformation rule is applied to the planning data group (DS), the planning data group (DS) can be repositioned from a starting point to an end point.

2. The display device (D) according to claim 1, characterized in that The display device (D) is configured to: - generating (GEN-PPI) at least one planning projection image (PPI) of a planning data set (DS), - additionally generating (GEN-AR) and displaying (VISU-AR) augmented reality and / or virtual reality (AR) based on at least one planning projection mapping (PPI), - in the event of a change in the virtual spatial positioning of the planning data set (DS) relative to the at least one projection mapping (PI), adjusting (ADJ-PPI) the at least one planning projection mapping (PPI).

3. The display device (D) according to claim 2, characterized in that The display device (D) is designed to generate (GEN-PPI) at least one planned projection map (PPI) along a projection direction of at least one projection map (PI).

4. The display device (D) according to claim 1 or 2, characterized in that: The display device (D) is configured to: - generating (GEN-RP) a back projection (RP) of at least one projection map (PI), - additionally generating (GEN-AR) and displaying (VISU-AR) an augmented reality and / or a virtual reality (AR) based on at least one back projection (RP), - in the event of a change in the virtual spatial positioning of the planning data set (DS) relative to the at least one projection mapping (PI), adjusting (ADJ-RP) the at least one backprojection (RP).

5. The display device (D) according to claim 1, characterized in that The display device (D) is designed to determine at least one spatial degree of freedom of the transformation rule based on geometrical properties of the planning data set (DS) and / or the image data set (ID).

6. The display device (D) according to claim 1 or 2, characterized in that: The sensor unit (S) comprises an optical and / or tactile and / or electromagnetic and / or acoustic sensor and / or a line of sight direction acquisition unit, The sensor unit (S) is constructed to collect (CAP-UI) user input (UI) to collect the positioning and / or movement of at least one body part of the user (U) and / or the voice input of the user (U) and / or the direction of sight of the user (U).

7. The display device (D) according to claim 1 or 2, characterized in that: The planning data set (DS) and / or the image data set (ID) have predetermined image information, Therein, the display device (D) is designed to additionally generate (GEN-AR) and display (VISU-AR) augmented reality and / or virtual reality (AR) based on predetermined image information.

8. The display device (D) according to claim 7, characterized in that The display device (D) is also configured to select and / or generate predetermined image information based on a user input (UI).

9. The display device (D) according to claim 1 or 2, characterized in that: The display device (D): - having a first display unit for displaying (VISU-AR) augmented reality and / or virtual reality (AR), - having a second display unit (41) for displaying at least one projection map (PI) and / or at least one planning projection map (PPI), The sensor unit (S) is further configured to acquire spatial relative positioning of the first display unit and the second display unit (41) relative to each other. Therein, the display device (D) is constructed to adjust (ADJ-AR) augmented reality and / or virtual reality (AR) based on the acquired relative positioning.

10. A system for medical imaging and display, the system comprising a display device (D) according to any one of the preceding claims and a medical imaging device (37), in, The medical imaging device (37) is designed to record and / or to provide planning data sets (DS) and / or image data sets (ID).

11. A method for registering a planning data set (DS) with an image data set (ID), the method comprising: a) receiving (REC-DS) a planning data set (DS) of a check object (31), b) receiving (REC-ID) an image data set (ID) having at least one projection mapping (PI) of an object under examination (31), c) generating (GEN-AR) an augmented reality and / or virtual reality (AR) based on the planning data set (DS) and at least one projection mapping (PI), d) displaying (VISU-AR) an augmented reality and / or virtual reality (AR) graphic display by means of a display device (D) having a sensor unit (S), e) acquiring (CAP-UI) user input (UI) by means of a sensor unit (S), f) adjusting the virtual spatial positioning of the planning data set (DS) relative to the at least one projection mapping (PI) in augmented reality and / or virtual reality (AR) based on a user input (UI), The method further includes determining, based on user input (UI), a transformation rule for adjusting the virtual space positioning of a planning data group (DS) relative to at least one projection mapping (PI), so that when the transformation rule is applied to the planning data group (DS), the planning data group (DS) can be repositioned from a starting point to an end point.

12. The method according to claim 11, characterized in that Repeat steps d) to f) until the termination condition (A) occurs.

13. The method according to claim 11 or 12, characterized in that: Step c) further comprises applying a transformation rule to register the planning data set (DS) with the image data set (ID).

14. The method according to claim 11 or 12, characterized in that: Step c) further comprises generating (GEN-PPI) at least one planning projection mapping (PPI) of the planning data set (DS) along a projection direction of the at least one projection mapping (PI), wherein an augmented reality and / or virtual reality (AR) is additionally generated (GEN-AR) based on at least one planning projection mapping (PPI), Therein, step f) further comprises adjusting (ADJ-PPI) at least one planning projection mapping (PPI) in the event of a change in the virtual spatial positioning of the planning data set (DS) relative to the at least one projection mapping (PI).

15. The method according to claim 14, characterized in that Repeating steps d) to f) until a termination condition (A) occurs, wherein the termination condition (A) comprises a comparison between the degree of agreement and a predetermined threshold value, Therein, the degree of consistency evaluates the consistency between at least one planning projection mapping (PPI) and at least one projection mapping (PI).

16. The method according to claim 11 or 12, characterized in that: Step c) further comprises generating (GEN-RP) a back projection (RP) of at least one projection map (PI), wherein an augmented reality and / or a virtual reality (AR) is additionally generated based on at least one back projection (RP), Therein, step f) further comprises adjusting at least one back projection (RP) in the event of a change in the virtual spatial positioning of the planning data set (DS) relative to the at least one projection mapping (PI).

17. The method according to claim 16, characterized in that Repeating steps d) to f) until a termination condition (A) occurs, wherein the termination condition (A) comprises a comparison of a further degree of agreement with a further predetermined threshold value, Therein, the further degree of agreement assesses the agreement between at least one backprojection (RP) and the planning data set (DS).

18. A computer program product comprising a program and being directly loadable into a memory of a programmable computing unit of a provision unit (PRVS.D), the computer program product having program means for carrying out a method according to any one of claims 11 to 17 when the program is executed in the computing unit of the provision unit (PRVS.D).

Images