Method for controlling a display, computer program and mixed reality display device

IN595065BActive Publication Date: 2026-07-10APOQLAR GMBH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
IN202237008533
Authority / Receiving Office
IN · IN
Patent Type
Patents
Current Assignee / Owner
Priority Date
2019-09-09
Filing Date
2022-02-18
Publication Date
2026-07-10
Estimated Expiration
2040-09-09

AI Technical Summary

Technical Problem

Accurate alignment of virtual and real objects in mixed reality applications, particularly in medical settings, is challenging due to registration issues, which can compromise the success of medical treatments and patient safety.

Method used

A method involving the generation of 3D point clouds from real and medical imaging data, using semantic segmentation to determine transformation matrices for precise alignment of virtual medical imaging data with real-world views, employing techniques like convolutional neural networks and Iterative Closest Point algorithms.

Benefits of technology

This method enables accurate and reliable alignment of virtual medical imaging data with real-world views, enhancing the safety and success of medical treatments by providing a precise virtual fusion of medical imaging data and real-world anatomy.

✦ Generated by Eureka AI based on patent content.
Patent Text Reader

Abstract

Method for controlling a display of a mixed reality display device, wherein source and target point clouds representing a treatment object's surface are generated from image data and from medical imaging data of the treatment object. A number of segmentation masks are determined in the point clouds by applying semantic segmentation. A transformation between the source point cloud and the target point cloud is determined using the segmentation masks, and at least a part of the medical imaging data is superimposed on the treatment object using the determined transformation.
Need to check novelty before this filing date? Find Prior Art

Description

The invention is related to a method for controlling a display of a mixed reality device.Furthermore, the invention is related to a computer program having program codemeans adapted to perform such a method.Moreover, the invention is related to a mixed reality display device with such a computer program.In general, the invention concerns the field of visualization of virtual information incombination with the real environment. The virtual information is superimposed onreal objects on a display of a display device. This area is commonly called "mixed reality".The "virtuality continuum" extends from the purely real environment to the purely virtual environment, including augmented reality and augmented virtuality ranging be20 tween these extrema. The term "mixed reality" is generally defined as "anywhere between the extrema of the virtuality continuum", i. e. mixed reality generally comprisesthe complete virtuality continuum except for pure reality and pure virtuality. In thecontext of the present application, the term "mixed reality" may particularly refer to"augmented reality".Mixed reality technology is especially promising for medical applications, e. g. formedical surgery or other medical treatment. For example, medical imaging data (CTimages, MRI images etc.) visualizing the anatomy and / or the physiological processesof a human or animal body can be superimposed on the real world view of the bodyby means of a mixed reality display device. In this way, for example, surgeons can besupported during surgery by virtually placing such medical imaging data directly onthe treatment object, i. e. on the patient's body or a part thereof.One of the most important challenges in mixed reality is the registration problem, i. e.the problem of properly aligning the objects in the real world and the objects in thevirtual world with respect to each other. Without accurate registration, the illusion thatthe two worlds coexist will be compromised. More seriously, in medical applications,risks for the medical treatment's success and even for the patient's health may resultfrom inaccurate registration. As a result, it is crucial that the virtual information beingvisualized, e. g. the medical imaging data, precisely matches the real world, i. e. thetreatment object, in its position, size and perspective.From EP 2 874 556 B1, a method and a corresponding system based on augmentedreality are known that enable instrument guidance in surgeries and other interven15 tional procedures. For this purpose, an interventional path for use in the interventional procedure is obtained, wherein the interventional path is planned based on 3Dimage data of a patient's interior, and a camera image of the patient's exterior is obtained during the interventional procedure. A spatial correspondence is establishedbetween the camera image and the 3D image data and a view of the interventionalpath is calculated that corresponds with the camera image. Finally, the view of the interventional path is combined with the camera image to obtain a composite image,which is displayed on a display.It is an object of the present invention to provide an improved technique for visualiz25 ing virtual information in medical applications that allows an improved alignment ofvirtual and real objects.The object of the invention is achieved by a method for controlling a display of amixed reality display device with the features of claim 1.According to the invention, the method comprises at least the following steps:a) providing an image data set comprising a plurality of images of a treatmentobject, wherein the treatment object is a patient's body or a part thereof and theimages depict the treatment object from different perspectives,b) generating a 3D target point cloud from the image data set, wherein the targetpoint cloud comprises a multitude of points defined in a three-dimensionalcoordinate system and the points represent the treatment object's surface,c) determining a number of semantic segmentation masks in the target point cloudby applying semantic segmentation,d) providing a medical imaging data set that comprises medical imaging data ofthe treatment object,e) generating a 3D source point cloud from the medical imaging data set, whereinthe source point cloud comprises a multitude of points defined in a three10 dimensional coordinate system and the points also represent the treatmentobject's surface,f) determining a number of semantic segmentation masks in the source pointcloud by applying semantic segmentation,g) determining a transformation between the source point cloud and the targetpoint cloud using the segmentation masks of the source point cloud and thesegmentation masks of the target point cloud, andh) visualizing at least a part of the medical imaging data on the display, whereinthe medical imaging data is superimposed on the treatment object and alignedwith the treatment object using the transformation between the source pointcloud and the target point cloud.The steps of the method do not have to be executed in the specified order and the invention is not limited accordingly, i.e. the alphabetic order of the letters does not imply a specific sequence of steps a) to h). For example, as a matter of course, steps a)to c) could be executed after steps d) to f), or some of the method's steps could beexecuted in parallel.The invention proposes a method for controlling the display of the mixed reality device to superimpose medical imaging data on the treatment object. The inventionthus proposes a method for controlling the display of the mixed reality device to visualize medical imaging data on the treatment object.In the context of the present application, the term "treatment object" refers to a patient's body or a part thereof. The patient may be a human or an animal, i.e. the treat35 ment object may be a human or animal body or a part thereof.The terms "mixed reality display device" and "mixed reality device" are used interchangeably. The term "computer" is uses in its broadest sense, i.e. it refers to anyprocessing device that can be instructed to carry out sequences of arithmetic and / orlogical operations.The term "2D" refers to two-dimensional coordinates. The term "3D" refers to threedimensional coordinates. The term "4D" refers to four-dimensional coordinates.In addition to the display, the mixed reality device may comprise a computer and amemory. The mixed reality device may also comprise a plurality of computers. Moreover, the mixed reality device may comprise a camera, in particular a 3D camera system, and / or a number of sensors, in particular at least one depth sensor, e. g. a timeof-flight depth sensor. The mixed reality display device may also comprise a position15 ing system and / or an inertial measurement unit.In step a), an image data set comprising a plurality of images of the treatment objectis provided, wherein the images depict the treatment object from different perspectives. These images represent a real world view of the treatment object. They can begenerated, for example, by means of a camera and / or a depth sensor, in particularby means of a camera and / or a depth sensor of the mixed reality device.In step d), a medical imaging data set comprising medical imaging data of the treatment object is provided. This medical imaging data represents the virtual informationto be visualized on the display. The medical imaging data may comprise, for example, sectional images of the treatment object. The medical imaging data may be generated, for example, using a medical imaging method like magnetic resonance imaging (MRI). The medical imaging data may be generated before and / or during themedical treatment, e. g. before and / or during surgery.In steps b) and e), three-dimensional point clouds are generated that represent thetreatment object'. For this purpose, methods of 3D reconstruction can be used.In steps c) and f), semantic segmentation is applied to determine a number of seg35 mentation masks in the target point cloud and the source point cloud, respectively.Semantic segmentation (also referred to as semantic image segmentation) can bedefined as the task of clustering parts of images together that belong to the same object class. In the context of the present application, the object class may be, for example, a specific part of the treatment object's anatomy. If the treatment object is ahuman head, for example, simple object classes may include nose, ears, mouth,eyes, eyebrows etc. The term "semantic segmentation mask" refers to a part (or segment) of the image that has been determined, using semantic segmentation, as belonging to the same object class. Semantic segmentation can be performed in 2Ddata, i. e. based on pixels, or in 3D data, i. e. based on voxels.In step g), a transformation between the source point cloud and the target point cloudis determined using the segmentation masks of the source point cloud and the segmentation masks of the target point cloud. In this step, a transformation can be determined that, when applied to one of the point clouds, aligns the points of both pointclouds with each other. The transformation can include translation and / or rotation.For example, the transformation can have the form of a transformation matrix, in particular the form of a 4x4 transformation matrix. The determined transformation cantransform, in particular approximately, the source point cloud to the target point cloudor vice versa.In addition to the segmentation masks, other parameters can be used as inputs fordetermining the transformation between the point clouds. In particular, the transformation between the source point cloud and the target point cloud can be determinedusing the segmentation masks of the source point cloud and the segmentation masksof the target point cloud and the coordinates of the points of the source point cloudand the coordinates of the points of the target point cloud.In step h), the medical imaging data is visualized on the display, wherein the medicalimaging data is superimposed on the real world view of the treatment object andaligned with it using the transformation determined in step g). In doing so, a virtual fusion of the visualized medical imaging data and the real world view of the treatmentobject is created.The invention is based on the finding that by using semantic segmentation, the regis35 tration problem can be solved more accurately and more reliably. This is achieved bydetermining a transformation between the source point cloud and the target pointcloud based on the segmentation masks of both point clouds.By way of example, an optimization function used to determine the transformationbetween the source point cloud and the target point cloud can be designed to favortransformations that accurately match corresponding semantic segmentation masksin both point clouds, i. e. transformations that accurately match semantic segmentation masks with identical and / or similar object classes (e.g. nose, ear, eyes, eyebrows). This can be realized, for example, by using a four-dimensional optimizationalgorithm to determine the transformation between the source point cloud and thetarget point cloud, wherein for each point of the respective point cloud, the objectclass (e.g. nose, ear, mouth) of the corresponding semantic segmentation mask is interpreted as a fourth dimension of the point (in addition to the point's 3D coordinates). For example, a 4D variant of the Iterative Closest Point (ICP) algorithm maybe used for this purpose.The invention allows aligning the virtual information of the medical imaging data veryaccurately and reliably with the real world view of the treatment object. As a result, animproved virtual fusion of the visualized medical imaging data and the real world viewof the treatment object can be created, and risks for the medical treatment's successand for the patient's health can be avoided.According to an advantageous embodiment of the invention, it is proposed that thedisplay is designed as an optical see-through display, in particular as an optical see25 through head-mounted display.Such embodiments of the invention provide the advantage of a realistic and intuitiveperception of the real environment for the user, e. g. a surgeon.According to another advantageous embodiment of the invention, it is proposed thatthe mixed reality display device comprises or consists of a head-mounted mixed reality display device and / or mixed reality smartglasses. The mixed reality display devicemay, for example, comprise or consist of a Microsoft HoloLens device or a MicrosoftHoloLens 2 device or a similar device.Such embodiments of the invention provide the advantage that they are convenientto use and at the same time offer powerful hardware for visualizing virtual informationto the user. In particular, many head-mounted mixed reality display devices andmixed reality smartglasses comprise, in addition to the display, powerful and versatileintegrated hardware components including high-performance processors andmemory, 3D camera systems and time-of-flight depth sensors, positioning systemsand / or an inertial measurement units.According to another advantageous embodiment of the invention, the mixed realitydisplay device may comprise an additional external computer, e. g. an externalserver, which is connected to the display and adapted to execute at least parts of themethod proposed according to the invention. The mixed reality display device may,for example, comprise a head-mounted mixed reality display device and / or mixed reality smartglasses and an additional computer, e. g. an external server, which is con15 nected to the head-mounted mixed reality display device and / or mixed reality smartglasses, respectively, by means of a wired or wireless connection. The externalserver may be designed as a cloud server.Such embodiments provide the advantage of additional computation power for com20 plex and computationally intensive operations, which are particularly common in therelevant fields of computer vision and computer graphics.According to another advantageous embodiment of the invention, all components ofthe mixed reality display device may be integrated in the head-mounted mixed realitydisplay device and / or the mixed reality smartglasses. This provides the advantage ofa compact and hence highly mobile mixed reality display device.According to another advantageous embodiment of the invention, it is proposed thatthe medical imaging data is generated using at least one of the following medical im30 aging methods: magnetic resonance imaging (MRI), computed tomography (CT),cone beam computed tomography (CBCT), digital volume tomography (DVT), fluoroscopic intraoperative images, X-rays, radiography, ultrasonography, endoscopyand / or nuclear medicine imaging.This provides the advantage that the results of powerful and versatile modern medical imaging methods can be beneficially utilized during medical treatment, e.g. duringsurgery, by means of mixed reality visualization. This allows, for example, visualizingthe patient's internal organs and / or carcinomas and / or other internal defects of thepatient's body.According to another advantageous embodiment of the invention, it is proposed thatthe semantic segmentation masks in the target point cloud and / or the semantic segmentation masks in the source point cloud are determined using a convolutional neu10 ral network configured for semantic segmentation.Semantic segmentation by means of convolutional neural networks (CNNs) hasmade impressive progress in the last few years. Appropriately designed and trainedCNNs allow reliable, accurate and fast semantic segmentation of 2D and even 3Dimage data. Embodiments of the invention that employ a CNN for semantic segmentation hence provide the advantage that the powerful capabilities of CNNs can be utilized to improve mixed reality visualization. For example, the U-Net CNN architecturemay be used for this purpose, i. e. the convolutional neural network configured for semantic segmentation may be designed as a U-NET CNN.The CNN can be trained for semantic segmentation of a body or a specific part of thebody using appropriate training data sets that include semantic segmentation maskslabeled with their respective object class (e.g. nose, ear, eyes, eyebrows in case of atraining set for a human head).According to another advantageous embodiment of the invention, it is proposed thatstep c) comprises the following:determining a number of semantic segmentation masks in the images of theimage data set by applying semantic segmentation to the images of the imagedata set, in particular using a convolutional neural network configured forsemantic segmentation, and determining the semantic segmentation masks in the target point cloud using thesemantic segmentation masks in the images of the image data set.According to this embodiment, it is proposed that semantic segmentation masks aredetermined in the images of the image data set. The images may particularly bedesigned as 2D images, in particular 2D RGB images. Based on these semanticsegmentation masks in the images, 3D semantic segmentation masks can bedetermined in the target point cloud. For example, each point of the 3D target pointcloud could be projected to the 2D image to determine the semantic segmentationmask in the 2D image that corresponds to the respective point in the 3D target pointcloud.Such an embodiment provides the advantages that it facilitates semanticsegmentation and allows using sophisticated available methods for semanticsegmentation in 2D images, in particular in 2D RGB images, to determine thesemantic segmentation masks. For example, powerful convolutional neural networksand corresponding training data sets are available for semantic segmentation in 2DRGB images. Using their potential allows a fast, accurate and reliable semanticsegmentation of the 2D images. These results can be transferred to the 3D pointcloud to utilize them for the purposes of the claimed method, i. e. for determining anaccurate transformation between the source point cloud and the target point cloud.According to another advantageous embodiment of the invention, it is proposed thatthe image data set comprises a plurality of visual images and / or depth images of thetreatment object and the 3D target point cloud is generated from the visual imagesand / or the depth images, in particular using a photogrammetry method and / or adepth fusion method.The visual images may be generated by means of a camera, in particular a 3Dcamera system. The visual images may be designed as, for example, RGB imagesand / or grayscale images. The depth images may be generated by means of a 3Dscanner and / or a depth sensor, in particular by means of a 3D laser scanner and / or atime-of-flight depth sensor. The depth images may be designed as depth maps. Thecombination of visual images and depth images may be designed RGB-D imagedata. The images of the image data set may thus be designed as RGB-D images.The 3D target point cloud may be generated from the visual images and / or the depthimages using 3D reconstruction methods, in particular active and / or passive 3Dreconstruction methods.The 3D target point cloud may be generated from the visual images usingphotogrammetry. In particular, the 3D target point cloud may be generated from thevisual images using a Structure-from-Motion (SfM) process and / or a Multi-ViewStereo (MVS) process. For example, the COLMAP 3D reconstruction pipeline can beused for this purpose.The 3D target point cloud may be generated from the depth images using depthfusion, i. e. using 3D reconstruction from multiple depth images. The depth fusionmay be based on the Truncated Signed Distance Function (TSDF). For example, thePoint Cloud Library (PCL) may be used to generate the depth images using depthfusion. For example, Kinect Fusion may be used as a depth fusion method. Inparticular, a Kinect Fusion implementation contained in PCL may be used for thispurpose, e.g. KinFu.Such embodiments of the invention that comprise generating the 3D target pointcloud from visual images and / or depth images, as explained above, provide theadvantage that an accurate and detailed reconstruction of the treatment object'ssurface can be achieved.According to another advantageous embodiment of the invention, it is proposed thatstep c) comprises the following:determining a number of semantic segmentation masks in the visual imagesand / or in the depth images of the image data set by applying semanticsegmentation to the visual images and / or the depth images of the image dataset, in particular using a convolutional neural network configured for semanticsegmentation, and determining the semantic segmentation masks in the target point cloud using thesemantic segmentation masks in the visual images and / or the depth images ofthe image data set.Such embodiments provide the advantages that they facilitate semantic segmentation and allow using sophisticated available methods for semantic segmentation in2D images and / or in depth images to determine the semantic segmentation masks.The resulting semantic segmentation masks in the visual images and / or in the depthimages can be transferred to the 3D point cloud to utilize them for the purposes ofthe claimed method, i. e. for determining an accurate transformation between thesource point cloud and the target point cloud. In doing so, a fast, accurate and detailed reconstruction of the treatment object's surface can be achieved.According to another advantageous embodiment of the invention, it is proposed thatthe image data set comprises a plurality of visual images and depth images of thetreatment object and step b) comprises the following:generating a first 3D point cloud from the visual images of the image data set,generating a second 3D point cloud from the depth images of the image data set,and generating the 3D target point cloud using the first 3D point cloud and the second3D point cloud, in particular by merging the first 3D point cloud and the second3D point cloud.Such an embodiment provides the advantage that accuracy and completeness of the3D reconstruction can be improved and hence accuracy and completeness of the resulting target point cloud can be improved. This is achieved by merging the first 3Dpoint cloud, which may be a RGB-based point cloud, with the second 3D point cloud,which is a depth-based point cloud.According to another advantageous embodiment of the invention, it is proposed thatstep f) comprises determining a number of semantic segmentation masks in the medical imagingdata of the medical imaging data set by applying semantic segmentation to themedical imaging data of the medical imaging data set, in particular using aconvolutional neural network configured for semantic segmentation anddetermining the semantic segmentation masks in the source point cloud usingthe semantic segmentation masks in the medical imaging data of the medicalimaging data set.According to this embodiment, it is proposed that semantic segmentation masks aredetermined in the medical imaging data of the medical imaging data set. The medicalimaging data may comprise 2D medical images, in particular 2D sectional images,and semantic segmentation masks may be determined in these 2D medical images.Based on these semantic segmentation masks in the medical imaging data, 3D semantic segmentation masks can be determined in the source point cloud. For example, each point of the 3D source point cloud could be projected to the 2D medical image to determine the semantic segmentation mask in the 2D medical image that corresponds to the respective point in the 3D source point cloud.Such an embodiment provides the advantages that it facilitates semantic segmentation and allows using sophisticated available methods for medical semantic segmentation in 2D images to determine the semantic segmentation masks. For example,powerful convolutional neural networks and corresponding training data sets areavailable for medical semantic segmentation in 2D medical images. An example includes the U-Net convolutional neural network architecture. Using the potential ofthese CNNs allows fast and precise semantic segmentation of the 2D medical images. These results can be transferred to the 3D point cloud to utilize them for thepurposes of the claimed method, i. e. for determining an accurate transformation be20 tween the source point cloud and the target point cloud.The medical imaging data may also comprise 3D medical imaging data, for example,a 3D medical imaging model of the treatment object. The 3D medical imaging datamay be reconstructed from a plurality of 2D medical images, in particular 2D sec25 tional images, of the treatment object. These 2D medical images may be generatedusing a medical imaging method. In such embodiments where the medical imagingdata also comprises 3D medical imaging data, the semantic segmentation masksmay also be determined in the 3D medical imaging data. 3D semantic segmentationmethods, in particular based on convolutional neural networks configured for 3D se30 mantic segmentation, may be used for this purpose. Based on the semantic segmentation masks in the 3D medical imaging data, 3D semantic segmentation masks canbe determined in the source point cloud.According to another advantageous embodiment of the invention, it is proposed thatstep c) comprises determining the semantic segmentation masks in the targetpoint cloud by directly applying semantic segmentation to the target point cloud,in particular using a convolutional neural network configured for semanticsegmentation, and / or step f) comprises determining the semantic segmentation masks in the sourcepoint cloud by directly applying semantic segmentation to the source point cloud,in particular using a convolutional neural network configured for semanticsegmentation.According to another advantageous embodiment of the invention, it is proposed thatthe transformation between the source point cloud and the target point cloud is determined in step g) by means of an Iterative Closest Point (ICP) algorithm using the coordinates of the points of the source point cloud and the segmentation masks of thesource point cloud and the coordinates of the points of the target point cloud and thesegmentation masks of the target point cloud.According to this embodiment, it is proposed that an Iterative Closest Point (ICP) algorithm is used to determine the transformation between the source point cloud andthe target point cloud. In particular, a 4D variant of the ICP algorithm may be used forthis purpose. The coordinates of the points and the segmentation masks of thesource point and target point cloud are used as inputs for the ICP algorithm. This canbe realized, for example, by using a 4D variant of the ICP algorithm to determine thetransformation between the source point cloud and the target point cloud, wherein foreach point of the respective point cloud, the object class (e.g. nose, ear, mouth) ofthe corresponding semantic segmentation mask is interpreted as a fourth dimensionof the point (in addition to the point's 3D coordinates). By including the semantic segmentation masks in the ICP algorithm as explained above, the ICP algorithm takesinto account not only the spatial information of the points of the point clouds (i.e. theircoordinates), but their semantics, i.e. their meaning, too.The semantic segmentation masks may also be used to determine an initial estimation of the transformation between the source point cloud and the target point cloudfor the ICP algorithm, i. e. to determine an initial alignment.The embodiments explained above provide the advantage that they improve theguidance of the ICP algorithm to find an optimal transformation between the sourcepoint cloud point cloud and the target point cloud. For example, by using semanticsegmentation masks for determining the transformation, it can be avoided that theICP algorithm finds a local optimum as a solution. Therefore, the accuracy of thetransformation and hence the alignment of the medical imaging data with the treatment object can be improved.According to another advantageous embodiment of the invention, it is proposed thatstep a) comprises removing a background and / or other irrelevant parts from theimages of the treatment object using semantic segmentation, in particular using aconvolutional neural network configured for semantic segmentation, and / orstep d) comprises removing a background and / or other irrelevant parts from themedical imaging data of the treatment object using semantic segmentation, inparticular using a convolutional neural network configured for semanticsegmentation.Irrelevant parts may be parts of the images or the medical imaging data, respectively,that are not relevant for the purposes of medical imaging and / or not relevant for su20 perimposing the medical imaging data on the treatment object, such as the hair of ahuman head.Such embodiments provide the advantage that they facilitate generating the pointclouds, determining the semantic segmentations masks and determining the transfor25 mation between the point clouds.According to another advantageous embodiment of the invention, it is proposed thatthe medical imaging data set comprises a 3D medical imaging model of the treatmentobject, wherein the 3D medical imaging model is reconstructed from a plurality of 2Dsectional images of the treatment object generated by a medical imaging method.Such embodiments provide the advantage that they can facilitate generating the 3Dsource point cloud, while at the same time the accuracy of the resulting 3D sourcepoint cloud can be improved.According to another advantageous embodiment of the invention, it is proposed thatin step a), the image data set comprising the images of the treatment object is created by means of a camera, in particular a 3D camera system, of the mixed realitydisplay device, and / or by means of a depth sensor, in particular by means of a time5 of-flight depth sensor, of the mixed reality display device.In this way, the image data set, which is necessary for generating the target pointcloud, can be created and provided for the purposes of the invention in a very simpleand user-friendly way. For example, using the mixed reality display device's cameraand / or depth sensor, the images of the image data set can be created by automatically scanning the treatment object. In particular, if the mixed reality display devicecomprises or consists of a head-mounted mixed reality display device and / or mixedreality smartglasses, the camera and / or the depth sensor can be integrated into thehead-mounted mixed reality display device and / or mixed reality smartglasses, re15 spectively. In this case, the user can simply direct his head towards the part of thetreatment object to be scanned.According to another advantageous embodiment of the invention, it is proposed thatin step a), when the images are created by means of the camera of the mixedreality display, the position of the camera is determined in a three-dimensionalcoordinate system for each image and stored as 3D camera position and instep b), the target point cloud is generated using the 3D camera positions, and / orin step a), when the images are created by means of the camera of the mixedreality display, the orientation of the camera is determined in a three-dimensionalcoordinate system for each image and stored as 3D camera orientation and instep b), the target point cloud is generated using the 3D camera orientations.Such embodiments provide the advantage that they allow a particularly precise andreliable generation of the target point cloud from the image data set.According to another advantageous embodiment of the invention, it is proposed thatany position, any orientation and any transformation determined in steps a) to h) isdetermined markerless and / or using Simultaneous Localization and Mapping(SLAM).Such embodiments provide the advantage that they are particularly comfortable forthe user, as no optical markers or other fiducial markers have to be provided.The object of the invention is further achieved by a computer program having pro5 gram code means adapted to perform a method as described above when the computer program is executed on a computer.The object of the invention is further achieved by a mixed reality display device having a display, a computer and a memory, wherein the computer program describedabove is stored in the memory and the computer is adapted to execute the computerprogram.The computer program may be designed as a distributed computer program. Thecomputer and the memory may be designed as a distributed computer and a distrib15 uted memory, respectively, i. e. the computer that is adapted to execute the computer program may comprise two or more computers. The distributed memory maycomprise multiple memories, wherein each memory may store at least a part of thedistributed computer program, and each of the two or more computers may beadapted to execute a part of the distributed computer program.In addition, the mixed reality display device may have an interface that is adapted toreceive the medical imaging data from an external source. The interface may beadapted, for example, to be connected to a medical imaging device, e. g. a magneticresonance imaging (MRI) device and / or a computer tomography (CT) device, and / orto a memory that stores the medical imaging data. This may include a wired and / orwireless connection.As mentioned above, the mixed reality display device may comprise or consist of ahead-mounted mixed reality display device and / or mixed reality smartglasses. Themixed reality display device may, for example, comprise or consist of a Microsoft HoloLens device or a Microsoft HoloLens 2 device or a similar device.As mentioned above, the mixed reality display device may comprise an additional external computer, e. g. an external server, which is connected to the display andadapted to execute at least parts of the method proposed according to the invention.The mixed reality display device may, for example, comprise a head-mounted mixedreality display device and / or mixed reality smartglasses and an additional computer,e. g. an external server, which is connected to the head-mounted mixed reality display device and / or mixed reality smartglasses, respectively, by means of a wiredand / or wireless connection. The external server may be designed as a cloud server.As mentioned above, all components of the mixed reality display device may be integrated in the head-mounted mixed reality display device and / or the mixed realitysmartglasses. In the following, the invention will be explained in more detail using the exemplaryembodiments schematically shown in the attached drawings. The drawings show thefollowing:Figure 1 - a schematic representation of a mixed reality display device according to the invention;Figure 2 - a schematic representation of a method for controlling a mixed realitydisplay device according to the invention;Figure 3 - a schematic representation of a 3D target point cloud with semanticsegmentation masks;Figure 4 - a schematic representation of a 3D source point cloud with semanticsegmentation masks.Figure 1 shows a schematic representation of a mixed reality display device 1, whichcomprises a head-mounted mixed reality display device in the form of a pair of mixedreality smartglasses 5. In this exemplary embodiment, the mixed reality smartglasses 5 are of the type Microsoft HoloLens. The smartglasses 5 have amemory 13a and a computer 11a connected with the memory 13a, wherein the computer 11a comprises several processing units, namely a CPU (Central ProcessingUnit), a GPU (Graphics Processing Unit) and a HPU (Holographic Processing Unit).Moreover, the smartglasses 5 of the mixed reality display device 1 have a camera 9in the form of a 3D camera system. The camera 9 is adapted to create visual imagesof a treatment object 15 from different perspectives. Furthermore, the smartglasses 5comprise a plurality of sensors 7, including a time-of-flight depth sensor that isadapted to create depth images of the treatment object 15 from different perspectives. In this exemplary embodiment, the treatment object 15 is the head of a humanpatient 17.In addition, the smartglasses 5 of the mixed reality display device 1 have display 3,which is designed as an optical see-through head-mounted display in this exemplaryembodiment. The see-through display 3 is adapted to visualize virtual information, inthis exemplary embodiment medical imaging data, by superimposing the virtual information on the real view of the treatment object 15.Furthermore, figure 1 illustrates that the mixed reality display device 1 comprises aserver 21, which is connected to the smartglasses 5 via a wireless and / or wired connection. The server 21 comprises a memory 13b and a computer 11b connected withthe memory 13b, wherein the computer 11b comprises a CPU and a GPU. Theserver 21 has an interface 19 that is adapted to receive medical imaging data froman external source, namely from a memory that stores the medical imaging data, viaa wired and / or wireless connection.Figure 2 shows a schematic representation of an exemplary method for controlling amixed reality display device according to the invention.In step a), an image data set is provided that comprises a plurality of images of thetreatment object 15, wherein the images depict the treatment object from differentperspectives. In this exemplary embodiment, the image data set comprises a pluralityof visual images in the form of 2D RGB images and a plurality of depth images in theform of depth maps. The visual images are created by means of the camera 9 of themixed reality display device 1, and the depth images are created by means of thedepth sensor 7 of the mixed reality display device 1 (see figure 1). In other words,this means that an image data set containing RGB-D image data is created. For thispurpose, the user wearing the smartglasses 5, e. g. a surgeon, can simply direct hishead towards the treatment object 15 in different perspectives and the treatment object 15 is automatically scanned from different perspectives using the camera 9 andthe depth sensor 7. When the images are created, the position and the orientation ofthe camera 9 are determined in a three-dimensional coordinate system for each im25 age and stored as 3D camera position and 3D camera orientation, respectively. Forthis purpose, Simultaneous Localization and Mapping (SLAM) is used. The 3D camera position and 3D camera orientation for each image can be referred to as extrinsiccamera parameters. The image data set including the extrinsic camera parameters istransferred from the smartglasses 5 to the server 21 afterwards.In step b), the server 21 generates a 3D target point cloud from the image data set,wherein the target point cloud comprises a multitude of points defined in a three-dimensional coordinate system and the points represent the surface of the treatmentobject 15. For this purpose, the extrinsic camera parameters, i. e. the 3D camera po35 sitions and the 3D camera orientations, are used.The 3D target point cloud is generated from the visual images and the depth images.For this purpose, a first 3D point cloud is generated from the visual images of the image data set using a photogrammetry method, namely by using the COLMAP 3D re5 construction pipeline including a Structure-from-Motion (SfM) process and a MultiView Stereo (MVS) process. Moreover, a second 3D point cloud is generated fromthe depth images of the image data set using depth fusion, i. e. using 3D reconstruction from multiple depth images. For this purpose, For example, the Point Cloud Library (PCL) is used, namely the Kinect Fusion implementation KinFu contained inPCL. Afterwards, the 3D target point cloud is generated by merging the first 3D pointcloud and the second 3D point cloud.In step c), a number of semantic segmentation masks in the target point cloud is determined by applying semantic segmentation.Firstly, a number of semantic segmentation masks is determined in the 2D visual images of the image data set by applying semantic segmentation to these 2D RGB images. The semantic segmentation masks define an object class for every pixel ofeach 2D visual image, wherein the object classes may include an object class"other", "void" or similar for areas of the image that cannot be matched otherwise. Fordetermining the semantic segmentation masks in the 2D visual images, a convolutional neural network configured (CNN) for semantic segmentation is used, namely aCNN based on the U-Net CNN architecture (U-Net CNN). The U-Net CNN is trainedfor semantic segmentation of the treatment object 15, i.e. for semantic segmentationof a human head, using appropriate training data sets that include semantic segmentation masks labeled with their respective object class (e.g. nose, ear, eyes, eyebrows).Secondly, the semantic segmentation masks in the 3D target point cloud are deter30 mined using the previously determined semantic segmentation masks in the 2D visual images of the image data set. For this purpose, each point of the 3D target pointcloud is projected to a number of 2D RGB images to determine the semantic segmentation mask in the respective 2D image that corresponds to the point in the 3Dtarget point cloud.Figure 3 schematically illustrates the results of determining the semantic segmentation masks in the target point cloud. The figure shows a 3D target point cloud 23 thathas been generated from the image data set. In the 3D target point cloud 23, severalsemantic segmentation masks 25a, 25b, 25c have been determined. Two semanticsegmentation masks 25a represent the patient's right eye and left eye, respectively.Another semantic segmentation mask 25b represents the patient's nose and anothersemantic segmentation mask 25c represents the patient's mouth.Referring now back to figure 2, in step d), a medical imaging data set that comprisesmedical imaging data of the treatment object is provided. In this exemplary embodiment, the medical imaging data comprises a plurality of 2D sectional images of thetreatment object 15 that were created by means of magnetic resonance imaging(MRI) before a surgery. These 2D MRI images, in the form of DICOM data, are received by the server 21 via the server's interface 19. The server 21 reconstructs a 3Dmedical imaging model of the treatment object 15 from the plurality of 2D MRI imagesof the treatment object 15.The medical imaging data may also comprise metadata, e. g. in the form of attributesstored as tags in DICOM data. Examples of such metadata include information abouta slice thickness of sectional images and / or information about pixel spacing. Suchmetadata can be used for reconstructing the 3D imaging model and / or for generatingthe 3D source point cloud.In step e), a 3D source point cloud is generated from the medical imaging data set,wherein the source point cloud comprises a multitude of points defined in a three-dimensional coordinate system and the points also represent the surface of the treatment object 15. For this purpose, the Point Cloud Library (PCL) is used.In step f), a number of semantic segmentation masks is determined in the sourcepoint cloud by applying semantic segmentation.Firstly, a number of semantic segmentation masks is determined in the 2D MRI images of the medical imaging data set by applying semantic segmentation to these images. The semantic segmentation masks define an object class for every pixel ofeach 2D MRI image, wherein the object classes may include an object class "other","void" or similar for areas of the image that cannot be matched otherwise. For determining the semantic segmentation masks in the 2D MRI images, a convolutional neural network configured (CNN) for semantic segmentation is used, namely a CNNbased on the U-Net CNN architecture (U-Net CNN). The U-Net CNN is trained for se5 mantic segmentation of the treatment object 15, i.e. for semantic segmentation of ahuman head, using appropriate training data sets that include semantic segmentationmasks labeled with their respective object class (e.g. nose, ear, eyes, eyebrows).Secondly, the semantic segmentation masks in the 3D source point cloud are deter10 mined using the previously determined semantic segmentation masks in the 2D MRIimages of the medical imaging data set.Figure 4 schematically illustrates the results of determining the semantic segmentation masks in the source point cloud. The figure shows a 3D source point cloud 27that has been generated from the medical imaging data set. In the 3D souce pointcloud 27, several semantic segmentation masks 29a, 29b, 29c have been determined. Two semantic segmentation masks 29a represent the patient's right eye andleft eye, respectively. Another semantic segmentation mask 29b represents the patient's nose and another semantic segmentation mask 29c represents the patient'smouth.Referring now back to figure 2, in step g), a transformation between the source pointcloud and the target point cloud is determined using the segmentation masks of thesource point cloud and the segmentation masks of the target point cloud. In this ex25 emplary embodiment, a 4x4 transformation matrix is determined that includes translation and rotation and, when applied to the source point cloud, aligns the points of thesource point cloud with the points of the target point cloud. The determined 4x4transformation matrix hence serves the purpose of transformation between the different view poses (position and orientation) under which the source point cloud and thetarget point cloud have been acquired.In this exemplary embodiment, a 4D variant of the Iterative Closest Point (ICP) algorithm is used to determine the 4x4 transformation matrix. The coordinates of thepoints and the segmentation masks of the source point and target point cloud areused as inputs for the algorithm, wherein for each point of the respective point cloud,the object class (e.g. nose, ear, mouth) of the corresponding semantic segmentationmask is interpreted as a fourth dimension of the point (in addition to the point's 3Dcoordinates). The 4D ICP's optimization function is designed to favor transformationsthat accurately match corresponding semantic segmentation masks in both pointclouds (nose and nose, ear and ear, mouth and mouth etc.).The 4x4 transformation matrix and the medical imaging data are transferred fromserver 21 to mixed reality smartglasses 5.In step h), at least a part of the medical imaging data, i. e. the MRI imaging data, isvisualized on the optical see-through display 3 of the smartglasses 5 (see figure 1),wherein the medical imaging data is superimposed on the real view of the treatmentobject 15 and aligned with the treatment object 15 using the transformation betweenthe source point cloud and the target point cloud, i. e. using the 4x4 transformationmatrix determined in step g).In this way, a surgeon using the a mixed reality display device 1 can be supportedduring surgery by virtually visualizing the anatomy of the treatment object 15, as it isindicated by the MRI images, accurately aligned with the real world view of the treat20 ment object 15.In this exemplary embodiment, steps a) and h) are executed by mixed reality smartglasses 5, while steps b) to g) are executed by server 21. In other embodiments, additional steps or all steps of the method may be executed by mixed reality smartglasses 5. List of reference signs1 mixed reality display device3 display5 mixed reality smartglasses7 sensors9 camera11a, 11b computer13a, 13b memory15 treatment object17 patient19 interface21 server23 target point cloud25a, 25b, 25c segmentations masks of the target point cloud27 source point cloud29a, 29b, 29c segmentation masks of the source point cloud

Claims

1. Method for controlling a display of a mixed reality display device (1), the method comprising at least the following steps: a) providing an image data set comprising a plurality of images of a treatment object (15), wherein the treatment object (15) is a patient's body or a part thereof and the images depict the treatment object (15) from different perspectives, b) generating a 3D target point cloud (23) from the image data set, wherein the target point cloud (23) comprises a multitude of points defined in a three-dimensional coordinate system and the points represent the treatment object's surface, c) determining a number of semantic segmentation masks (25a, 25b, 25c) in the target point cloud (23) by applying semantic segmentation, d) providing a medical imaging data set that comprises medical imaging data of the treatment object (15), e) generating a 3D source point cloud (27) from the medical imaging data set, wherein the source point cloud (27) comprises a multitude of points defined in a three-dimensional coordinate system and the points also represent the treatment object's surface, f) determining a number of semantic segmentation masks (29a, 29b, 29c) in the source point cloud (27) by applying semantic segmentation, g) determining a transformation between the source point cloud (27) and the target point cloud (23) using the segmentation masks (29a, 29b, 29c) of the source point cloud (27) and the segmentation masks (25a, 25b, 25c) of the target point cloud (23), and h) visualizing at least a part of the medical imaging data on the display (3), wherein the medical imaging data is superimposed on the treatment object (15) and aligned with the treatment object (15) using the transformation between the source point cloud (27) and the target point cloud (23).

2. The method according claim 1, characterized in that the display (3) is designed as an optical see-through display (3), in particular as an optical see-through head-mounted display (3), and / or the mixed reality display device (1) comprises or consists of a head-mounted mixed reality display device (1) and / or mixed re5 ality smartglasses (5).

3. The method according to claim 1 or 2, characterized in that the medical imaging data is generated using at least one of the following medical imaging methods: magnetic resonance imaging (MRI), computer tomography (CT), radiography, ultrasonography, endoscopy and / or nuclear medicine imaging.

4. The method according to any of the preceding claims, characterized in that the semantic segmentation masks (25a, 25b, 25c) in the target point cloud (23) and / or the semantic segmentation masks (29a, 29b, 29c) in the source point cloud (27) are determined using a convolutional neural network configured for semantic segmentation.

5. The method according to any of the preceding claims, characterized in that step c) comprises the following: determining a number of semantic segmentation masks in the images of the image data set by applying semantic segmentation to the images of the image data set, in particular using a convolutional neural network configured for semantic segmentation, and determining the semantic segmentation masks (25a, 25b, 25c) in the target point cloud (23) using the semantic segmentation masks in the images of the image data set.

6. The method according to any of the preceding claims, characterized in that the image data set comprises a plurality of visual images and / or depth images of the treatment object (15) and the 3D target point cloud (23) is generated from the visual images and / or the depth images, in particular using a photogrammetry method and / or a depth fusion method.

7. The method according to claim 6, characterized in that step c) comprises the following: determining a number of semantic segmentation masks in the visual images and / or in the depth images of the image data set by applying semantic segmentation to the visual images and / or the depth images of the image data set, in particular using a convolutional neural network configured for semantic segmentation, and determining the semantic segmentation masks (25a, 25b, 25c) in the target point cloud (23) using the semantic segmentation masks in the visual images and / or the depth images of the image data set.

8. The method according to any of the preceding claims, characterized in that the image data set comprises a plurality of visual images and depth images of the treatment object (15) and step b) comprises the following: generating a first 3D point cloud from the visual images of the image data set, generating a second 3D point cloud from the depth images of the image data set, and generating the 3D target point cloud (23) using the first 3D point cloud and the second 3D point cloud, in particular by merging the first 3D point cloud and the second 3D point cloud.

9. The method according to any of the preceding claims, characterized in that step f) comprises determining a number of semantic segmentation masks in the medical imaging data of the medical imaging data set by applying semantic segmentation to the medical imaging data of the medical imaging data set, in particular using a convolutional neural network configured for semantic segmentation and determining the semantic segmentation masks (29a, 29b, 29c) in the source point cloud (27) using the semantic segmentation masks in the medical imaging data of the medical imaging data set.

10. The method according to any of the preceding claims, characterized in that step c) comprises determining the semantic segmentation masks (25a, 25b, 25c) in the target point cloud (23) by directly applying semantic segmentation to the target point cloud (23), in particular using a convolutional neural network configured for semantic segmentation, and / or step f) comprises determining the semantic segmentation masks (29a, 29b, 29c) in the source point cloud (27) by directly applying semantic segmentation to the source point cloud (27), in particular using a convolutional neural network configured for semantic segmentation.

11. The method according to any of the preceding claims, characterized in that the transformation between the source point cloud (27) and the target point cloud (23) is determined by means of an Iterative Closest Point (ICP) algorithm using the coordinates of the points of the source point cloud (27) and the segmentation masks (29a, 29b, 29c) of the source point cloud (27) and the coordinates of the points of the target point cloud (23) and the segmentation masks (25a, 25b, 25c) of the target point cloud (23).

12. The method according to any of the preceding claims, characterized in that step a) comprises removing a background and / or other irrelevant parts from the images of the treatment object (15) using semantic segmentation, in particular using a convolutional neural network configured for semantic segmentation, and / or step d) comprises removing a background and / or other irrelevant parts from the medical imaging data of the treatment object (15) using semantic segmentation, in particular using a convolutional neural network configured for semantic segmentation.

13. The method according to any of the preceding claims, characterized in that the medical imaging data set comprises a 3D medical imaging model of the treatment object (15), wherein the 3D medical imaging model is reconstructed from a plurality of 2D sectional images of the treatment object (15) generated by a medical imaging method.

14. The method according to any of the preceding claims, characterized in that in step a), the image data set comprising the images of the treatment object (15) is created by means of a camera (9), in particular a 3D camera system, of the mixed reality display device (1) and / or by means of a depth sensor (7), in particular by means of a time-of-flight depth sensor (7), of the mixed reality display device (1).

15. The method according to claim 14, characterized in that in step a), when the images are created by means of the camera (9) of the mixed reality display, the position of the camera (9) is determined in a three-dimensional coordinate system for each image and stored as 3D camera position and in step b), the target point cloud (23) is generated using the 3D camera positions, and / or in step a), when the images are created by means of the camera (9) of the mixed reality display, the orientation of the camera (9) is determined in a three-dimensional coordinate system for each image and stored as 3D camera orientation and in step b), the target point cloud (23) is generated using the 3D camera orientations.

16. The method according to any of the preceding claims, characterized in that any position, any orientation and any transformation determined in steps a) to h) is determined markerless and / or using Simultaneous Localization and Mapping (SLAM).

17. Computer program having program code means adapted to perform a method according to any of the preceding claims when the computer program is executed on a computer (11a, 11b).

18. Mixed reality display device (1) having a display (3), a computer (11a, 11b) and a memory (13a, 13b), wherein the computer program according to the previous claim is stored in the memory (13a, 13b) and the computer (11a, 11b) is adapted to execute the computer program.