Mixed reality view

Abstract

Computer implemented method for providing display data for a semi-transparent see through viewing device in a medical environment, the method comprising the steps: providing a 2D and / or 3D model of a first object (step S1); receiving first positioning data of the first object (step S2); receiving second positioning data of a second object (step S3); acquiring a free space based on the second positioning data (step S4); determining display data for displaying on a semi-transparent see through viewing device based on the 2D and / or 3D model, the acquired determined free space, the first positioning data and the second positioning data (step S5); and providing the determined display data for displaying on the semi-transparent see through viewing device (step S6).

Description

[0001] Mixed reality view

[0002] FIELD OF THE INVENTION

[0003] The present disclosure relates to a computer implemented method for providing display data for a semi-transparent see through viewing device in a medical environment, to a data processing apparatus, to a computer program, to a computer readable storage medium, to a system for providing display data for the semi-transparent see through viewing device, and to a use of a semitransparent see through viewing device.

[0004] TECHNICAL BACKGROUND

[0005] Augmented reality head mounted displays can be used to overlay virtual objects on the real environment. This technology is well known and already used in the medical environment. For example, a 3D model of an anatomical body part is overlaid to its physical counterpart (i.e. , the anatomical body part), such that a user of the augmented reality head mounted display can see the 3D model of the anatomical body part at the same location as the physical anatomical body part. This gives a user the possibility to see what is inside the body of the patient. However, sometimes the 3D model may disturb the view of the user onto the physical anatomical body part. It has been found that a further need exists to provide a method for providing display data for such a semi-transparent see through viewing device in a medical environment.

[0006] EXEMPLARY SHORT DESCRIPTION OF THE INVENTION

[0007] The invention proposes a computer implemented method for providing display data for semi-transparent see through viewing device in a medical environment. The display data is displayed on the semi-transparent see

[0008] AO:QS through viewing device. The display data comprises a model (e.g., 2D or 3D model) of an anatomical body part (e.g., a spine) with a so called free space. The free space is rendered transparent such that a user of the transparent see through viewing device sees in that free space area the physical anatomical body part and outside the free space the model. The free space corresponds for example to an instrument (e.g. an operating knife). Thereby, it is possible to show a user information of the inside of the anatomical body part and simultaneously not to hide the view onto the instrument.

[0009] GENERAL DESCRIPTION OF THE INVENTION

[0010] It is an object of the present invention to provide a computer implemented method for providing display data for a semi-transparent see through viewing device in a medical environment. These and other objects, which become apparent upon reading the following description, are solved by the subject matter of the independent claims. The dependent claims refer to preferred embodiments of the invention.

[0011] According to a first aspect of the present disclosure a computer implemented method for providing display data for a semi-transparent see through viewing device in a medical environment is provided, the method comprises the steps: providing a model of a first object (step S1 ); receiving first positioning data of the first object (step S2); receiving second positioning data of a second object (step S3); acquiring a free space based on the second positioning data (step S4); determining display data for displaying on the semi-transparent see through viewing device based on the model, the acquired determined free space, the first positioning data and the second positioning data (step S5); and providing the determined display data for displaying on the semi-transparent see through viewing device (step S6).

[0012] The term display data, as used herein, is to be understood broadly and may relate to data used to adapt one or more pixels in a digital display device. The display data may comprise a specific colour (e.g. blue or green) for the respective pixel in a digital display.

[0013] The term semi-transparent see through viewing device, as used herein, is to be understood broadly and may relate to a digital display configured to display one or more pixels in one or more colours and to display one or more pixels transparent to the user. The semi-transparent see through viewing device may be a head mounted device. The semi-transparent see through viewing device may be a movable device. The movable device may comprise a movable stand or movable arm. The semi-transparent see through viewing device may comprise one or more position sensors to track its own position and / or orientation. The semi-transparent see through viewing device may comprise one or more interfaces in order to exchange data with peripherals (e.g., a data processing apparatus used to carry the described method). The semitransparent see through viewing device may be augmented reality glasses (i.e. augmented reality glasses or mixed reality glasses) that can be worn by a user. The augmented reality glasses may allow a user to see real objects (i.e. physical reality) and at least partly overlaid models onto this objects (i.e. augmented reality).

[0014] The term medical environment, as used herein, is to be understood broadly and may relate to a room used to carry out a medical treatment (e.g., a surgery or medical examination). The medical environment may comprise an anatomical body part to be treated, a patient bed, one or more medical machines (e.g. a C arm CT), and / or a tracking unit.

[0015] The term model, as used herein, is to be understood broadly and may relate to a geometric model. The geometric model may be a 2D model. The geometric model may be a 3D model. The geometric model may be a combination of a 2D model and a 3D model. The geometric model may relate to the first object. The first object may be an anatomical body part. For example, the anatomical body part may be a vertebrae, a spine, or a joint. The geometric model may be rendered in order to be displayed on the semi- transparent see through viewing device. The geometric model may be derived from medical image data. The medical display data may comprise e.g. X-Ray data, CT data, and / or MRT data. The model may be registered to the first positioning data of the first object. The model may additionally comprise a planned implant, a target trajectory for such an implant. The model may be received from a data base. The model may be generated during a medical procedure by means of an imaging device (e.g. C arm CT). The model may be modified by a user during a medical procedure for example by adding a planned implant.

[0016] The term first object, as used herein, is to be understood broadly and may relate to any anatomical body part. For example, the first object may be a spine, a joint, an arm bone.

[0017] The term first positioning data, as used herein, is to be understood broadly and may particularly relate to spatial coordinates. The first positioning data may comprise three spatial coordinates and three orientations coordinates. The first positioning data may be received by optical tracking a marker arranged at the first object. The first positioning data may be received by optical tracking the first object without a marker. The first positioning data may be received by electromagnetic tracking of the first object. The first positioning data may be determined by a tracking unit. The tracking unit may be an optical tracking unit. The tracking unit may be an electromagnetic tracking unit. The first positioning data may be transmitted from the tracking unit to a data processing apparatus carrying out the method described above.

[0018] The term second object, as used herein, is to be understood broadly and may relate to any instrument or the like used in a medical treatment. The second object may comprise a knife, a drilling sleeve, and a tactile rod.

[0019] The term second positioning data, as used herein, is to be understood broadly and may particularly relate to spatial coordinates. The second positioning data may comprise three spatial coordinates and three orientations coordinates. A spatial coordinate may relate to a translatory degree of freedom. An orientation may relate to rotatory degree of freedom. The second positioning data may be used to determine a pose of the second object. The second positioning data may be received by optical tracking a marker arranged to the second object. The second positioning data may be received by optical tracking the second object without a marker. The second positioning data may be received by electromagnetic tracking of the second object. The second positioning data may be determined by a tracking unit. The tracking unit may be an optical tracking unit. The tracking unit may be an electromagnetic tracking unit. The second positioning data may be transmitted from the tracking unit to a data processing apparatus carrying out the method described above.

[0020] The term free space, as used herein, is to be understood broadly and means particularly an area that is displayed in the semi-transparent see through viewing device transparent such that a user looking through the semitransparent see through viewing device can see at least a part of the second object and / or the first object in a physical reality. This means particularly that the free space is rendered transparent and that no part of a model of the first object is displayed in this area. The user can see the physical reality in the area of the free space. The free space can be a 2D space or a 3D space. The free space may have fixed dimensions or dynamic dimensions varying over time. The dimensions may be predefined. The dimension of the free space may depend on dimensions of the first object and / or the second object.

[0021] The term acquiring a free space, as used herein, is to be understood broadly and may relate to a determination based on the second positioning data of the second object. For example, the free space may be defined by a predefined area (e.g., a 3D volume (e.g., a ball) or a 2D area (e.g., a circle)) and the positioning data of the second object and / or the positioning data of the first object. For example, the free space may comprise geometric data of the second object. The geometric data of the second object may comprise spatial dimensions of the second object. The acquiring may comprise a calculation. For example, if the second object is not in vicinity of the first object, no free space has to be determined as the model overlaid onto the first object doesn’t hide the second object. For example, if the second object is arranged above the first object and in a view of a semi-transparent see through viewing device, the free space has to be determined such that this area can be rendered transparent.

[0022] The term determining display data for displaying, as used herein, is to be understood broadly and may comprise a rendering of one or more pixels of a semi-transparent see through viewing device. The determining may comprise a registration of the model of the first object to the first object based on the first positioning data. The determining may comprise a registration of the free space to the first object and / or second object. The determining may comprise the process of rendering the model and the free space. The term rendering relates in general to generating an image from a 2D model and / or a 3D model. The rendering here particularly considers the model and the free space.

[0023] In other words, the invention is based on the finding that virtual objects can be layered onto the real environment. For example, this can be done by augmented reality head mounted displays (e.g., augmented reality glasses). These augmented reality head mounted displays can be either optical see through devices (i.e., the display is semi-transparent) or video see through devices (i.e. the real environment is filled of cameras embedded in the augmented reality head mounted display and streamed to the displays). The virtual objects may be 2D or 3D objects that represent a physical object, such as anatomical body parts (e.g. spine or head model, reconstructed from CT / MRI scans), surgical instruments, tracking markers, or implants. A virtual object can be overlaid to its physical counterpart (overlaid virtual object): it is for example possible to display a 3D model of an anatomical body part such that the user of the AR-HMD can see the 3D model at the same location as the physical anatomical body part. This is particularly useful during minimal invasive surgery, as it gives the surgeon the possibility to look virtually inside the body of the patient. However, the overlaid virtual object may sometimes be disturbing for the surgeon. For example, the 3D spine model may hide the skin and the trajectory’s entry point (intersection between the skin surface and the trajectory) while making a skin incision or while navigating / drilling a hole into the pedicle. The idea of the invention is to mask / hide automatically a part of the overlaid virtual object based on the position of the navigated instrument. The invention proposes a plurality of possibilities to provide such a “modified image” by providing display data for displaying on the semi-transparent see through viewing device. It should be noted that the display data are merely provided for displaying on a display of a semi-transparent see through viewing device. The computer implemented method can therefore not be seen as therapeutic, medical or diagnostic method practiced on the human or the animal body.

[0024] In an embodiment, the semi-transparent see through viewing device may be a head mountable device; and the method may further comprise the step: obtaining third positioning data of the head mountable device and determining the display data based on the obtained third positioning data.

[0025] A head mountable device may be worn by user. The head mountable semitransparent see through viewing device may preferably be glasses. The head mountable semi-transparent see through viewing device may preferably comprise two single spectacle lenses. On each of the two spectacle lenses a specific image may be presented such that the user can see a 3D model with the free space through the two spectacle lenses. The head mountable semitransparent see through viewing device may be a so called mixed reality glasses. The third positioning data may be determined by a tracking unit. The third positioning data may comprise three spatial coordinates and three orientations coordinates. The third positioning data may be received by optical tracking a marker arranged to the head mountable device. The third positioning data may be received by optical tracking the first object without a marker. The head mountable semi-transparent see through viewing device may be able to determine its positioning data by itself, e.g. by means of a simultaneous localization and mapping (slam) algorithm. The head mountable semitransparent see through device may therefore be equipped with one or more cameras. Based on the camera information data, the slam algorithm may determine the positioning data of the head mountable semi-transparent see through viewing device in relation to other objects such as the first object. This may be advantageous as the head mountable semi-transparent see through viewing device doesn’t have to be equipped with markers. The third positioning data may be received by electromagnetic tracking of the third object. The third positioning data may be determined by a tracking unit. The tracking unit may be an optical tracking unit. The tracking unit may be an electromagnetic tracking unit. The third positioning data may transmitted from the tracking unit to a data processing apparatus carrying out the method described above. By taking into account the position of the head mountable device a viewing direction of a user can be determined. The display data can therefore consider the viewing direction or angle of a user in order to provide her / him always a correct image for her / his specific viewing angle. This may increase the accuracy of the overlaid model and also the transparent rendered free space in the overlaid model. The free space may be determined as a spherical element. This may be advantageous as it simplifies respectively reduces a calculation and / or programming effort for different viewing angles. In summary, the head mountable device may allow a user different views on the first object, wherein from each view a free space allows a free view on the second object. This may advantageously increase visibility of the first object in combination with the model and the second object. This may increase the quality and accuracy of the image presented through the semi-transparent see through viewing device to a user.

[0026] In an embodiment, the semi-transparent see through viewing device may be a movable display; and wherein the method may further comprise the step: obtaining fourth positioning data of the movable display and determining the display data based on the obtained fourth positioning data.

[0027] The movable display may a display on a stand or an arm that can be moved over the first object. The movable display may be movable in three spatial directions and in three orientations. For example, the movable display may be arranged between a user’s head and the first object. A tracking unit may track a position of the movable display. The tracking unit may transmit the positioning data of the movable device to data processing apparatus carrying out the method described above. The tracking may comprise optical tracking and / or electromagnetic tracking. Based on the fourth positioning data a theoretic viewing angle of a user in front of the movable device can be estimated or determined. Based on the fourth positioning data display data can determined that consider the position of the movable display and particularly a viewing angle of a user. In sum, this may increase a visibility of the first object in combination with the model and the second object. This may increase the quality and accuracy of the image presented to a user.

[0028] In an embodiment, the free space is rendered transparent on a display of the semi-transparent see through viewing device.

[0029] For example, the method may firstly render the complete model for the display of the semi-transparent see through viewing device such that the complete model is overlaid onto the first object when looking through the display. In a next step, the determined free space is rendered transparent in the display such that a user looking on a free space area of the display though display directly looks onto the first object. A head mountable semi-transparent see though viewing device is in general not able to display a pixel in black. When such a semi-transparent see though viewing device gets as display data for a pixel black as colour it displays the pixel transparent. The method uses this aspect by rendering the free space black such that is displayed transparent. In other words, and with particular reference to optical see through devices, a pixel whose colour is “black” means that it is not supplied by electrical energy, so that within the semi-transparent display of the optical see through device, the user can see through the display as it is transparent at the corresponding area. This may require less effort than determining a “cut model” from the model and the acquired free space. In sum, this may be advantageous in terms of efficiency of the method. In an embodiment, the method may comprise the step: receiving a target trajectory of the second object; wherein the step S5 of determining display data may be based on the received target trajectory of the second object, wherein the display data may comprise a representation of the target trajectory, and wherein the representation of the target trajectory may be superimposed on the model and the free space.

[0030] The term target trajectory, as used herein, is to be understood broadly and may relate to a desired position of the second object. For example, the target trajectory may relate to arrangement of a drilling sleeve. The target trajectory and corresponding positioning data may be obtained by a HMI (e.g., a physician inputs the target trajectory) or by interface to a data base (e.g., electronic medical record system). The representation of the target trajectory may be a single arrow. The representation may be superimposed on the display of the semi-transparent see though viewing device such that firstly the model is shown with the free space and superimposed on the model and the free space the representation is shown. This may be advantageous as a user sees in the free space the first object, the second object and the representation of the target trajectory of the second object. This may advantageously simplify a navigation of a second object (e.g., a scalpel) into a first object (e.g., a body part).

[0031] In an embodiment, the method may further comprise the step: determining a current trajectory of the second object based on the received second positioning data, wherein the step S5 of determining display data may be based on the determined current trajectory of the second object, wherein the display data may comprise a representation of the current target trajectory, and wherein the representation of the current target trajectory may be superimposed on the 3D model and the free space.

[0032] The representation of the current trajectory of the second object may be a single arrow. In other words, the representation may be superimposed on the free space in the display such that user sees the model, the first object in the free space, the second object in the free space, and a virtual representation of the second object in the free space. In combination with a representation of a target trajectory (as described above), this may be advantageous as it simplifies a navigation of a second object into a first object.

[0033] In an embodiment, the method may further comprise the step: receiving fifth positioning data of a third object, wherein the step S5 of determining display data may be based on the received fifth positioning data of a third object, wherein the display data may comprise a representation of the third object, and wherein the representation of the third object may be superimposed on the 3D model and the free space.

[0034] The third object may be an implant or the like. The fifth positioning data may be received from a data base storing respective planning data. The fifth positioning data may be received via an HMI (e.g., a physician inputs a desired position for an implant). The representation may be an arrow or the like. In other words, a virtual representation of the third object may be superimposed over the model and the corresponding free space.

[0035] In an embodiment, a shape of the free space may have a fixed radius.

[0036] The free space may be a 2D or a 3D space. In case the space is a 2D free space, the free space has a fixed circle shape. In case the space is a 3D free space, the free space has a fixed ball shape. The fixed ball is advantageous in comparison to the fixed circle shape as it doesn’t have to be adapted for different viewing angles of a user looking through the semi-transparent see through viewing device. The fixed radius may be chosen in dependency of one or more second objects. The fixed radius may be a simple but efficient way to implement the method.

[0037] In an embodiment, a shape of the free space depends on a geometry of the first object and / or the second object. The free space may not be a predefined free space with a fixed radius. Instead, the free space may take into account the geometry of the second object (e.g., in case the second object has a large tip, the free space is larger as in case the second object has a small tip). The free space may take into account the geometry of the first object (e.g., the first object is an oblong shaped spine, the free space may have an oblong shape). The dependency of the free space from the geometry of the first object and / or second object may increase an efficiency during navigation of the second object in relation to the first object.

[0038] In an embodiment, a shape of the free space may depend on a speed of the second object.

[0039] The speed of the second object may be determined by analysing a change of the second positioning data and a corresponding time. The speed may be determined by acceleration sensor applied to the second object. For example, in case a high speed is determined, the shape of the free space may be enlarged and vice versa. This may be advantageous as it reduces the risk of an unwanted collision during a fast navigation of the second object in relation to the first object, since a bigger part of the first object, for example, is seen compared to a free space that has a smaller spatial extension.

[0040] In an embodiment, the display data may comprise a transition area surrounding the free space, and wherein the transition area may comprise a fading effect.

[0041] The transition area may relate to a layer with a constant layer width. The fading effect may relate to a transition from transparent to coloured, wherein coloured relates to pixels adapted to display the model. The fading effect may improve a representation of the free space, the model and the first object. The fading effect may also contribute to enhance the depth effect and / or increase the distinguishability between real world and virtual representation. In an embodiment, the step S5 of determining display data may comprise a sequence of rendering, wherein the sequence of rendering may be: generating pixel data for the 3D model, generating pixel data for the free space, optionally generating pixel data for the representation of the target trajectory of the second object, optionally generating pixel data for the representation of the current trajectory of the second object, and optionally generating pixel data for the representation of the third object.

[0042] The above outlined sequence may advantageously reduce a programming effort as no sectional models (i.e. cut model) have to be calculated. Instead, the outlined sequence of the model, the free space, the representation of the target trajectory of the second object, the representation of the current trajectory and the representation of the third object is rendered. This allows to simply superimpose the model, the free space and the one or more representations onto the first object and the second object (i.e. physical reality).

[0043] A further aspect of the present disclosure relates to a data processing apparatus comprising means for carrying out the steps of the method described above.

[0044] A further aspect of the present disclosure relates to a computer program comprising instructions, which, when the program is executed by a computer, cause the computer to carry out the method described above.

[0045] This program can be seen as a computer program element and may be part of an existing computer program, but it can also be an entire program by itself. For example, the program may be used to update an already existing computer program to get to the present invention.

[0046] In this aspect, the invention is directed to a computer program which, when running on at least one processor (for example, a processor) of at least one computer (for example, a computer) or when loaded into at least one memory (for example, a memory) of at least one computer (for example, a computer), causes the at least one computer to perform the above-described method according to the first aspect. The invention may alternatively or additionally relate to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the steps of the method according to the first aspect. A computer program stored on a disc is a data file, and when the file is read out and transmitted it becomes a data stream for example in the form of a (physical, for example electrical, for example technically generated) signal. The signal can be implemented as the signal wave which is described herein. For example, the signal, for example the signal wave is constituted to be transmitted via a computer network, for example LAN, WLAN, WAN, for example the internet. The invention according to this aspect therefore may alternatively or additionally relate to a data stream representative of the aforementioned program.

[0047] A further aspect relates to a computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method described above.

[0048] Thus, this aspect of the invention is directed to a non-transitory computer- readable program storage medium on which the program according to the previous aspect is stored. The computer readable medium may be seen as a storage medium, such as for example, a USB stick, a CD, a DVD, a data storage device, a hard disk, or any other medium on which a program element as described above can be stored.

[0049] A further aspect of the present disclosure relates to a system for providing display data for a semi-transparent see through viewing device, the system comprising: a semi-transparent see through viewing device; a tracking unit configured to: -track first positioning data of a first object, second positioning data of a second object and third positioning data of the head mountable see through viewing device;

[0050] -provide the tracked first positioning data, second positioning data and third positioning data to a calculation unit; wherein the calculation unit is configured to:

[0051] - receive a model of the first object,

[0052] - determine a free space based on the second positioning data,

[0053] -determine display data for displaying on the semi-transparent see through device based on the model, the determined free space, the first positioning data, the second positioning data and the third positioning data,

[0054] -provide the determined display data for displaying on the semi-transparent see through viewing device; and wherein wherein the semi-transparent see through viewing device is configured to: display the determined display data.

[0055] The calculation unit may be a single entity or distributed on a plurality of entities. The tracking unit may be a single entity or distributed on a plurality of entities. The tracking unit may be an optical tracking unit or an electromagnetic tracking unit.

[0056] In an embodiment, the semi-transparent see through viewing device may be a head mountable device.

[0057] A last aspect of the present disclosure relates to a use of a semi-transparent see through viewing device in a method described above or in a system described above.

[0058] DEFINITIONS

[0059] In this section, definitions for specific terminology used in this disclosure are offered which also form part of the present disclosure. Computer implemented method

[0060] The method in accordance with the invention is for example a computer implemented method. For example, all the steps or merely some of the steps (i.e. less than the total number of steps) of the method in accordance with the invention can be executed by a computer (for example, at least one computer). An embodiment of the computer implemented method is a use of the computer for performing a data processing method. An embodiment of the computer implemented method is a method concerning the operation of the computer such that the computer is operated to perform one, more or all steps of the method.

[0061] The computer for example comprises at least one processor and for example at least one memory in order to (technically) process the data, for example electronically and / or optically. The processor being for example made of a substance or composition which is a semiconductor, for example at least partly n- and / or p-doped semiconductor, for example at least one of II-, III-, IV-, V-, Vl-sem iconductor material, for example (doped) silicon and / or gallium arsenide. The calculating or determining steps described are for example performed by a computer. Determining steps or calculating steps are for example steps of determining data within the framework of the technical method, for example within the framework of a program. A computer is for example any kind of data processing device, for example electronic data processing device. A computer can be a device which is generally thought of as such, for example desktop PCs, notebooks, netbooks, etc., but can also be any programmable apparatus, such as for example a mobile phone or an embedded processor. A computer can for example comprise a system (network) of "sub-computers", wherein each sub-computer represents a computer in its own right. The term "computer" includes a cloud computer, for example a cloud server. The term "cloud computer" includes a cloud computer system which for example comprises a system of at least one cloud computer and for example a plurality of operatively interconnected cloud computers such as a server farm. Such a cloud computer is preferably connected to a wide area network such as the world wide web (WWW) and located in a so-called cloud of computers which are all connected to the world wide web. Such an infrastructure is used for "cloud computing", which describes computation, software, data access and storage services which do not require the end user to know the physical location and / or configuration of the computer delivering a specific service. For example, the term "cloud" is used in this respect as a metaphor for the Internet (world wide web). For example, the cloud provides computing infrastructure as a service (laaS). The cloud computer can function as a virtual host for an operating system and / or data processing application which is used to execute the method of the invention. The cloud computer is for example an elastic compute cloud (EC2) as provided by Amazon Web Services™. A computer for example comprises interfaces in order to receive or output data and / or perform an analogue-to-digital conversion. The data are for example data which represent physical properties and / or which are generated from technical signals. The technical signals are for example generated by means of (technical) detection devices (such as for example devices for detecting marker devices) and / or (technical) analytical devices (such as for example devices for performing (medical) imaging methods), wherein the technical signals are for example electrical or optical signals. The technical signals for example represent the data received or outputted by the computer. The computer is preferably operatively coupled to a display device which allows information outputted by the computer to be displayed, for example to a user. One example of a display device is a virtual reality device or an augmented reality device (also referred to as virtual reality glasses or augmented reality glasses) which can be used as "goggles" for navigating. A specific example of such augmented reality glasses is Google Glass (a trademark of Google, Inc.). An augmented reality device or a virtual reality device can be used both to input information into the computer by user interaction and to display information outputted by the computer. Another example of a display device would be a standard computer monitor comprising for example a liquid crystal display operatively coupled to the computer for receiving display control data from the computer for generating signals used to display image information content on the display device. A specific embodiment of such a computer monitor is a digital lightbox. An example of such a digital lightbox is Buzz®, a product of Brainlab AG. The monitor may also be the monitor of a portable, for example handheld, device such as a smart phone or personal digital assistant or digital media player.

[0062] The invention also relates to a program which, when running on a computer, causes the computer to perform one or more or all of the method steps described herein and / or to a program storage medium on which the program is stored (in particular in a non-transitory form) and / or to a computer comprising said program storage medium and / or to a (physical, for example electrical, for example technically generated) signal wave, for example a digital signal wave, carrying information which represents the program, for example the aforementioned program, which for example comprises code means which are adapted to perform any or all of the method steps described herein.

[0063] Within the framework of the invention, computer program elements can be embodied by hardware and / or software (this includes firmware, resident software, micro-code, etc.). Within the framework of the invention, computer program elements can take the form of a computer program product which can be embodied by a computer-usable, for example computer-readable data storage medium comprising computer-usable, for example computer-readable program instructions, "code" or a "computer program" embodied in said data storage medium for use on or in connection with the instruction-executing system. Such a system can be a computer; a computer can be a data processing device comprising means for executing the computer program elements and / or the program in accordance with the invention, for example a data processing device comprising a digital processor (central processing unit or CPU) which executes the computer program elements, and optionally a volatile memory (for example a random access memory or RAM) for storing data used for and / or produced by executing the computer program elements. Within the framework of the present invention, a computer-usable, for example computer-readable data storage medium can be any data storage medium which can include, store, communicate, propagate or transport the program for use on or in connection with the instruction-executing system, apparatus or device. The computer-usable, for example computer-readable data storage medium can for example be, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared or semiconductor system, apparatus or device or a medium of propagation such as for example the Internet. The computer-usable or computer-readable data storage medium could even for example be paper or another suitable medium onto which the program is printed, since the program could be electronically captured, for example by optically scanning the paper or other suitable medium, and then compiled, interpreted or otherwise processed in a suitable manner. The data storage medium is preferably a non-volatile data storage medium. The computer program product and any software and / or hardware described here form the various means for performing the functions of the invention in the example embodiments. The computer and / or data processing device can for example include a guidance information device which includes means for outputting guidance information. The guidance information can be outputted, for example to a user, visually by a visual indicating means (for example, a monitor and / or a lamp) and / or acoustically by an acoustic indicating means (for example, a loudspeaker and / or a digital speech output device) and / or tactilely by a tactile indicating means (for example, a vibrating element or a vibration element incorporated into an instrument). For the purpose of this document, a computer is a technical computer which for example comprises technical, for example tangible components, for example mechanical and / or electronic components. Any device mentioned as such in this document is a technical and for example tangible device.

[0064] Acquiring data

[0065] The expression "acquiring data" or determining data for example encompasses (within the framework of a computer implemented method) the scenario in which the data are determined by the computer implemented method or program. Determining data for example encompasses measuring physical quantities and transforming the measured values into data, for example digital data, and / or computing (and e.g. outputting) the data by means of a computer and for example within the framework of the method in accordance with the invention. The meaning of "acquiring data" also for example encompasses the scenario in which the data are received or retrieved by (e.g. input to) the computer implemented method or program, for example from another program, a previous method step or a data storage medium, for example for further processing by the computer implemented method or program. Generation of the data to be acquired may but need not be part of the method in accordance with the invention. The expression "acquiring data" can therefore also for example mean waiting to receive data and / or receiving the data. The received data can for example be inputted via an interface. The expression "acquiring data" can also mean that the computer implemented method or program performs steps in order to (actively) receive or retrieve the data from a data source, for instance a data storage medium (such as for example a ROM, RAM, database, hard drive, etc.), or via the interface (for instance, from another computer or a network). The data acquired by the disclosed method or device, respectively, may be acquired from a database located in a data storage device which is operably to a computer for data transfer between the database and the computer, for example from the database to the computer. The computer acquires the data for use as an input for steps of determining data. The determined data can be output again to the same or another database to be stored for later use. The database or database used for implementing the disclosed method can be located on network data storage device or a network server (for example, a cloud data storage device or a cloud server) or a local data storage device (such as a mass storage device operably connected to at least one computer executing the disclosed method). The data can be made "ready for use" by performing an additional step before the acquiring step. In accordance with this additional step, the data are generated in order to be acquired. The data are for example detected or captured (for example by an analytical device). Alternatively or additionally, the data are inputted in accordance with the additional step, for instance via interfaces. The data generated can for example be inputted (for instance into the computer). In accordance with the additional step (which precedes the acquiring step), the data can also be provided by performing the additional step of storing the data in a data storage medium (such as for example a ROM, RAM, CD and / or hard drive), such that they are ready for use within the framework of the method or program in accordance with the invention. The step of "acquiring data" can therefore also involve commanding a device to obtain and / or provide the data to be acquired. In particular, the acquiring step does not involve an invasive step which would represent a substantial physical interference with the body, requiring professional medical expertise to be carried out and entailing a substantial health risk even when carried out with the required professional care and expertise. In particular, the step of acquiring data, for example determining data, does not involve a surgical step and in particular does not involve a step of treating a human or animal body using surgery or therapy. In order to distinguish the different data used by the present method, the data are denoted (i.e. referred to) as "XY data" and the like and are defined in terms of the information which they describe, which is then preferably referred to as "XY information" and the like.

[0066] Registering

[0067] The n-dimensional image of a body is registered when the spatial location of each point of an actual object within a space, for example a body part in an operating theatre, is assigned an display data point of an image (CT, MR, etc.) stored in a navigation system.

[0068] Image registration

[0069] Image registration is the process of transforming different sets of data into one co-ordinate system. The data can be multiple photographs and / or data from different sensors, different times or different viewpoints. It is used in computer vision, medical imaging and in compiling and analysing images and data from satellites. Registration is necessary in order to be able to compare or integrate the data obtained from these different measurements. Marker

[0070] It is the function of a marker to be detected by a marker detection device (for example, a camera or an ultrasound receiver or analytical devices such as CT or MRI devices) in such a way that its spatial position (i.e. its spatial location and / or alignment) can be ascertained. The detection device is for example part of a navigation system. The markers can be active markers. An active marker can for example emit electromagnetic radiation and / or waves which can be in the infrared, visible and / or ultraviolet spectral range. A marker can also however be passive, i.e. can for example reflect electromagnetic radiation in the infrared, visible and / or ultraviolet spectral range or can block x-ray radiation. To this end, the marker can be provided with a surface which has corresponding reflective properties or can be made of metal in order to block the x-ray radiation. It is also possible for a marker to reflect and / or emit electromagnetic radiation and / or waves in the radio frequency range or at ultrasound wavelengths. A marker preferably has a spherical and / or spheroid shape and can therefore be referred to as a marker sphere; markers can however also exhibit a cornered, for example cubic, shape.

[0071] BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In the following, the invention is described with reference to the appended figures which give background explanations and represent specific embodiments of the invention. The scope of the invention is however not limited to the specific features disclosed in the context of the figures, wherein

[0073] Figure 1 shows an exemplary view of a superimposed model to an anatomical body part;

[0074] Figure 2 shows a schematic view of the method for providing display data for a semi-transparent see through viewing device in a medical environment; Figure 3 shows an exemplary view of a physician using a head mounted semi-transparent see through viewing device with the method for providing display data described above; and

[0075] Figure 4 shows an exemplary system for providing display data for a head mountable semi-transparent see through viewing device.

[0076] DESCRIPTION OF EMBODIMENTS

[0077] Figure 1 shows an exemplary view of a superimposed model 11 to an anatomical body part 10. The view may exemplarily relate to a view of a head mounted semi-transparent see through viewing device (e.g. a mixed reality glasses). The view comprises a real physical view on the anatomical body part 10. A model 11 of the spine is superimposed onto the physical anatomical body part 10. The view shows real physical parts and virtual parts. It can be seen that the model 11 hides the skin of the anatomical body part 10. A marker 12 is in the present example attached to the anatomical body part 10. The marker 12 comprises four physical marker spheres 12a. Further, the view comprises four virtual marker spheres 12b, as shown in Figure 1 , the four virtual marker spheres 12b being the augmented counterparts of the physical marker spheres 12a. The four virtual marker spheres 12b are augmented by the head mounted semi-transparent see through viewing device. By means of the marker 12 the position of the anatomical body part 10 can be acquired by a tracking system (not shown). By means of registering, the model 11 is superimposed onto the anatomical body part 10. Virtual representations 13 to 17 of target trajectories of instrument (e.g. a drilling sleeve) and implants are superimposed to the model 11 . These virtual representations that are presented as lines assist a physician to carry out a task (e.g. drilling a hole into the spine in order to place an implant). As can be seen from the view, a direct view onto the physical anatomical body part 10 is not available. The model 11 hides a contact between for example a drilling sleeve and the anatomical body part 10. This could be disadvantageous for a physician as he doesn’t see where the instrument is right now. The invention solves this problem by providing a so called free space in the model that is rendered transparent such that the physician has a free view in the area of the used instrument but can also see the model surrounding the free space and optionally also one or more virtual representations of target trajectories of instruments and / or implants or current trajectories of instruments and / or implants.

[0078] Figure 2 shows a schematic view of the method for providing display data for a semi-transparent see through viewing device in a medical environment. The medical environment may be a surgery room. The method comprises six steps S1 to S6. The method may be carried out by a data processing apparatus such as computer with one or more interfaces configured to exchange data with peripherals.

[0079] In step S1 a model of a first object is provided. The may be a 2D model or 3D model. In the present example, the model is a 3D model. The model is derived from one or more CT images acquired in previous CT imaging procedures. Alternatively or additionally the model may derived from one or more MRT images. The first object is in this example a spine of a human. Alternatively, the first object may be any other part of a human or an animal.

[0080] Step S2 comprises receiving positioning data from the first object. In the present example, a marker is attached to the first object. A tracking system (e.g. an optical tracking with one or more cameras) tracks the position of the first object. Alternatively or additionally, the tracking system may work without marker based on an optical tracking system or an electromagnetic tracking system.

[0081] Step S3 comprises receiving second positioning data of a second object. The second object is in the present example a drilling sleeve. The second object is in the present example tagged with a marker. An optical tracking system tracks the marker of the second object. Alternatively or additionally, the tracking system may work without a marker based on an optical tracking system or an electromagnetic tracking system. Step S4 comprises acquiring a free space based on the second positioning data. In this example, the free space comprises a shape of a ball with a fixed radius. The free space is exemplarily determined by the positioning data of the second object and the ball with the fixed radius. Alternatively, the free space may be determined by considering a geometry of the first object and the second object. Alternatively the free space may be acquired based on a speed of the second object.

[0082] Step S5 comprises determining display data for displaying based on the model, the acquired free space, the first positioning data and the second positioning data. The determining of the display data exemplarily may comprise registering the model onto the first object based on the first positioning data. It may further comprise registering the free space onto the second positioning data. Additionally, a position of the semi-transparent see through viewing device may be determined such that a viewing angle of the semi-transparent see through viewing device onto the first object and / or second object can be determined. Based on the registered model and the registered free space, display data is rendered for each glass of the head mounted semi-transparent see through viewing device (i.e. mixed reality glasses). The rendering may exemplarily be carried out in the following sequence: model and free space. The free space may be rendered transparent.

[0083] Step S6 comprises providing the determined display data for displaying on the semi-transparent see through viewing device. The determined display data may be transmitted by means of an interface to the semi-transparent see through viewing device. Based on the determined display data the plurality of pixels of the two glasses of the augmented reality device may be displayed in colour or transparent such that a user can see the model superimposed onto the first object and the free space transparent such that he can see the second object. Figure 3 shows an exemplary view of a physician using a head mounted semitransparent see through viewing device with the method for providing display data described above in a medical environment. The view shown through the head mounted semi-transparent see through viewing device comprises a spine 50 as a first object. The spine 50 is tagged with a marker 51 that is used for determining the position of the first object. The view comprises a model 52 of the spine superimposed onto the spine 50. The view further comprises the second object, an instrument 53 held by the physician. The instrument 53 is also tagged with a marker in order to determine the second positioning data of the second object. The view comprises the free space 54. The free space 54 is rendered transparent such that the physician can see onto the skin (in case of a minimal-invasive surgery) over the spine and also the instrument. In case of an open surgery, the free space 54 is rendered transparent such that the physician can see onto the spine and also the instrument. The border of the free space shows a fading effect 55. Additionally, a current trajectory 56 of the instrument and a target trajectory 57 of the instrument are shown. It should be noted that the view is continuously adapted dependent on the received first and second positioning data as well as the viewing angle of the head mounted semi-transparent see through viewing device. The view shows additionally some CT images 60 to 62 from planning data corresponding to the case. The view shows additionally a virtual representation 58 of a third object and a virtual representation 59 of a further object. The third object and the further object may be implants to be set into the spine. The virtual representation are displayed as lines.

[0084] Figure 4 shows an exemplary system 100 for providing display data for a head mountable semi-transparent see through viewing device 101. A user 104 wears the head mountable semi-transparent see through viewing device 101 . The system 100 comprises a tracking unit 102. The tracking unit 102 is configured track the first positioning data of a first object 103, second positioning data of a second object 105 and third positioning data of the head mountable semi-transparent see through viewing device 101. The system comprises further a calculation unit 110. In the present example, the calculation unit 110 is embedded in the head mountable semi-transparent see through viewing device 101 . The tracking unit 102 is configured to provide the tracked first positioning data, second positioning and third positioning data to the calculation unit. In the present example, the head mountable semitransparent see through viewing device 101 is part of the tracking unit 102, as it determines its own positioning data (e.g. by means of a simultaneous localization and mapping algorithm (slam) algorithm). The calculation unit 110 is configured to receive a 3D model of the first object 103 and to acquire a free space based on the second positioning data. The calculation unit 110 is configured to determine display data for displaying on the head mountable semi-transparent see through device 101 based on the 3D model, the determined free space, the first positioning data, the second positioning data and the third positioning data. The calculation unit is further configured to provide the determined display data for displaying on the head mountable semi-transparent see through viewing device 101. The head mountable semitransparent see through viewing device 101 is configured to display the determined display data.

[0085] REFERENCE SIGNS

[0086] 10 anatomical body part

[0087] 11 model, in particular 2D and / or 3D model

[0088] 12 marker

[0089] 12a physical marker sphere

[0090] 12b virtual marker sphere

[0091] 13 to 17 virtual representations

[0092] 50 first object

[0093] 51 marker

[0094] 52 model

[0095] 53 second object

[0096] 54 free space

[0097] 55 fading effect

[0098] 56 virtual representation of current trajectory

[0099] 57 virtual representation of target trajectory

[0100] 58, 59 virtual representation of third object

[0101] 60 to 62 CT images from intraoperative data or planning data

[0102] 100 system

[0103] 101 head mountable semi-transparent see through viewing device

[0104] 102 tracking unit

[0105] 103 first object

[0106] 104 user

[0107] 105 second object

[0108] 110 calculation unit

[0109] S1 providing a model

[0110] S2 receiving first positioning data

[0111] S3 receiving second positioning data

[0112] S4 acquiring a free space

[0113] S5 determining display data

[0114] S6 providing display data

Claims

Claims1. Computer implemented method for providing display data for a semitransparent see through viewing device in a medical environment, the method comprising the steps: providing a model of a first object (step S1 ); receiving first positioning data of the first object (step S2); receiving second positioning data of a second object (step S3); acquiring a free space based on the second positioning data (step S4); determining display data for displaying on the semi-transparent see through viewing device based on the model, the acquired free space, the first positioning data and the second positioning data (step S5); and providing the determined display data for displaying on the semi-transparent see through viewing device (step S6).

2. The method according to claim 1 , wherein the semi-transparent see through viewing device is a head mountable device; and wherein the method further comprises the step: obtaining third positioning data of the head mountable device and determining the display data based on the obtained third positioning data.

3. The method according to claim 1 or 2, wherein the semi-transparent see through viewing device is a movable display; and wherein the method further comprises the step: obtaining fourth positioning data of the movable display and determining the display data based on the obtained fourth positioning data.

4. The method according to any one of the preceding claims, wherein the free space is rendered transparent on a display of the semitransparent see through viewing device.

5. The method according to any one of the preceding claims, further comprising the step: receiving a target trajectory of the second object; wherein the step S5 of determining display data is based on the received target trajectory of the second object, wherein the display data comprises a representation of the target trajectory, and wherein the representation of the target trajectory is superimposed on the model and the free space.

6. The method according to any one of the preceding claims, further comprising the step: determining a current trajectory of the second object based on the received second positioning data, wherein the step S5 of determining display data is based on the determined current trajectory of the second object, wherein the display data comprises a representation of the current target trajectory, and wherein the representation of the current target trajectory is superimposed on the model and the free space.

7. The method according to any preceding claims, further comprising the step: receiving fifth positioning data of a third object, wherein the step S5 of determining display data is based on the received fifth positioning data of the third object, wherein the display data comprise a representation of the third object, and wherein the representation of the third object is superimposed on the model and the free space.

8. The method according to any one of the preceding claims, wherein a shape of the free space has a fixed radius.

9. The method according to any one of the preceding claims, wherein a shape of the free space depends on a geometry of the first object and / or the second object.

10. The method according to any one of the preceding claims, wherein a shape of the free space depends on a speed of the second object.11 . The method according to any one of the preceding claims, wherein the display data comprises a transition area surrounding the free space, and wherein the transition area comprises a fading effect.

12. The method according to any one of the claims, wherein the step S5 of determining display data comprises a sequence of rendering, wherein the sequence of rendering is: generating pixel data for the model, generating pixel data for the free space, optionally generating pixel data for the representation of the target trajectory of the second object, optionally generating pixel data for the representation of the current trajectory of the second object, and optionally generating pixel data for the representation of the third object.

13. A data processing apparatus comprising means for carrying out the steps of the method according to any one of the claims 1 to 12.

14. A computer program comprising instructions, which, when the program is executed by a computer, cause the computer to carry out the method according to any one of the claims 1 to 12.

15. A computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to carry out the method according to any one of the claims 1 to 12.

16. A system for providing display data for a semi-transparent see through viewing device, the system comprising: a semi-transparent see through viewing device; a tracking unit configured to:- track first positioning data of a first object, second positioning data of a second object and third positioning data of the see through viewing device;- provide the tracked first positioning data, second positioning data and third positioning data to a calculation unit; wherein the calculation unit is configured to:- receive a model of the first object,- determine a free space based on the second positioning data,- determine display data for displaying on the semi-transparent see through device based on the model, the determined free space, the first positioning data, the second positioning data and the third positioning data,- provide the determined display data for displaying on the semitransparent see through viewing device; and wherein the semi-transparent see through viewing device is configured to:- display the determined display data.

17. The system according to claim 16, wherein the semi-transparent see through viewing device is a head mountable device.

18. Use of a semi-transparent see through viewing device in a method according to any one of the claims 1 to 12 or in a system according to claims 16 or 17.

Images