Head mounted display system and method for controlling a medical imaging device

Abstract

The present invention relates to a head mounted display system (1) for controlling a medical imaging device (20), comprising: a head mounted display (10) to be worn by an operator, the head mounted display comprising a display screen (12) configured to display a medical image having a predetermined viewing center (121) and a command menu (122) providing at least a first set of commands (123), the first set of commands (123) being a differentiated first command field; a tracking system (14) for tracking movements representing eye movements and / or head movements of an operator (30), wherein commands in the at least the first set of commands (123) of the command menu (122) are selectable based on the tracked eye movements and / or head movements; and a control device (40) configured to receive the selected command and control the medical imaging device (20) accordingly.

Description

Technical Field

[0001] The present invention relates to a head mounted display system for controlling a medical imaging device, a method for controlling a medical imaging device and a computer program product for executing the method. Background Art

[0002] Medical imaging devices are often used to support surgeons during surgery by providing insight into or magnified images of the operating area and / or to record relevant images. However, during the surgical procedure, it may be necessary to adjust the viewing center, viewing angle, magnification, or other settings of the medical imaging device. Such manual adjustments require operator intervention, which may lead to interference with sterility, coordination of the operation, or distraction of the surgeon.

[0003] Different approaches have been proposed to minimize the disturbances caused by manual adjustments of medical imaging devices. For example, US 4,989,253 A discloses a microscope with a voice-activated control system that allows precise positioning of the microscope and precise focusing by means of voice commands issued by the microscope operator. Although manual intervention can be avoided, problems arise in the precise determination of the corresponding movement path by the surgeon and due to the need to use a predetermined vocabulary. In particular, voice commands can be accidentally initiated during communication between surgical personnel.

[0004] Alternatively, DE 4202505B4 proposes a non-contact guidance system for spatially positioning a surgical microscope by directly moving the surgical microscope according to the operator's head movement. However, such a system only allows adjustment of the surgical microscope that directly corresponds to the operator's head movement. In addition, an unexpected or erroneous movement of the operator's head may cause the surgical microscope to move with it, which may have serious consequences.

[0005] It is therefore an object of the present invention to provide control of a medical imaging device in a contactless manner with enhanced control options while minimizing the risk of initiating unintended commands. Summary of the invention

[0006] This object is solved by a head mounted display system for controlling a medical imaging device, a method for controlling a medical imaging device and a computer program product for executing the method as follows.

[0007] According to one aspect, a head-mounted display system for controlling a medical imaging device includes: a head-mounted display to be worn by an operator, the head-mounted display including a display screen configured to display a medical image with a predetermined observation center and a command menu providing at least a first set of commands, the first set of commands being a distinct first command field; a tracking system for tracking movements representing eye movements and / or head movements of the operator, wherein commands in the at least the first set of commands of the command menu are selectable based on the tracked eye and / or head movements; and a control device configured to receive the selected command and control the medical imaging device accordingly.

[0008] The medical image displayed in the display screen of the head-mounted display corresponds to the image received by the medical imaging device or at least a portion thereof. The predetermined observation center can be geometrically predetermined, for example, as the center of the image displayed in the display screen, or the predetermined observation center can be predetermined based on the current observation angle of the operator wearing the head-mounted display. Considering the observation angle perpendicular to the display screen or another predetermined observation angle with the display screen corresponding to the orientation of the usual operator, the observation angle can be derived from the tracked eye movement or the observation angle can be assumed by the position of the head-mounted display. Alternatively or in addition, the observation center can be determined, for example, based on a marker, taking into account the operating area or a specific part thereof. If, in addition to other aspects, the operating area or a specific part thereof is also taken into account for determining the predetermined observation center, the predetermined observation center can be defined, for example, according to the operator's observation angle, as long as the observation angle does not cause the observation center to exceed a predetermined distance from the operating area or a specific part thereof.

[0009] Next to the medical image, the display screen of the head mounted display displays a command menu, which provides at least a first set of commands for a first command field that is different. The command menu can be superimposed on the medical image or displayed in a separate portion of the display screen. In the case of a separate display portion for the command menu or at least a portion thereof, the medical image is not hidden by masking the command field. However, the superposition of the command menu or at least a portion thereof can be beneficial for saving display space of the medical image and / or reducing the distance to a predetermined observation center to select a command as described later. In such a case, the command menu or at least a portion thereof is arranged outside the predetermined observation center to minimize the interference of the operator when displayed. In one embodiment, the head mounted display system can be configured to switch between two modes, namely, a superposition mode and a separate display portion mode.

[0010] The command menu of at least a first set of commands provided as differentiated command fields may include command fields that control movement of the medical imaging device, medical imaging settings (such as magnification, brightness and / or contrast), and / or recording functions. Thus, the head mounted display system can not only enable control of movement of the medical imaging device to change the viewing center and / or viewing angle, but also allow settings that do not naturally correspond to movement of the operator's eyes or head.

[0011] A tracking system may be incorporated into the head mounted display that tracks movements representative of the operator's eye and / or head movements. Such a tracking system may include a GPS based tracker, an accelerometer, or an optical or electromagnetic tracker for head movements. For example, an optical tracker may include two cameras attached to the head mounted display that track head movements with reference to defined reference points in an operating room. Movements of the operator's eyes may be tracked by the cameras. Alternatively or additionally, the tracking system may be separate from the head mounted display, for example for redundancy reasons.

[0012] Advantageously, the tracking system is configured to track movements representing eye and / or head movements of the operator independently of an optical transmission path that may be interrupted by obstacles. Furthermore, movements representing eye and / or head movements of the operator involve tracking any object whose movement corresponds to the movement of the eye and / or head of the operator. Thus, the object that moves corresponding to the operator's head may be not only the operator's head itself, but may also be, for example, a head-mounted display or a portion thereof, such as a display screen, or may be other components fixed to the operator's head. In terms of the movement of the operator's eyes, such movement may be represented by tracking the movement of one or both eyes of the operator. Thus, if there is no need to distinguish between one eye and multiple eyes, the terms one eye and multiple eyes may be used synonymously.

[0013] Tracking movements representing head movements provides the advantage of selecting commands from the at least first set of commands of the command menu without having to move the eyes away from the operating area. Alternatively or additionally, movements representing eye movements of the operator can be used to select commands from the at least first set of commands of the command menu. This type of selection may be advantageous if head movements are hindered or limited, for example due to ergonomic reasons.

[0014] The selection of commands from at least a first set of commands of the command menu is based on movement representative of eye and / or head movement of the operator tracked by converting the direction and amount of movement at a position on the tracked display screen. If such a position corresponds to one of the differentiated first command fields, the corresponding command is selected. Therefore, the control device of the head mounted display system is configured to directly or indirectly receive the selected command by receiving and converting the tracked movement. Depending on the selected command, the control device controls the medical imaging device according to the selected command. The control performed by the control device is not limited to direct control of the medical imaging device. For example, the control device can be configured to control a robot to move a medical imaging device attached to such a robot.

[0015] The control device may be incorporated into the head mounted display or the medical imaging device, or may be provided as a separate control device, for example, provided as a personal computer that communicates with the head mounted display and the medical imaging device. Separating the control device from the head mounted display or the medical imaging device can reduce the weight of the corresponding components to be worn or moved, respectively.

[0016] In one embodiment, the distinct first command fields of the at least first group of commands of the command menu displayed on the display screen of the head mounted display are arranged at substantially equal first distances from a predetermined viewing center of the medical image.

[0017] Due to the substantially equal first distance from the predetermined observation center, the tracked movement to one of the different first command fields can be limited compared to the command fields arranged in series in a radial direction from the predetermined observation center. In addition, the arrangement of the first command fields within the substantially equal first distance from the predetermined observation center can allow tracking of the operator's head movement while still providing the ability to guide the eyes to the operating area. If the eye movement is tracked, the operating area may still be in the field of view. Therefore, since neither the operator's hand nor the operator's line of sight have to leave the operating area, control of the medical imaging device can be performed without interrupting the medical procedure.

[0018] The term "substantially equal distances" refers to distances which deviate from each other only with a small tolerance. Such a small tolerance does not affect the advantages mentioned above. The ideal arrangement is a circular arrangement of the differentiated first command fields, with a predetermined observation center as the center of the circle. However, the circular arrangement can also be slightly deformed, i.e. slightly elliptical, with slightly larger distances in the direction of the increased mobility of the operator's head or eyes. Thus, the slightly larger distances correspond to movements of the operator's head or eyes in the left and right directions relative to the operator's longitudinal axis or front-to-back axis.

[0019] In some embodiments, the command menu provides a second set of commands, which are displayed on the display screen of the head-mounted display and are preferably arranged at a second distance substantially equal to the predetermined observation center of the medical image, wherein the second distance of the differentiated second command field is greater than the first distance of the differentiated first command field.

[0020] Thus, the second set of command fields can be arranged and / or displayed independently of the first set of command fields. In particular, the differentiated second command fields of the second set of commands are arranged at a substantially equal second distance from a predetermined viewing center of the medical image to provide similar advantages as arranging the differentiated first command fields at a substantially equal first distance. By arranging the differentiated second command fields at a substantially equal second distance from the predetermined viewing center, wherein the second distance of the differentiated second command fields is greater than the first distance of the differentiated first command fields, the second set of commands surrounds the first set of commands when displayed. Preferably, the second set of commands is close to or adjacent to the first set of commands to provide the capability of the shortest distance and therefore provide movement to be tracked.

[0021] According to the present disclosure, the control device can be configured to deactivate the differentiated first command field when the tracking system indicates that the tracked movement has passed through one of the differentiated first command fields to reach one of the differentiated second command fields in a direction away from a predetermined observation center of the medical image.

[0022] Deactivation of the differentiated first command fields can prevent accidental selection of one of the differentiated first command fields when the tracked movement representing the operator's head or eye movement travels back through such fields. In other words, selection of the command fields is limited to one of the differentiated second command fields after the tracking system has indicated that the tracked movement has passed through one of the differentiated first command fields to one of the differentiated second command fields in a direction away from the intended viewing center of the medical image.

[0023] Advantageously, the control means is configured to distribute the commands to the first and second sets of commands based on a frequency and / or probability of use.

[0024] In terms of independently displaying the first and second sets of commands, for example, only displaying the second set of commands as needed or in certain circumstances, and / or arranging the differentiated second command fields at a substantially equal second distance from a predetermined observation center, the substantially equal second distance being greater than the substantially equal first distance, the first set of commands preferably includes functions that are used with a higher frequency of use and / or probability of use. Thus, in terms of the arrangement of the first and second sets of commands at different distances, most movements can be kept at a reduced level of movement. If the second set of commands is only displayed as needed or in certain circumstances, the second set of commands will not obscure the medical image when superimposed.

[0025] According to the present disclosure, the control device may be configured to adjust the distribution of commands to the first set of commands and the second set of commands based on a changed frequency of use and / or probability of use.

[0026] The adjustment of the distribution of commands to the first and second groups of commands may be input via the input means. Preferably, the control means is configured to automatically adjust the distribution, for example by analyzing the selected commands within a predetermined time or after a predetermined number of selected commands have been executed. The analysis may be based on determining a differential frequency of the selected commands or further taking into account weighting factors.

[0027] In some embodiments, the command menu further includes a confirmation field arranged in a predetermined viewing center of the medical image to confirm the selected command field.

[0028] Since the operator may perform the tracked movement and thereby may accidentally select the command field, the confirmation field prevents the execution of such a command. Here, the control device is configured to control the medical imaging device only according to the selected command if the tracked movement further corresponds to the selection of the confirmation field after the command has been selected. In addition, the confirmation field allows the execution of the command on demand, i.e., the command does not have to be executed immediately after its selection, but only when the confirmation field is selected. In particular, the command in such a configuration is only executed when the tracked movement representing the operator's head or eye movement reaches a predetermined observation center. In such a case, it must be assumed that the operator's line of sight can be directed to the predetermined observation center to mitigate the risk of initiating the execution of the command without visual control of the operating area.

[0029] Alternatively or additionally, the selected command field may be confirmed if no further movement representative of the operator's eye or head movement is tracked within a predetermined time frame or does not correspond to another command field. Alternatively, the control device is configured to control the medical imaging device according to the first selected command field.

[0030] Advantageously, the head mounted display system further comprises an input device, preferably a foot switch, which has to be activated as a prerequisite for selecting one of the command fields and / or displaying the command menu.

[0031] Since the medical imaging device does not need to be continuously controlled, the risk of accidentally selecting a command is further reduced if such a selection is only issued when the input device is activated. Advantageously, the tracking system does not track any movement as long as the input device is not activated, so as to avoid unnecessary data collection. In addition, the command menu can be displayed as needed only by activating the input device, so as to prevent the medical image from being obscured by the command menu when superimposed, or to display the medical image over the entire display area without reserving space for a separate display portion.

[0032] Using a foot switch as input device provides the option of activation without compromising sterility. Alternatively, the input device may be any other kind of switch that can be operated by a member of the surgical team taking into account sterility aspects and authorization.

[0033] According to the present disclosure, an input device may be configured to be activated only as long as a continuous activation action is applied.

[0034] For example, a command may only be selected by pressing a foot switch as input device. If the activation action is revoked or falls below a predetermined level, the ability to select commands is deactivated and the command menu may not be displayed further. Furthermore, the control device may be configured to reset the display settings after no further activation actions have been applied, i.e. the display settings for the command menu return to a default state. Alternatively, the last display settings may be stored and the command menu may be displayed with these settings the next time the input device is activated.

[0035] In some embodiments, the head mounted display system further includes a monitoring device configured to monitor the position of the display screen and issue a selection of one of the command fields and / or display a command menu only in a position of the display screen that represents an operator's use of the display screen.

[0036] Such a monitoring device can be incorporated into a head-mounted display. For example, the display screen of the head-mounted display can be movably supported by a hinge equipped with a sensor to detect the operating state of the display screen, i.e. the flip-down position of the display screen. Alternatively, the spatial position of the display screen can be detected and compared with the position of the operator's head. In one variant, the spatial position of the display screen can be detected and compared with a common or uncommon position of the operation, respectively, for example, the orientation of the display screen toward the ceiling of the operating room can be assumed to be an uncommon position, so that the selection of the command is not issued. In another variant, the monitoring device includes a contact switch that is activated only when the head-mounted display is at least worn by the operator. Although different monitoring options are described as alternatives, the combination has the potential to improve safety in terms of redundancy or consideration of multiple prerequisites.

[0037] Advantageously, the head mounted display system also provides different operating states and is configured to issue a selection of one of the command fields and / or display a command menu only in predetermined operating states.

[0038] The different operating states may be states representing non-surgical operations such as commissioning, maintenance, training, preparation, etc. Since no surgery is performed, these operating states may require fewer safety measures. However, in the case of surgery, different operating states with different safety levels may be available. In severe cases, the medical imaging device may not be allowed to move, and / or the medical image should be displayed without displaying any interfering command fields. Therefore, no command fields representing moving the medical device are issued, and / or the command menu or at least a portion thereof is not displayed.

[0039] According to the present disclosure, the tracking system may be configured to track movements representative of eye movements to control a viewing center of a medical image on a display screen, and to track movements representative of head movements to select one of the command fields.

[0040] The movement representing the eye movement controls the predetermined observation center of the medical image on the display screen so as to correspond to the observation angle in a predetermined relationship, preferably in accordance with the observation center of the operator. Therefore, the observation center of the medical image follows the eye movement without selecting a command area. However, such tracking can be issued on demand based on the tracked head movement by preselecting the corresponding command field. The control of the observation center of the medical image can be performed by the movement of the medical imaging device, and / or the control of the observation center of the medical image can be performed by moving the observation center of the medical image without moving the medical imaging device but moving the medical image. Therefore, even if the operator's head is moved to select one of the command fields, the operator can observe the predetermined observation center of the medical image. In addition, since certain functions can be controlled by eye movement without selecting the command field of the command menu, the number of command fields can be reduced.

[0041] In some embodiments, the head mounted display is configured to: display a visual indication in the display screen representing the tracked movement; and / or highlight a selected one of the command fields and / or highlight a confirmation field.

[0042] The visual indication may be a cursor-like item displayed in the display screen of the head mounted display to provide visual control. Alternatively or additionally, the selected command field and / or confirmation field may be highlighted to provide feedback to the operator upon such selection. The highlighting of the selected command field and / or confirmation field may be visual, auditory, and / or tactile. Such feedback enhances the operator's control capabilities and thus mitigates the risk of executing unintended commands.

[0043] According to the present disclosure, the medical imaging device may be a surgical microscope, preferably a robotic microscope.

[0044] The use of a head-mounted display system for controlling a surgical microscope enables the operator to perform surgery without moving his eyes and hands away from the operating area. This is particularly advantageous in view of surgical operations while using a surgical microscope, since changes in the position of the surgeon's hands and thus possible changes in the position of the surgical instruments not caused by the surgical operation itself should be avoided. Additionally, the surgical procedure can be performed and visually controlled while simultaneously controlling the microscope, for example changing the viewing angle or magnification. Furthermore, the risk of contamination is reduced.

[0045] Alternatively, the medical imaging device may be an endoscope.

[0046] Similar to the use of head-mounted displays for controlling microscopes, the use of head-mounted display systems for controlling endoscopes also allows the operator to perform surgery without moving his eyes and hands away from the operating area. Considering the advantages of the corresponding control of the microscope, in most endoscopic surgeries, if robot guidance is not used, the surgeon's hands must hold the surgical instruments in a relatively small and deep operating area. Therefore, removing the hands to control the endoscope may make it difficult to avoid further injuries caused by sharp surgical instruments and difficult to properly replace the instruments. Even if only one of the surgeon's hands is used to operate the surgical instruments, this hand may be needed to operate the endoscope. In such a case, the surgeon's hands cannot control the functions of the endoscope that are not incorporated in the handle of the endoscope or in close proximity without releasing the surgical instruments or endoscope. Although the endoscope or surgical instruments can be additionally supported or guided, for example, by a surgical robot, the surgeon's hands may be constrained by other instruments or control devices.

[0047] In a further aspect, the present disclosure relates to a method for controlling a medical imaging device, which includes a head mounted display system as described above, the method comprising the following steps:

[0048] a) Display the command menu,

[0049] b) tracking movements representing the operator's eye and / or head movements,

[0050] c) selecting one of the command fields in the command menu by means of tracked movements of the operator's eye and / or head movements, wherein, if only a first set of commands is displayed, the corresponding differentiated first command field is selected when the direction and amount of the corresponding movement corresponds to the position of one of the differentiated first command fields, or, if a first set of commands and a second set of commands are displayed, the corresponding differentiated second command field is selected when the direction and amount of the corresponding movement corresponds to the position of one of the differentiated second command fields, and the corresponding differentiated first command field is selected when the direction and amount of the corresponding movement corresponds to the position of one of the differentiated first command fields while no further movement exceeding a predetermined amount in the direction to one of the differentiated second command fields is tracked (123a), and

[0051] d) controlling the medical imaging device according to a command field selected from the command fields.

[0052] The direction and amount of movement can be tracked as relative or absolute coordinates. The direction and amount of movement can also be expressed by comparing spatial positions.

[0053] In the case where only the first set of commands is displayed, control of the medical imaging device may be performed according to the first or last selected differentiated first command field. The last selected differentiated first command field may be defined as the last selected differentiated first command field after a predetermined time has passed or after receiving a confirmation such as described below.

[0054] Similarly, if the first set of commands and the second set of commands are displayed, control of the medical imaging device may be performed according to the first or last selected differentiated first command field if no further movement beyond a predetermined amount in the direction to one of the differentiated second command fields is tracked after a predetermined time has elapsed or after a confirmation has been received. Control of the medical imaging device may be performed according to the first or last selected differentiated second command field. For the differentiated first command fields, the last selected differentiated second command field may be defined as the differentiated first command field that was last selected after a predetermined time has elapsed or after a confirmation has been received.

[0055] In some embodiments, the step of controlling the medical imaging device according to a command field selected from the command fields of the command menu also includes: confirming the selected command field by moving the operator's eyes or head to the confirmation field, wherein the selection is confirmed when the direction and amount of the corresponding movement correspond to the position of the corresponding confirmation field.

[0056] Thus, control of the medical imaging device does not rely on any time constraints which might be prone to accidental command execution, but rather selected commands must be intentionally confirmed to be executed.

[0057] In a further aspect, the present disclosure relates to a computer program product comprising a program code stored on a machine-readable medium and configured to, when executed on a data processing apparatus, cause the data processing apparatus to perform the method as described above.

[0058] The computer program product may provide the same functionality and advantages as described with respect to the head mounted display system and method for controlling an imaging device.

[0059] Other advantages, aspects and details of the present disclosure are subject to the claims, and the following description of the preferred embodiments applies the principles of the present disclosure and the accompanying drawings. In the accompanying drawings, the same reference numerals represent the same features and functions. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 is a schematic diagram of a head mounted display showing a display screen and a command menu according to an exemplary embodiment;

[0061] Figure 2 It shows that according to Figure 1 A schematic diagram of a head mounted display of a command menu and display screen with a selected and confirmed command field; and

[0062] Figure 3 is a schematic diagram of a disclosed head mounted display system according to an exemplary embodiment. DETAILED DESCRIPTION

[0063] Figure 1 A schematic diagram of a head mounted display 10 is shown, which shows a display screen 12. The display screen 12 shows a command menu 122, which includes a first group of commands 123 and a second group of commands 124 arranged around a predetermined viewing center 121. The first group of commands 123 is composed of four different first command fields 123a arranged at an equal first distance from the viewing center 121. The first group of commands 123 may include different first command fields 123a to control the magnification, viewing angle, viewing center or focal plane position. The first group of commands 123 is surrounded by a second group of commands 124. The second group of commands 124 includes seven different second command fields 124a arranged at an equal second distance from the viewing center 121. The equal second distance is greater than the equal first distance. Here, the equal second distance is selected to differentially arrange the differential second command fields 124a adjacent to the differential first command fields 123a in a direction extending radially from the observation center 121, thereby limiting the required movement of selecting different command fields 123a, 124a to be tracked to a minimum. The second set of commands 124 may include the differential second command fields 124a to control the position of the display screen 12, store images, adjust light levels, change working distances, store and call up robot positions, or open additional related functions such as viewing imaging data, switching to fluorescence mode, activating infrared cameras, opening surgical navigation, etc. In an exemplary embodiment, the color of the first set of commands 123 is different from the color of the second set of commands 124 for identification reasons. Different colors, shades and / or transparency levels can also be used to indicate the activation or deactivation status of the first set of commands 123 and the second set of commands 124 and / or indicate the activation or deactivation status of the differential first command fields 123a and / or the differential second command fields 124a.

[0064] also, Figure 1 A confirmation field 125 is shown located in the viewing center 121. Since the confirmation field 125 may cover the relevant portion of the medical image in the viewing center 121, the confirmation field 125 may be displayed only when needed, in particular, when the input device 80 ( Figure 3) such as a foot switch. Alternatively or additionally, the confirmation field 125 may be displayed only after the command fields 123a, 124a are selected. Even if the command menu 122 is not arranged in the viewing center 121, the command menu 122 or at least a portion thereof may be displayed only on demand.

[0065] Figure 2 shows the first command field 123a after it has been selected. Figure 1 Schematic diagram of an exemplary embodiment of the present invention, where the upper first command column 123a with a difference represents the control of the viewing angle. By using the tracking system 14 ( Figure 3 ) tracks the operator's head movement to detect the selection, wherein the command field 123a, 124a is selected when the direction and amount of the corresponding movement correspond to the corresponding differentiated first command field 123a or second command field 124a. Therefore, the tracked movement representing the operator's head movement in terms of direction and amount corresponds to the upper differentiated first command field 123a of the control representing the viewing angle. In order to indicate the selection, the upper differentiated first command field 123a of the control representing the viewing angle is highlighted, that is, the color of the differentiated first command field 123a is changed. The highlighting can also be performed by other visual effects such as changing the transparency level or mode, acoustic effects and / or tactile effects. Regarding the visual effect, the selected command field 123a, 124a may not have to be highlighted over its entire area, but only a highlighted frame may be provided.

[0066] In the embodiment shown, it is necessary to control the control device 40 ( Figure 3 ) for the medical imaging device 20( Figure 3 ) before controlling to execute the selected command. If the tracked movement representing the operator's head movement corresponds to the position of the confirmation field 125, the selected differentiated first command field 123a is confirmed. The received confirmation is indicated by highlighting the confirmation field 125.

[0067] A confirmation field is displayed in the viewing center 121. Therefore, after the operator's viewing can be assumed to be directed to the viewing center 121 as the relevant viewing on the operation area, the selected command is executed. In addition, the viewing center 121 can be assumed to be the preferred posture for the operator to proceed.

[0068] Figure 3 The exemplary embodiments are shown in Figure 1 and Figure 2The disclosed head-mounted display system 1 is an exemplary embodiment of the head-mounted display 10. The head-mounted display system 1 includes: the head-mounted display 10, a medical imaging device 20 for providing a medical image of a scene 50, a robot 60 for moving the medical imaging device 20, a user interface 70, and an input device 80.

[0069] The medical imaging device 20 includes a camera sensor 21 for capturing a medical image of a scene 50 via a lens 22 and a light 23 for illuminating the scene 50. The captured medical image is sent to the control device 40 to be forwarded to the head mounted display 10. The head mounted display 10 includes a display driver 11 to receive the medical image and send the medical image to the display screen 12 to display the medical image to the operator 30 via one eyepiece or two eyepieces 13. The user interface 70 may be an external screen controlled by the control device 40 to provide the operator 30 with different information and / or control capabilities independent of the control of the medical imaging device 20 through a command menu 122 displayed on the display screen 12 of the head mounted display 10.

[0070] In order to control the medical imaging device 20 based on tracked movements representative of the operator's head and / or eye movements, a tracking system 14 is incorporated into the head mounted display 10. For reasons of explanation, Figure 3 The description of the illustrated embodiment relates to tracked head movements of the operator 30. However, the same principles apply to tracked eye movements of the operator or a combination of tracked head and eye movements of the operator 30.

[0071] The tracking system 14 may be incorporated into the head mounted display 10 as a 3D acceleration sensor or an inertial measurement unit and provides information about the direction and amount of the tracked movement to the control device 40. The tracked movement representing the operator's head movement may be provided continuously or on demand. A demand may be expressed by activating the input device 80 to allow the operator to control the medical imaging device 20 based on such tracked movement. In addition, upon activation of the input device 80, such as a foot switch, the control device 40 causes the head mounted display 10 to display a command menu 122 on the display screen 12. This may be described with reference to FIG. Figure 1 and Figure 2 The selection and confirmation of the command to be executed. However, other options for selecting or selecting and confirming a command according to the present disclosure are also applicable.

[0072] In order to mitigate the risk of accidental selection and execution of commands, the control device 40 may be configured to only display the command menu 122 and thus allow the selection of commands as long as a continuous activation action is applied to the input device 80. In addition, Figure 3The head mounted display 10 of the exemplary embodiment of the embodiment includes a monitoring device 15 configured to monitor the position of the display screen 12 and issue a selection of one of the command fields 123a, 124a and / or display the command menu 122 only in a position of the display screen 12 that represents the operator 30's use of the display screen 12. Figure 3 In the exemplary embodiment of FIG. 1 , the monitoring device 15 is implemented as a contact switch which is closed when the display screen 12 is flipped downward in the operating state. However, the monitoring device may also be incorporated in the tracking system 14 .

[0073] although Figure 3 The control device 40 in FIG. 1 only shows a connection to the robot 60 to move the medical imaging device 20 and a connection to the light 23 of the medical imaging device 20 to control the brightness of the light 23, but the control device 40 may be configured to control more functions to affect the medical image displayed in the display screen 12 of the head mounted display 10. The number of connections is limited only for clarity of description.

[0074] It should be noted that the examples given are specific embodiments and are not intended to limit the scope of protection given in the claims. In particular, individual features of one embodiment may be combined with another embodiment. Figure 1 or Figure 2 In the illustrated embodiment, the first group of commands 123 includes four different first command fields 123a. Even though four different first command fields 123a can represent the main tracking motion that can be easily implemented and controlled, the first group of commands 123 can also include more different first command fields 123a or fewer different first command fields 123a. The same applies to the number of different second command fields 124a provided by the second group of commands 124. In addition, Figure 1 or Figure 2 The exemplary embodiment shown in FIG. 1 displays the first group of commands 123 and the second group of commands 124 simultaneously. However, the first group of commands 123 and the second group of commands 124 may also be displayed independently, for example, as required, or may also be displayed independently according to different operation states. Furthermore, even though the confirmation field 125 may be preferably located in the observation center 121, the confirmation field may be located near the observation center 121 as long as it can be assumed that the operator can view the observation center 121 from such a position.

[0075] Reference numerals list

[0076] 1Head-mounted display system

[0077] 10. Head-mounted display

[0078] 11Display Driver

[0079] 12 Display

[0080] 13 Eyepiece

[0081] 14Tracking System

[0082] 15 Surveillance Device

[0083] 20 Medical Imaging Systems

[0084] 21 Camera Sensors

[0085] 22 lenses

[0086] 23 Light

[0087] 30 Operators

[0088] 40 Control device

[0089] 50 scenes

[0090] 60 Robots

[0091] 70 User Interface

[0092] 80 Input Devices

[0093] 121 Observation Center

[0094] 122 Command Menu

[0095] 123 The first set of commands

[0096] 123a First command field

[0097] 124 The second set of commands

[0098] 124a Second command field

[0099] 125 Confirmation field

Claims

1. A head mounted display system (1) for controlling a medical imaging device (20), comprising: A head mounted display (10) to be worn by an operator (30), the head mounted display comprising a display screen (12) configured to display a medical image having a predetermined viewing center (121) and a command menu (122) providing at least a first set of commands (123), the first set of commands (123) being a differentiated first command field, a tracking system (14) for tracking movements representing eye movements and / or head movements of the operator, wherein commands of at least the first set of commands (123) of the command menu (122) can be selected based on the tracked eye movements and / or head movements, and a control device (40) configured to receive the selected command and control the medical imaging device (20) accordingly, The command menu (122) further includes a confirmation field (125) arranged at the predetermined observation center (121) of the medical image to confirm the selected command field (123a, 124a), Therein, the predetermined observation center (121) is determined as a relevant observation of the operating area taking into account the operating area or a specific part of the operating area.

2. The head mounted display system (1) according to claim 1, wherein: The differentiated first command fields (123a) of at least the first group of commands (123) of the command menu (122) displayed on the display screen (12) of the head mounted display (10) are arranged at a first distance substantially equal to the predetermined observation center (121) of the medical image.

3. The head mounted display system (1) according to claim 2, wherein: The command menu (122) provides a second group of commands (124), which are differentiated second command fields (124a) displayed on the display screen (12) of the head-mounted display (10) and arranged at a second distance substantially equal to the predetermined observation center (121) of the medical image, wherein the second distance of the differentiated second command field (124a) is greater than the first distance of the differentiated first command field (123a).

4. The head mounted display system (1) according to claim 3, wherein: The control device (40) is configured to deactivate the differentiated first command field (123a) when the tracking system (14) indicates that the tracked movement has passed through one of the differentiated first command fields (123a) and reached one of the differentiated second command fields (124a) in a direction away from the predetermined observation center (121) of the medical image.

5. The head mounted display system (1) according to claim 3 or 4, wherein: The control device (40) is configured to distribute commands to the first group of commands (123) and the second group of commands (124) based on frequency and / or probability of use.

6. The head mounted display system (1) according to claim 5, wherein: The control device (40) is configured to adjust the command distribution to the first set of commands (123) and the second set of commands (124) based on the changed frequency and / or probability of use.

7. The head mounted display system (1) according to claim 1, wherein: The head mounted display system (1) further comprises an input device (80) which has to be activated as a prerequisite for selecting one of the command fields (123a, 124a) and / or displaying the command menu (122).

8. The head mounted display system (1) according to claim 7, wherein: The input device (80) is configured to be activated only as long as a continuous activation action is applied.

9. The head mounted display system (1) according to claim 1, wherein: The head mounted display system (1) further comprises a monitoring device (15) configured to monitor the position of the display screen (12) and issue a selection of one of the command fields (123a, 124a) and / or display the command menu (122) only in a position of the display screen (12) representing that the display screen (12) is used by the operator (30).

10. The head mounted display system (1) according to claim 1, wherein: The head mounted display system (1) also provides different operating states and is configured to issue a selection of one of the command fields (123a, 124a) and / or display the command menu (122) only in a predetermined operating state.

11. The head mounted display system (1) according to claim 1, wherein: The tracking system (14) is configured to: track movements representing eye movements to control the predetermined viewing center (121) of the medical image on the display screen; and track movements representing head movements to select one of the command fields (123a, 124a).

12. The head mounted display system (1) according to claim 1, wherein: The head mounted display (10) is configured to: display a visual indication representing the tracked movement in the display screen (12); and / or highlight a selected one of the command fields (123a, 124a) and / or highlight a confirmation field (125).

13. The head mounted display system (1) according to claim 1, wherein: The medical imaging device (20) is a surgical microscope.

14. The head mounted display system (1) according to claim 1, wherein: The medical imaging device (20) is an endoscope.

15. The head mounted display system (1) according to claim 13, wherein: The surgical microscope is a robotic microscope.

16. The head mounted display system (1) according to claim 7, wherein: The input device is a foot switch.

17. A method for controlling a medical imaging device (20), comprising a head mounted display system (1) according to one of claims 1 to 16, the method comprising the following steps: a) displaying the command menu (122), b) tracking movements representative of eye movements and / or head movements of said operator, c) selecting one of the command fields (123a, 124a) of the command menu (122) by tracking the movement of the operator's eye movements and / or head movements, wherein, if only the first group of commands (123) is displayed, the corresponding differentiated first command field (123a) is selected when the direction and amount of the corresponding movement correspond to the position of one of the differentiated first command fields (123a); or if the first group of commands and the second group of commands (124) are displayed, the corresponding differentiated second command field (124a) is selected when the direction and amount of the corresponding movement correspond to the position of one of the differentiated second command fields (124a), and the corresponding differentiated first command field (123a) is selected when the direction and amount of the corresponding movement correspond to the position of one of the differentiated first command fields (123a) and further movement exceeding a predetermined amount in the direction to one of the differentiated second command fields (124a) is not tracked, and d) controlling the medical imaging device (20) according to a selected one of the command fields (123a, 124a).

18. The method according to claim 17, wherein: The step of controlling the medical imaging device according to one of the selected command fields (123a, 124a) of the command menu (122) also includes: confirming the selected command field (123a, 124a) by moving the operator's eyes or head to the confirmation field (125), wherein the selection is confirmed when the direction and amount of the corresponding movement correspond to the position of the corresponding confirmation field (125).

19. A computer program product comprising a program code stored on a machine-readable medium and configured to, when executed on a data processing device, cause the data processing device to perform the method according to claim 17 or 18.

Images