Apparatus and method for contactless haptic control

Abstract

A first aspect relates to a human system interface (HSI) for a contactless control device, in particular for a medical procedure, the HSI being configured to: generate a control space; measuring the position of the user, in particular the user's hand, in the control space; providing a first contactless tactile stimulus to the user in the control space based on the measured position; measuring a non-contact haptic input from the user in the control space; providing a second tactile stimulus to the user according to the tactile input; based on the haptic input, a parameter of the HSI is changed and / or provided to a device connected to the HSI.

Description

Technical Field

[0001] The present disclosure relates to apparatus and methods for performing non-contact tactile control of an apparatus, in particular a medical apparatus such as a microscope. In addition, a medical apparatus such as a microscope is disclosed that includes an apparatus for non-contact tactile control or interfaces with an apparatus for non-contact tactile control, or is configured to perform a non-contact tactile control method. Background Art

[0002] Contactless haptic feedback involves providing a sense of touch without physical contact. Various technologies have been explored, including ultrasound-based systems that produce a virtual sense of touch on the skin by stimulating mechanoreceptors. Other approaches involve electrical stimulation of the skin to produce a sense of touch. These advances aim to enhance user experience in virtual reality, gaming, and human-computer interaction by providing a realistic sense of touch without the need for physical contact.

[0003] Clinical or medical procedures (e.g., surgery) can be complex operations in which a human (e.g., a surgeon) may be faced with a variety of different tasks to handle the medical devices involved (e.g., a microscope or other medical or clinical device). Control of these devices still requires the medical staff (i.e., the user) to mechanically interact with the medical or clinical device. This may result in unnecessary cognitive burden on the user and further risk of contamination of the user with bacteria and germs. Improvements in these areas are desirable. Summary of the invention

[0004] It is an object of the present disclosure to improve user interaction with a device, in particular with a clinical or medical device and / or during a clinical or medical procedure.

[0005] This object is solved by the disclosed embodiments, which are particularly defined by the subject matter of the independent claims. The dependent claims provide information for further embodiments. Various aspects and embodiments of these aspects are also disclosed in the following overview and description, which provide additional features and advantages.

[0006] A first aspect relates to a human system interface (HSI) for a contactless control device, in particular for medical procedures, the human system interface being configured to: generate a control space; measure a position of a user, in particular a hand of the user, in the control space; provide a first contactless tactile stimulus to the user in the control space based on the measured position; measure contactless tactile input from the user in the control space; provide a second tactile stimulus to the user based on the tactile input; and based on the tactile input, change parameters of the HSI and / or provide parameters to a device connected to the HSI.

[0007] A human system interface (HSI) refers to a device used to interact between a human (sometimes also called an operator) and an operational hardware and / or software technology. It can allow the user to effectively control and monitor the operational technology. The purpose of HSI is to facilitate seamless and intuitive interaction between human users and operational technology, especially in complex environments such as clinical and / or medical settings.

[0008] HSI can play an important role in clinical and / or medical procedures by facilitating communication and control between medical personnel and complex clinical or medical systems. HSI (such as touch screens, graphical interfaces and control panels) allow clinical / medical personnel to enter commands, monitor vital signs, adjust settings and access patient data during surgery, diagnostic procedures or medical device operation. In the following, the term "medical" can refer to analytical, clinical, healthcare-related and / or medical topologies.

[0009] The user can interact with the HSI directly or indirectly. In direct interaction, the user interacts with the HSI with the body, such as with one or more fingers, one or two hands, palms, arms and / or head. Indirect interaction refers to the user interacting with the HSI through means such as pencils, sticks, gloves, etc. For the sake of brevity, all these types of direct or indirect interactions are referred to as user interactions.

[0010] A control space in the sense of the present disclosure is a space in which the HSI is effective. This means that the control space is a space in which the user can be observed through the HSI. Additionally or alternatively, the control space can be a space in which the user can be stimulated by tactile feedback through the HSI. Therefore, the generation of the control space is based on sensors and / or tactile actuators of the HSI. The control space can be one-dimensional, two-dimensional or three-dimensional.

[0011] Measuring the user's position involves registering the user (e.g., the user's hand) in the control space. The position may be derived directly by measuring an absolute or relative position, and / or indirectly by measuring parameters and determining the position based on the measured parameters.

[0012] The provision of the first tactile control stimulus may be related to a state in which a human-system interaction may begin. The first tactile stimulus may inform the user that the HSI has registered him. Additionally or alternatively, the HIS may utilize the first tactile stimulus to inform the user that further movement (i.e., a change in measured position) may result in interaction with a device connected to the HSI or with a feature of the HSI (e.g., a knob). The first tactile stimulus may be applied to the portion of the user whose position is measured. Additionally or alternatively, tactile stimuli may be provided to different portions of the user.

[0013] The measurement of contactless tactile input by the user may be based on measuring a change in a position-based quantity of a part of the user, ie, a change in the position of a finger of the user, through which the connected device and / or HSI is controlled.

[0014] A second tactile stimulus may be provided to display to the user the user's interaction with the connected device and / or with the HSI. Typically, the second tactile stimulus may be provided by the same means as the first tactile stimulus. Additionally or alternatively, the second tactile stimulus may be provided by a different tactile input means. The second tactile stimulus may be applied to the portion of the user where the position is measured. Additionally or alternatively, it may be provided to a different portion of the user. Specifically, the second tactile stimulus may be applied to the same portion of the user as the first tactile stimulus, such as a finger or palm.

[0015] Parameters provided to a device connected to the HSI may include tactile control information provided by a user and / or user interaction with the HSI. Parameters may be variables or factors that may be adjusted or manipulated to affect the behavior, performance, and / or results of a device connected to the HSI and / or any process performed by the connected device. The choice of parameters will depend on the nature of the connected system and / or process to be controlled by the user.

[0016] The HSI according to the first aspect may improve user interaction and experience by combining contactless tactile sensing and contactless tactile feedback for controlling connected devices and / or the HSI itself.

[0017] Embodiments of the first aspect relate to a human system interface (HSI) comprising one or more of the following sensors to measure tactile input: a camera; an accelerometer; an optical position tracker.

[0018] In order to measure the user position and / or tactile input, i.e. a part of the body and / or a part of the human guided means, the position of one or more cameras or optical trackers can be selected, whose position allows the camera system to have a clear view of the area where the tactile input will occur. In order to facilitate identification, the user can wear markers, for example attached to gloves. In the case of a surgeon, these can be, for example, one or more colored fingertips of a surgical glove. In the case of an accelerometer, it can be mounted on the part of the user that should be measured. For example, an accelerometer can also be included in a (surgical) glove. In order to obtain accurate results, the position-based data of multiple sensors can be integrated.

[0019] Markers and / or accelerometers or other means for detecting location-based information may be incorporated into a wearable device, such as a glove and / or bracelet.Based on suitable sensors for tactile input, tactile input with sufficient resolution may be obtained.

[0020] Embodiments of the first aspect relate to a human system interface (HSI) comprising one or more of the following actuators to provide contactless tactile stimulation to a user within a controlled space: - ultrasonic-based actuator; - laser-based actuator; - magnetic-field-based actuator; - air-vortex-based actuator; - glove-based actuator; - exoskeleton-based actuator; - vibrotactile actuator.

[0021] Ultrasonic-based actuators can provide non-contact tactile stimulation through ultrasound waves, which produce tactile or haptic feedback on the user's skin without the need for physical contact. Ultrasonic actuators generate high-frequency sound waves, typically above the audible range (>20kHz). These sound waves can be generated by piezoelectric transducers that convert electrical energy into mechanical vibrations. The generated ultrasound waves can be focused into a narrow beam directed at the user's skin (e.g., the skin of a fingertip or hand). This focusing can be achieved using various technologies, such as a phased array of ultrasonic actuators. When the focused ultrasound beam interacts with the skin, it can generate a force. The force exerted by the ultrasound waves causes localized vibrations on the surface of the skin, producing the sensation of tactile feedback. Different tactile sensations can be generated by varying various parameters of the ultrasound, including intensity, frequency, duration, and / or spatial distribution. By manipulating these parameters, different types of tactile sensations, such as texture, vibration, or pressure, can be generated. Ultrasound-based tactile feedback has several advantages, including the ability to create virtual or simulated tactile sensations without physical contact, precise control over the generated sensations, and the potential to create localized and detailed tactile effects.

[0022] Laser-based actuators use laser beams to produce localized heat and / or pressure effects on the skin, thereby producing tactile stimulation. Therefore, a laser source can be used to produce a focused laser beam. The laser beam can be in the visible spectrum or infrared spectrum, depending on the specific application and the desired effect. The laser beam can be focused into a smaller or larger spot using a lens or optical system. The size and shape of the spot can be adjusted based on the desired tactile effect and the target area on the skin. When the focused laser beam interacts with the skin, a localized thermal effect can be produced. The laser energy is absorbed by the skin, causing a rapid increase in temperature in the target area. This temperature change can activate thermal receptors in the skin, resulting in a perception of warmth or heat-related tactile feedback. In addition to the thermal effect, the laser beam can also cause a rapid expansion of the skin due to the generation of sound waves. This expansion can generate pressure waves that stimulate mechanoreceptors in the skin, thereby producing a sense of touch, i.e., tactile feedback. Laser parameters such as power, duration, and spot size can be controlled and modulated to produce different tactile sensations. By varying these parameters, a range of tactile sensations can be produced, including pressure, vibration, or texture.

[0023] A magnetic field based actuator may include a coil and / or a magnet and / or a ferromagnetic material. When current is passed through the coil, it generates a magnetic field. The interaction between this magnetic field and the magnet or ferromagnetic material generates a force that generates tactile feedback. By controlling the current in the coil, the strength and direction of the tactile feedback can be modulated.

[0024] Air vortex-based actuators can produce tactile feedback by generating directional pulses or airflows that interact with the user's skin. These air vortices can produce tactile sensations on the surface of the skin, simulating touch or haptic feedback. Air vortex-based actuators consist of a chamber or nozzle connected to a controlled air supply system. The air supply system sprays compressed air into the chamber or nozzle. The generated vortex bursts propagate through the air towards the user's skin. The tactile stimulation can be controlled and modulated by adjusting various parameters, such as the speed, size, and frequency of the vortex bursts. By manipulating these parameters, different tactile sensations can be generated, including tapping, knocking, or vibration. To produce more complex tactile effects, multiple air vortex-based actuators can be used in an array or grid configuration. By coordinating the activation of different actuators, spatial and temporal patterns of tactile feedback can be generated on the skin.

[0025] Additionally, semi-contactless tactile stimulation can be provided by vibrotactile actuators, such as piezoelectric actuators based on gloves and / or bracelets, and / or exoskeleton-based actuators. In these cases, the user can wear the appropriate device and perceive the tactile stimulation, which presents the sensation of touching the HSI and / or the connected device.

[0026] Embodiments of the first aspect relate to a human system interface (HSI) configured to provide non-contact tactile stimulation over a two-dimensional area.

[0027] By providing non-contact tactile stimulation on a two-dimensional area, a three-dimensional control space can be generated. The non-contact stimulation can be provided by a tactile actuator array (e.g., an ultrasonic actuator array). Based on the tactile actuator array, a focused non-contact tactile stimulation, such as a focused ultrasonic-based tactile stimulation, can be generated. The two-dimensional area can be aligned parallel to the visual display device, for example, by arranging the non-contact tactile actuator array behind the visual display. The two-dimensional area can be a rectangle, a circle, or an ellipsoid. Additionally or alternatively, the two-dimensional area can have a form so that it can be placed on a device connected to the HSI, for example, on a surface. Based on the two-dimensional area for generating tactile stimulation, an increased control space adapted to the operating environment can be provided.

[0028] Embodiments of the first aspect relate to a human system interface (HSI) configured to: generate a control space based on an intersection of a field of view of a camera and a stimulation field of a two-dimensional tactile stimulation area; and visualize the control space for a user.

[0029] At the intersection, the user (e.g. the position of a human hand) can be registered, and the user's tactile input can be measured. In addition, tactile stimulation for the user can be generated at the intersection. The intersection can be displayed to the user via a visual display capable of displaying a three-dimensional space (e.g. a 3D screen and / or goggles). For example, a space can be displayed to the user in which a camera can measure the position of the user's fingers, and at the same time, the position where the ultrasonic tactile display is effective can be displayed to the user. The intersection of the two spaces can be visualized by different colors to indicate the control space to the user. Additionally or alternatively, the user can be notified when the control space is entered by non-contact tactile stimulation.

[0030] Embodiments of the first aspect are directed to a human system interface (HSI) configured to provide a contactless touch control element generated by contactless tactile stimulation and controllable by tactile input.

[0031] The control element driven by tactile stimulation may include any component or feature that can be used to manipulate, adjust or manage the HSI and / or the device connected to the HSI. The control element can enable the user to interact with the HSI and / or the connected device and affect its behavior. The control element may include a switch for binary control. A knob for multi-state or (quasi) continuous control. They may also include buttons, sliders and / or knobs. In addition, the control element may include a visual keyboard and / or a visual dial that can be used via tactile control. Additionally or alternatively, the control element may be a contactless touchpad that is configured to detect touches and / or gestures in an area / space defined by the touchpad. By using one or more control elements driven by tactile stimulation, the user can intuitively control the HSI or the connected device.

[0032] Embodiments of the first aspect relate to a human system interface (HSI), wherein the control element is a knob configured to be pressed, slid and / or rotated based on tactile input.

[0033] Embodiments of the first aspect relate to a human system interface (HSI) wherein a visual representation of a knob is presented to a user via a display and / or goggles.

[0034] Other means for providing three-dimensional visual stimulation are also possible, such as holographic projectors / fans, stereoscopic displays and / or light field displays.

[0035] Embodiments of the first aspect relate to a human system interface (HSI) configured to provide visual and / or auditory stimulation based on operation of a knob and / or based on operation of a connected device via tactile input.

[0036] Visual and / or auditory stimulation may impart information independent of the contactless tactile stimulation. Thereby, the contactless tactile control may be enriched by additional information, in particular by information that is difficult to represent by means of tactile stimulation. Such information may, for example, relate to the functionality of the tactile control element indicating the kind of control element. For example, a visual stimulus may be provided to color-code a knob so that it can be distinguished from another knob.

[0037] Additionally or alternatively, the visual and / or auditory stimulation may impart information related to the information provided by the tactile stimulation, in particular similar or identical information. Thus, an integrated perception may be generated for the user, which makes it easier for the user to perceive the controlled state. For example, if the user turns a knob, an acoustic frequency sweep or an increase in volume may indicate that the knob is deviating from its zero state. Additionally or alternatively, if the knob reaches a limit, a simultaneous and corresponding sound may generate an integrated perception that helps the user understand that the knob cannot be turned further.

[0038] Embodiments of the first aspect relate to a human system interface (HSI) configured to directly control a connected device.

[0039] Thus, the HSI may be directly attached to the connected device. Additionally or alternatively, the haptic actuator is attached to the connected device.

[0040] In the case of direct control, contactless tactile control can be achieved without the use of virtual control elements generated by tactile actuators. In addition, the stimulus provided by the tactile actuator can directly represent the state of the connected device. For example, a microscope can be oriented by direct contactless tactile control. The tactile control then drives the motor that moves the microscope. If the microscope is constantly moving, a small tactile stimulus can be fed back, because the direct interaction also only generates a small reaction force. If the movement of the microscope accelerates, a higher tactile stimulus can be fed back, indicating that the user would feel an increase in inertia if the user moved the microscope directly.

[0041] Additionally or alternatively, the connected device may have one or more sensors to measure parameters and / or states affected by the user using contactless tactile control. In this case, information about the parameters / states may be reflected to the HSI, and the tactile stimulation fed back to the user may be based on the reflected information. Thus, the user can obtain direct feedback of the connected device being controlled.

[0042] Embodiments of the first aspect relate to a human system interface (HSI) which is configured to provide a control configuration type which can be assigned to parameters of a control element and / or to parameters of a connected device.

[0043] A control configuration may represent, for example, control of a parameter with a constraint. If the device is moving in free space, such a control configuration may have inertia-related control. Once the parameter reaches a constraint, such as a boundary of free space, the tactile stimulation will increase. Such a configuration may also be used for a virtual knob. In order to appropriately adjust the control configuration to the actual use case, the configuration may have one or more adjustable parameters, such as the range of the controlled parameter, the magnitude of the tactile stimulation when the parameter constraint is reached, and / or the behavior in free space.

[0044] Embodiments of the first aspect relate to a human system interface (HSI) configured to provide contactless tactile feedback information to indicate the quality of a user interaction and / or to guide a user to provide tactile input.

[0045] Non-contact tactile stimulation can also help the user align to interact properly with the tactile display. Thereby, the user can become more confident in the use of the control elements, and the HSI becomes easier and faster to use. In particular, the tactile feedback can indicate whether the user's hand is properly aligned with the tactile control elements of the HSI and / or the connected device. Additionally or alternatively, the tactile feedback can indicate whether the gesture is recognized by the HSI and / or whether the gesture has not been recognized by the HSI. Additionally or alternatively, the non-contact tactile feedback can indicate whether the user (e.g., the user's hand) is misaligned in and / or during the use of the non-contact control.

[0046] A second aspect relates to a method for controlling a device, in particular a medical device, comprising the following steps: obtaining location-based information of a user; providing a first non-contact tactile stimulus to the user; obtaining non-contact tactile input from the user; providing a second tactile stimulus to the user based on the tactile input; and based on the tactile input, changing parameters of the HSI and / or providing parameters to a device connected to the HSI.

[0047] Embodiments according to the second aspect relate to a method for controlling a device, wherein provision of a first contactless tactile stimulus represents and / or determines an initial state of the HSI and / or of a connected device.

[0048] For example, if the microscope should be moved by contactless tactile control, the position of the user (e.g., the user's hand) when the first tactile stimulus is provided indicates the initial position of the microscope. This is independent of whether the microscope is controlled directly or via a control element. If a control element, such as a knob, is used, the initial contact can be associated with the zero position of the control element or the "OFF" position of the control element.

[0049] A third aspect relates to a medical device, in particular a microscope, comprising an HSI according to the first aspect; and / or configured to interact with an HSI according to the first aspect; and / or to perform a method according to the second aspect.

[0050] The medical devices that may be controlled by the HSI according to the first aspect may be numerous and include, for example:

[0051] Microscope: Interaction with the microscope on the HSI according to the first aspect may involve, for example, adjustment of the focus, adjustment of the visual axis and / or adjustment of visual parameters such as brightness or contrast.

[0052] Patient monitors: These devices display real-time data of vital signs such as heart rate, blood pressure, blood oxygen saturation and respiratory rate. They may have, for example, a graphical interface or a digital display and a contactless tactile control interface according to the first aspect of the present disclosure to manipulate and / or switch between visualized data.

[0053] Infusion pumps: Infusion pumps deliver fluids, medications or nutrients to patients through a vein. They can be controlled with an HSI according to the first aspect, where healthcare professionals can set infusion rates, dosages and customize delivery parameters through contactless tactile feedback.

[0054] Ventilators: Ventilators help patients breathe when they cannot breathe independently. They can be controlled with an HSI according to the first aspect to enable the user to adjust ventilation parameters, monitor respiratory data, and set alarms. According to an embodiment of the present disclosure, the touch screen interface of the ventilator can be implemented with non-contact tactile control.

[0055] Electrocardiograph (ECG / EKG) machines: These devices can record and / or display the electrical activity of a patient's heart. They can be controlled with an HSI according to the first aspect having a graphical and / or tactile representation of the heart's electrical signals, thereby allowing a healthcare professional to interpret and analyze the data. According to an embodiment of the present disclosure, the interface of the ECG / EKG device has one or more control functions based on contactless tactile control.

[0056] Surgical Robots: Robotic-assisted surgical systems may incorporate an HSI that provides the surgeon with a three-dimensional view of the surgical site and enables precise control of robotic instruments based on the HSI according to the first aspect. The surgeon may use the HSI according to the first aspect to manipulate the robotic arm during minimally invasive surgery. The HSI for the surgeon operating the surgical robot may include non-contact tactile feedback to adjust, tune, and / or move the robot. For the latter, the HSI with non-contact tactile feedback may be attached directly to the robot or used to remotely operate the system.

[0057] Magnetic Resonance Imaging (MRI) Machines: MRI scanners can use non-contact tactile HSI to control imaging parameters and visualize captured images. Operators can adjust scan protocols, select imaging sequences, and view real-time images through an interface that can be specially equipped with non-contact tactile control capabilities. Non-contact tactile control capabilities can also be constructed into mobile MRI scanners.

[0058] The interaction of the medical device with the HSI according to the first aspect may in particular be performed via an application user interface (API).The medical device may also be implemented and / or driven according to the method of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] Further advantages and features are derived from the following embodiments, some of which are referred to the accompanying drawings. The drawings do not always show the embodiments to scale. The dimensions of the various features may be enlarged or reduced, in particular for clarity of description. For this purpose, the drawings are at least partially illustrated.

[0060] Figure 1 A human system interface (HSI) for contactless tactile control according to an embodiment of the present disclosure is shown.

[0061] Figure 2 A haptic actuator system and a haptic measurement system for HSI according to an embodiment of the present disclosure are shown.

[0062] Figure 3 Direct contactless tactile control of a connected device according to an embodiment of the present disclosure is shown.

[0063] Figure 4 A tactile control element for indirect contactless tactile control of a connected device is shown according to an embodiment of the present disclosure.

[0064] Figure 5 A method for controlling a connected device with HSI according to an embodiment of the present disclosure is shown.

[0065] Figure 6 A microscope system controlled by HSI according to an embodiment of the present disclosure is shown.

[0066] In the following description, reference is made to the accompanying drawings which form a part of the present disclosure and which show specific aspects that can be understood of the present disclosure. The same reference numerals refer to features that are the same or at least similar in function or structure.

[0067] Typically, the disclosure of the described method also applies to a corresponding device (or apparatus) for performing the method or a corresponding system including one or more devices, and vice versa. For example, if a specific method step is described, the corresponding device may include features for performing the described method step, even if the feature is not explicitly described or shown in the figure. On the other hand, if a specific device is described, for example, based on a functional unit, the corresponding method may include one or more steps for implementing the described function, even if these steps are not explicitly described or shown in the figure. Similarly, a system may be provided with corresponding device features or features for performing specific method steps. Unless otherwise explicitly stated, the features of the various exemplary aspects and embodiments described in the context may be combined. DETAILED DESCRIPTION

[0068] Figure 1 A human system interface (HSI) for contactless tactile control according to an embodiment of the present disclosure is shown. HSI 100 includes a visual tactile display 102. Through the visual tactile display, the HSI can apply visual and tactile stimulation in a three-dimensional space. The HSI also includes two cameras 104, through which a three-dimensional image of the space in front of the visual tactile display 102 can be observed to measure tactile input. The HSI can define the intersection of the space where the tactile stimulation is applied and the space where the HSI can measure the tactile input as a control space 106. The control space 106 is visualized to the user through the screen 102. Therefore, the user knows when tactile input can be provided and when tactile input cannot be provided. For contactless visual tactile control tasks, virtual buttons 108a, 108b are provided. The positions of the virtual buttons 108a, 108b are determined before operation. The virtual button has two areas, namely a "button standby area" 108a and a "button activation area" 108b.

[0069] The button 108a, 108b is configured to be pressed along the linear degree of freedom 110. It can be pressed, for example, by the fingertip of the user's hand 112. In the "button standby area", a first weak tactile stimulus informs the user that the user's fingertip is in a position where the user can press the virtual button. The "button activation area" provides a second tactile stimulus that is stronger than the first tactile stimulus to indicate that the virtual button is pressed. The first tactile feedback indicates a successful alignment of the fingertip with the virtual button. The second tactile stimulus indicates the pressing of the button. In order to provide a strong stimulus, the pressing of the button is supported by a visual indication that the button is pressed (visually the button becomes smaller in the direction indicated by the degree of freedom 110) and an audible indication (e.g. a click sound).

[0070] Figure 2A tactile display 200 for HSI according to an embodiment of the present disclosure is shown. The tactile display 100 includes a tactile actuator system 202 and a tactile measurement system 220. The tactile actuator system 202 includes a plurality of ultrasound sources 204 arranged in a rectangular shape. The array of ultrasound sources 204 is configured to generate a focused ultrasound beam 206 over a rectangular space 208. The space 208 has a length x=40 cm and a height y=25 cm. The array of ultrasound sources generates a focused beam effective to a depth z=20 cm.

[0071] The tactile measurement system 204 of the display 200 includes two cameras that can observe the entire area / volume of the tactile actuator system 200. The tactile measurement system 204 is able to observe and measure the finger position 210 in the volume in which the tactile actuator system 202 is active. Therefore, the control volume in which tactile input can be observed and tactile stimulation can be applied is equal to the volume in which the ultrasound source array is active (40 cm x 25 cm x 20 cm).

[0072] The control space can be freely configured with control elements 230, such as buttons, sliders and knobs. The control elements are generated by tactile stimulation and supported by information related to other modalities such as acoustics and / or vision. The control elements can be used to adjust the parameters or functions of the HSI itself. Additionally or alternatively, the parameters or functions of the connected device can be controlled by the control elements. Additionally or alternatively, gestures independent of the control elements can be detected in the control space. Gestures can also be used to control the HSI itself or the connected device. In a specific embodiment, the control space can be divided so that a part provides control elements. In another part of the control space, gestures are observed and tactile feedback is provided. The tactile feedback can be related to the provided gestures and / or the recognition of one or more gestures. For example, the tactile feedback provides the user with information whether the gesture is correctly recognized and / or whether the user's tactile input is correctly provided. The separation of the control space relative to the control elements and gesture recognition can also be implemented in a time-related manner in addition or alternatively, so that, for example, the control space first performs gesture recognition and then the control space can be controlled by the control element.

[0073] Figure 3 Direct non-contact tactile control 300 of a connected device 302 is shown in accordance with an embodiment of the present disclosure. The connected device 302 to be controlled is a microscope. The direct tactile control shown is used to manipulate an optical axis 312 of the microscope 302, which is directed to the location of a sample 314, such as in a surgical cavity. The HSI is attached to the microscope 302. The HSI consists of an array of ultrasonic actuators 304, which are configured to provide tactile stimuli H to the fingertips of a human operator 306 via an ultrasound beam 306. The tactile input of the human operator is measured by a three-dimensional (3D) camera system 308.

[0074] Direct tactile control of the optical axis 312 of the microscope 302 is achieved via a tactile stimulation configuration 320, which is mapped to the movable angular range of the microscope 302. The position of the user's finger is measured by a three-dimensional camera system 308. Once the user's finger is registered in the control space (not shown), the HSI looks for a sliding gesture. If a sliding gesture is recognized, the HSI confirms this with a first tactile feedback signal 322 and calibrates the system so that the microscope can start moving according to the sliding motion. In other words, the HSI locks into the microscope to follow the user's hand movement. The movement of the fingertip 310 within the sliding gesture is then converted into movement of the microscope. For every 10° movement of the microscope, an additional tactile stimulation 324 is provided. When the microscope 302 approaches the end of its movable range, an increased tactile stimulation 326 is provided. If the end of the range is reached, an additional acoustic signal is provided.

[0075] In addition to three-dimensional (3D) boundaries and regions that are fixed in space, regions and boundaries that are positioned relative to the position of the microscope are also useful. For example, when a hand is used to virtually move and guide the microscope, a "virtual grip zone" can be defined that is always around the microscope. This means that there is an area / space that, if the hand is in this area / space, can be detected as a potential control aid. In this case, it is useful to provide tactile feedback as to whether the user's hand is in the correct position.

[0076] Figure 4 A tactile control element and its functionality for indirect contactless tactile control of a connected device according to an embodiment of the present disclosure is shown. The left side of the figure shows an HSI 400 with a tactile control element (rotating knob). The HSI includes a visual tactile display 402 and a camera system 404 for measuring tactile user input. The visual tactile display can display a knob 406, which can be rotated by a human user 408 to manipulate a parameter of the HSI or a connected device.

[0077] exist Figure 4 , the function of the knob 406 is described. The knob 406 can be rotated to the right and left. Depending on the rotation angle associated with the parameter to be manipulated, Provide tactile stimulation (H) to the user. The stimulation is constructed to be similar to the behavior of a mechanical rotating knob. While rotating the knob, the user obtains a tactile stimulation 424 to indicate how far he has rotated the knob. Confirmation of the rotation is provided by discrete tactile clicks similar to a real rotating knob. Confirmation that the knob is in the zero position is displayed by a larger tactile click. This is also visually displayed to the user on the three-dimensional (3D) screen of the visual tactile display 402. If the knob is rotated to a position close to its end point, the user receives increased tactile feedback to notify the user of the end point of the rotation range. In other words, the HSI indicates that the end point of the rotation range has been reached by providing a gradually increasing tactile intensity indication near the end point of the range. Through tactile feedback, the user can even perceive the position of the knob without looking at the visual display. This is particularly helpful for medical procedures.

[0078] Figure 5 A method for controlling a connected device with HSI according to an embodiment of the present disclosure is shown.

[0079] In step 510, when the user's hand moves near the microscope and enters the predetermined control space, the microscope locks and follows the user's hand. Therefore, the HSI needs to obtain the position of the user's hand. This can be achieved by, for example, a position measurement system (e.g., a camera) attached to or connected to the HSI.

[0080] After the user's hand has been registered in the control space, a first non-contact tactile stimulus is provided to the user in step 520. The first non-contact tactile stimulus indicates to the user that the HSI has assigned the position of the user's hand to control a parameter. The parameter may be a parameter of the HSI itself and / or a parameter of a device connected to the HSI.

[0081] In a following step 530, the tactile input from the user is measured by the HSI. By means of this measurement, the control input of the user is recorded. In parallel with the recording of the control input of the user, in a further step 540, a second tactile stimulus is provided to the user. This stimulus may be a static stimulus, which is quasi-independent of the user's input and only informs the user of the correct alignment of the user's input. Additionally or alternatively, the stimulus may vary according to the movement of the user's hand. Thus, complex feedback may be provided to the user, as described above.

[0082] After recording the tactile input, in step 550 , parameters of the HSI and / or the connected device are adapted in relation to the tactile input.

[0083] The tactile feedback can serve as a guide for alignment and confirmation of user actions. Another way to describe the concept of programming tactile feedback is to define three-dimensional boundaries and / or regions that define the type of tactile feedback. In the above example, the virtual boundaries can be derived from the physical limitations of the microscope, i.e., the microscope arm cannot move beyond, or the limitations of the virtual control elements. Alternatively, the virtual boundaries can be determined by other limitations (such as alignment with the surgical cavity), for example, if the microscope is limited to an angle that allows viewing in the surgical cavity.

[0084] Some embodiments relate to a microscope comprising: Figures 1 to 5 Alternatively, the microscope may be combined with Figures 1 to 5 One or more of the systems described herein are part of or connected to the system. Figure 6 A schematic diagram of a system 600 configured to perform the methods described herein is shown. The system 600 includes a microscope 610 and a computer system 620. The microscope 610 is configured to capture images and is connected to the computer system 620. The computer system 620 is configured to perform at least a portion of the methods described herein. The computer system 620 can be configured to execute a machine learning algorithm. The computer system 620 and the microscope 610 can be separate entities, but can also be integrated together in a common housing. The computer system 620 can be part of a central processing system of the microscope 610, and / or the computer system 620 can be part of a subcomponent of the microscope 610, such as a sensor, actuator, camera, or lighting unit of the microscope 610.

[0085] The computer system 620 may be a local computer device (e.g., a personal computer, notebook, tablet computer, or mobile phone) having one or more processors and one or more storage devices, or may be a distributed computer system (e.g., a cloud computing system having one or more processors and one or more storage devices distributed in various locations, for example, at a local client and / or one or more remote server farms and / or data centers). The computer system 620 may include any circuit or combination of circuits. In one embodiment, the computer system 620 may include one or more processors that may be of any type. As used herein, a processor may refer to any type of computing circuit, such as, but not limited to, a microprocessor, a microcontroller, a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction character (VLIW) microprocessor, a graphics processor, a digital signal processor (DSP), a multi-core processor, a field programmable gate array (FPGA) such as a microscope or a microscope component (e.g., a camera), or any other type of processor or processing circuit. Other types of circuits that may be included in the computer system 620 may be custom circuits, application specific integrated circuits (ASICs), etc., such as one or more circuits (e.g., communication circuits) used in wireless devices such as mobile phones, tablet computers, notebooks, two-way radios, and similar electronic systems. The computer system 620 may include one or more storage devices, which may include one or more storage elements suitable for a particular application, such as a main memory in the form of a random access memory (RAM), one or more hard disk drives, and / or one or more drives for handling removable media, such as compact disks (CDs), flash memory cards, digital video disks (DVDs), etc. The computer system 620 may also include a display device, one or more speakers, and a keyboard and / or controller, which may include a mouse, a trackball, a touch screen, a voice recognition device, or any other device that allows a system user to input information to and receive information from the computer system 620.

[0086] Some or all method steps can be performed by (or using) hardware devices, such as processors, microprocessors, programmable computers or electronic circuits. In some embodiments, some one or more of the most important method steps can be performed by such devices.

[0087] Depending on the specific implementation requirements, embodiments of the present invention may be implemented in hardware or software. Implementation may be performed using a non-transitory storage medium, such as a digital storage medium, such as a floppy disk, DVD, Blu-ray, CD, ROM, PROM and EPROM, EEPROM or flash memory, on which electronically readable control signals are stored that cooperate (or are capable of cooperating) with a programmable computer system to perform the corresponding method. Thus, the digital storage medium may be computer readable.

[0088] Some embodiments according to the invention comprise a data carrier having electronically readable control signals, which are capable of cooperating with a programmable computer system, such that one of the methods described herein is performed.

[0089] Generally, embodiments of the present invention can be implemented as a computer program product with a program code, when the computer program product runs on a computer, the program code is operative for performing one of the methods.For example, the program code can be stored on a machine readable carrier.

[0090] Other embodiments comprise the computer program for performing one of the methods described herein, stored on a machine readable carrier.

[0091] In other words, an embodiment of the inventive method is, therefore, a computer program having a program code for performing one of the methods described herein, when the computer program runs on a computer.

[0092] Therefore, other embodiments of the present invention are a storage medium (or data carrier, or computer-readable medium) comprising a computer program stored thereon, which, when the computer program is executed by a processor, is used to perform one of the methods described herein. The data carrier, digital storage medium or recording medium is generally tangible and / or non-transient. Other embodiments of the present invention are devices described herein comprising a processor and a storage medium.

[0093] Further embodiments of the invention are, therefore, a data stream or a sequence of signals representing the computer program for performing one of the methods described herein.The data stream or the sequence of signals may, for example, be designed to be transmitted via a data communication connection (eg via the Internet).

[0094] Other embodiments comprise a processing means, for example a computer or a programmable logic device, constructed or adapted to perform one of the methods described herein.

[0095] Other embodiments comprise a computer having installed thereon the computer program for performing one of the methods described herein.

[0096] Other embodiments according to the invention include an apparatus or system configured to transmit (e.g., electronically or optically) a computer program for performing one of the methods described herein to a receiver. For example, the receiver may be a computer, a mobile device, a storage device, etc. The apparatus or system may, for example, include a file server for transmitting the computer program to the receiver.

[0097] In some embodiments, a programmable logic device (e.g., a field programmable gate array) can be used to perform some or all of the functions of the methods described herein. In some embodiments, a field programmable gate array can collaborate with a microprocessor to perform one of the methods described herein. Typically, these methods are preferably performed by any hardware device.

[0098] As used herein, the term "and / or" includes any and all combinations of one or more of the associated listed items and may be abbreviated as " / ".

[0099] Although some aspects have been described in the context of an apparatus, it is clear that these aspects also represent a description of a corresponding method, where a block or device corresponds to a method step or a feature of a method step. Similarly, aspects described in the context of a method step also represent a description of a corresponding block or item or feature of a corresponding device.

[0100] Reference numerals list

[0101] 100 HSI

[0102] 102 Visual Tactile Display

[0103] 104 Camera

[0104] 106 Control Space

[0105] 108a Virtual button spare area

[0106] 108b Virtual button activation area

[0107] 110 button degrees of freedom

[0108] 112 User's Hand

[0109] 200 Tactile Display

[0110] 202 Haptic Actuator System

[0111] 204 Ultrasound Source

[0112] 206 Ultrasonic beam

[0113] 208 Ultrasonic beam space

[0114] 210 Finger Position

[0115] 220 Tactile Measurement System

[0116] 230 Control elements

[0117] 300 Contactless tactile control

[0118] 302 connected devices

[0119] 304 Ultrasonic Actuator Array

[0120] 306 Human Operator

[0121] 308 3D Camera System

[0122] 312 Optical Axis

[0123] 314 samples

[0124] 320 Tactile stimulation configuration

[0125] 322 Haptic activation signal

[0126] 324 Motion Signal

[0127] 326 Boundary Tactile Stimulation

[0128] 400 HSI

[0129] 402 Visual Tactile Display

[0130] 404 Camera System

[0131] 406 Knob

[0132] 408 Users

[0133] 424 Tactile stimulation signal

[0134] 510 Register User

[0135] 520 Provides the first tactile stimulation

[0136] 530 Measuring tactile input

[0137] 540 Providing a second tactile input

[0138] 550 Adaptation parameters

[0139] 600 System

[0140] 610 Microscope

[0141] 620 Computer Systems

Claims

1. A human system interface (HSI) for contactless control of a device, in particular for medical procedures, the human system interface being configured as follows: - generating a control space (106); - measuring the position of a user (112), in particular a user's hand, in the control space; - providing a first non-contact tactile stimulus (108a, 108b) to the user in the control space based on the measured position; -Measure contactless tactile input from the user in the control space; - providing a second tactile stimulus (108a, 108b) to the user based on the tactile input; - Based on the tactile input, changing parameters of the HSI and / or providing parameters to a device connected to the HSI.

2. The HSI of claim 1, comprising one or more of the following sensors to measure tactile input: - Camera (104); - accelerometer; -Optical position tracker.

3. The HSI of any preceding claim, comprising one or more of the following actuators to provide non-contact tactile stimulation to a user within the control space: - an ultrasound based actuator (204); - Laser-based actuators; - Magnetic field based actuators; - Actuators based on air vortexes; - Glove-based actuators; -Exoskeleton-based actuators; -Vibrotactile actuator.

4. The HSI according to any of the preceding claims, configured to provide contactless tactile stimulation over a two-dimensional area (200).

5. The HSI according to the preceding claim, which is configured as follows: - generating a control space (106) based on the intersection of the field of view of the camera (104) and the stimulation field of the two-dimensional tactile stimulation area (200); and - Visualize the control space for the user.

6. The HSI of any one of the preceding claims, configured to provide a contactless touch control element (108a, 108b) generated by a contactless tactile stimulus and controllable by a tactile input.

7. The HSI of the preceding claim, wherein the control element is a knob (108a, 108b, 230, 406) configured to be pressed, slid and / or rotated based on tactile input.

8. The HSI of the preceding claim, wherein a visual representation of the knob (108a, 108b, 230, 406) is presented to the user via a display and / or goggles.

9. The HSI of any of the preceding claims, configured to provide visual and / or auditory stimulation based on operation of a knob (108a, 108b, 230, 406) and / or based on operation of a connected device (302) via tactile input.

10. The HSI as claimed in claim 1, which is designed to directly control a connected device (302).

11. The HSI according to claim 1, which is designed to provide a control configuration type (320, 420) which can be assigned to a parameter (φ) of a control element (406) and / or to a parameter (φ) of a connected device (302).

12. The HSI of any preceding claim, configured to provide contactless tactile feedback information to indicate the quality of a user interaction and / or to guide a user to provide tactile input.

13. A method for controlling a device, in particular in a medical procedure, comprising the steps of: - Obtaining location-based information of the user (310); - providing a first non-contact tactile stimulus to the user (306); - Obtaining contactless tactile input from the user; - providing a second tactile stimulus to the user based on the tactile input (306); - Based on the tactile input, changing parameters of the HSI and / or providing parameters to a device connected to the HSI.

14. The method according to the preceding claim, wherein the provision of the first non-contact tactile stimulus represents and / or determines an initial state of the HSI and / or a connected device (322).

15. A medical device, in particular a microscope, - comprising a HSI according to one of the preceding claims; and / or - configured to interact with an HSI according to one of the preceding claims; and / or - Carrying out the method according to the preceding claim.

Images