Method for controlling a device, in particular a hand prosthesis or a robot arm, and arrangement for carrying out the method

Abstract

Method for controlling a device, in particular a hand prosthesis (1) or a robot arm, characterized in that a) at least one marker (5) positioned on or in relation to the device is detected by a camera (11) mounted on an operator, b) wherein the camera (11) attached to the operator is positioned on the operator's head in such a way as to move with the head when it moves, c) from the detection of at least one marker (5) a predefined movement of the operator's head together with the camera (11) is detected, d) wherein the predefined movement of the operator's head in the form of a line of sight (15) is detected by the camera (11) using camera tracking and e) is used to trigger a corresponding action of the device.

Description

[0001] The invention relates to a method for controlling a device, in particular a hand prosthesis or a robotic arm. Such methods find application, for example, in the field of controlling electric neurohand prostheses.

[0002] Current approaches to controlling such electrical neurohand prostheses are based on measuring electromyography (EMG) signals from existing muscles or evaluating thought patterns using a brain-machine interface and deriving commands for the prosthesis from them.

[0003] In the field of neuroprosthetics, for example, in "David P. McMullen, ea (July 22, 2014). Demonstration of a Semi-Autonomous Hybrid Brain-Machine Interface Using Human Intracranial EEG, Eye Tracking, and Computer Vision to Control a Robotic Upper Limb Prosthetic. IEEE TRANSACTIONS ON NEURAL SYSTEMS AND REHABILITATION ENGINEERING," a complex system was implemented with which robotic arm prostheses can be controlled. A brain-machine interface is used for this purpose. Eye tracking is used to select options. Such approaches are frequently found in the literature, for example in "HALES, Jeremy; ROZADO, David; MARDANBEGI, Diako: Interacting with objects in the environment by gaze and hand gestures. In: Proceedings of the 3 rdInternational Workshop on Pervasive Eye Tracking and Mobile Eye-Based Interaction. 2013, pp. 1-9." Further robot arm controls are also known from WO 2018 / 165047 A1 and "ZENG, Hong [et al.]: Closed-loop hybrid gaze brain-machine interface based robotic arm control with augmented reality feedback. Frontiers in neurorobotics, 2017, 11th vol., p. 60." Furthermore, conventional prostheses are often controlled using EMG signals.

[0004] However, these approaches have the disadvantage that they are not very intuitive and require extensive training by users or operators.

[0005] The present invention is therefore based on the object of creating a method for controlling a device, in particular a hand prosthesis or a robotic arm, that avoids the aforementioned disadvantages and enables convenient control. Furthermore, the invention is based on the object of enabling trouble-free and precise control of a device, in particular a hand prosthesis or a robotic arm, in a simple manner, thereby also preventing operating errors.

[0006] This object is achieved according to the invention by a method having the features of claim 1.

[0007] According to the invention, a camera worn by a user or operator, preferably on the head, is used to control or activate a device, in particular a (randomly active, motorized) hand prosthesis or a (randomly active, motorized) robotic arm, by means of camera tracking. For this purpose, markers, such as LEDs, are positioned on or in relation to the device, which are detected or recognized (located) by the camera, preferably within a defined area of ​​a defined, in particular central, line of sight.

[0008] Such detection and thus control of the device is only possible when the line of sight is directed primarily toward the markers or, regardless of the marker position, to a predefined position (e.g., top left). This alone can easily prevent unintentional control of the device and thus incorrect operation, ensuring robust control.

[0009] The line of sight thus represents the viewing direction of the user or the observer, whereby this viewing direction in the sense of the invention is understood to mean the orientation of the user's field of vision, in particular the orientation of augmented reality (AR) glasses to the device, and not the viewing direction of the eyes.

[0010] Furthermore, according to the invention, only upon such detection of a marker or orientation of the line of sight to the marker is a feedback event output on an output unit and displayed to an operator. Only under this condition can commands for controlling the device be generated, thus further reducing the risk of incorrect operation. According to the invention, these commands are generated by detecting (different) predefined movements of the operator (together with the camera). The operator's movements cause a corresponding movement of the line of sight (of the camera), preferably in a defined region, which is detected by camera tracking and used to trigger a corresponding action of the device.

[0011] For the purposes of the invention, predefined movements also include commands learned or refined during use (by means of teach-in, AI algorithms, etc.).

[0012] This control of a device is intuitive and easy to learn by using the body movements, especially the head movements, of an operator. Furthermore, in an advantageous embodiment of the invention, the output of a feedback event makes it clear to the operator when movements are interpreted as control signals, thus reducing or even eliminating the risk of unintentional operating errors.

[0013] An output unit within the meaning of the invention can be, for example, an audio output unit, a visual output unit (lamp, screen, etc.) or a haptic output unit (pressure on the skin surface).

[0014] In a preferred embodiment of the invention, the camera arranged on the operator is arranged on the head of the operator in such a way that the camera and thus the field of view of the camera, in particular its line of sight, is moved when the head is moved.

[0015] For example, the camera of augmented reality (AR) glasses, which are put on and worn by an operator to control a device, can be used for this purpose. In this case, a display of AR glasses can be used as the output unit, allowing not only the operator's head movement to be used to control the device, but also the display of feedback signals and possibly the movement of the line of sight for the operator to be conveniently displayed.

[0016] In this embodiment of the invention, the camera is an integral component of the AR glasses, so that the line of sight, preferably the central line of sight, is defined in a fixed relationship to the AR glasses. By tracking the markers on the device, in particular the hand prosthesis, their position and orientation relative to the line of sight can be tracked.

[0017] In a further embodiment of the invention, an (opening) window, particularly in the form of a virtual command field, is displayed as a feedback event in the AR glasses. Its position is fixed or fixed (i.e., at a fixed position) relative to the position of the device. The virtual command field is thus spatially fixed to the markers and is preferably displayed when the line of sight is in a defined spatial proximity to the command field.

[0018] Of course, it's also conceivable for such a window to be permanently displayed on the AR glasses display. A different action, such as a lighting up of the window frame, a change in the window's shading or background color, or other information on the AR glasses display, could then occur as a feedback event.

[0019] In such a window, not only information from the device but also the movements of the line of sight within the window can be displayed in a convenient way for the operator. The movement can also be displayed, for example, as a tracer, as feedback for the user.

[0020] For example, various predefined lines, figures, etc. can be displayed in this window along with information about the corresponding control commands to be executed (as a preview). These lines and figures can be followed by the operator's head movement using the currently displayed line of sight. However, it is also conceivable that movements of the line of sight can be detected without such a preview.

[0021] In a further embodiment of the invention, so-called eye tracking can also be used to enable more detailed control commands and / or an expansion of the command set. For example, eye tracking could be used to incorporate additional detected actions, such as blinking or the operator's gaze direction. Of course, such eye tracking requires an additional camera, which is directed at one of the operator's eyes instead of the surroundings. Such an eye-tracking camera can preferably also be part of AR glasses.

[0022] The various embodiments of the invention enable a simple and robust method for controlling the device, in particular the prosthesis. Since in most cases the device to be controlled and its actions (e.g., the gripping process of the prosthesis) are visually controlled by the operator, it is particularly advantageous to create a visual control mechanism with a head-mounted camera, in particular AR glasses, and camera tracking, allowing the device to be controlled with minimal head movements relative to the camera.

[0023] In a further embodiment of the invention, eye tracking can also be used to trigger actions or control commands for the device, or to detail actions already generated (using camera tracking). For this purpose, eye alignment is observed using an additional eye-tracking camera, and predefined actions, such as blinking, looking down, etc., are recognized to trigger corresponding actions to control the device.

[0024] Further advantageous embodiments of the invention emerge from the dependent claims. The invention is explained in more detail below using an exemplary embodiment, shown in the drawing, of an operator who controls a hand prosthesis using AR glasses.

[0025] The drawing shows: Fig. 1 a schematic representation of an operator wearing AR glasses and a hand prosthesis; Fig. 2 an enlarged schematic representation of a section of Fig. 1 and Fig. 3 a schematic representation according to Fig. 1 with additional eye tracking.

[0026] The Fig. The hand prosthesis 1 shown in Figure 1, as a non-limiting example of a device to be controlled, has a number (1-n) of markers, for example four markers 5 as shown, preferably in the form of LEDs, in order to be recognized by a camera 11 in its position within a coordinate system B.

[0027] The camera 11 is worn by a user or an operator, for example as part of AR glasses 7, so that the position of the camera and thus its field of view 13 changes with the movement of the field of view 9 of the user, in particular caused by a head movement, in the coordinate system B.

[0028] If the hand prosthesis 1 with the markers 5 is located in the field of view 13 of the camera, the markers 5 can be recognized. If the camera 11, preferably mounted on the AR glasses 7 (or corresponding camera image analysis software), sees or recognizes the markers 5, the position and orientation of the prosthesis with 2 to 6 degrees of freedom in the coordinate system A of the glasses 7 can be determined by tracking, depending on the number of marker LEDs. The transformation between coordinate system B of the hand prosthesis 1 and coordinate system A of the viewer or AR glasses 7 is determined using a suitable (trigonometric) algorithm. This creates the prerequisite for evaluating the viewing direction (line of sight 15) of the viewer or user. For the purposes of the invention, the user's viewing direction is understood to mean the orientation of the AR glasses 7 relative to the hand prosthesis 1 and not the viewing direction of the eyes.

[0029] In addition, movements of the hand prosthesis 1 can be detected from the position changes of the markers 5 detected by camera tracking and related to the coordinate system A of the camera.

[0030] In a preferred application of the invention, the camera 11 has a large aperture angle and thus the largest possible field of view 13 (for example, greater than or equal to 80°). The distance between the camera 11 or AR glasses 7 and the markers 5 to be tracked is typically less than one meter.

[0031] Within the field of view 9 of the viewer or the AR glasses 7, a preferably central line of sight 15 is defined as the viewing direction, the direction of which changes according to a change in the field of view 13 of the camera 11 and thus the user's viewing direction. Consequently, the viewing direction of the defined line of sight 15 relative to the prosthesis 1 can be determined from the movements of the glasses 7, or the movement of the line of sight 15 relative to the hand prosthesis 1 can be specified. This creates the prerequisite for evaluating the viewer's viewing direction.

[0032] If the line of sight 15 passes a defined region in the coordinate system B, for example an area close to the markers 5, or if markers 5 of a prosthesis 1 are detected within the field of view 13 of the camera 11, or within a predefined smaller section, for example a slightly enlarged area around the line of sight 15, a user is informed, as in Fig. 2, a window in the form of a visual command field 3 (of any size, for example, approximately 10 to 15 cm from the viewer's perspective) is displayed on the display of the AR glasses 7, preferably near the prosthesis. This command field 3 is located in coordinate system B, so that a movement of the user's field of vision 9 and thus of the visual ray 15 (in coordinate system A) results in a relative movement to each other.

[0033] The movements of the visual ray 15 on the command field 3 can, as explained below, be evaluated, for example, by means of a corresponding evaluation device (including software) in order to determine a corresponding control command for the prosthesis 1.

[0034] As from Fig. As can be seen in Figure 2, the line of sight 15 can be moved on the command field 3 by a movement of the user's field of vision 9, in particular by a head movement. Preferably, the already performed movement 17 of the line of sight 15 on the command field 3 is displayed visibly to the user, for example, as an oblique line (from top left to bottom right), preferably in the form of a tracer. This provides the user with feedback on their movements of the viewing direction or line of sight 15 on the command field 3.

[0035] In this way, a user can draw specific figures that are used to control the hand prosthesis 1.

[0036] For example, simple different figures, such as lines of different orientations (horizontal from left to right, vertical from top to bottom, from top left to bottom right, from bottom left to top right and in the opposite direction, etc.), circles, or similar, can be predefined for different actions of the prosthesis 1. If a user draws a corresponding figure with the line of sight 15 in the command field 3 and recognizes it as such (within a predefined tolerance), the action assigned to the recognized figure is executed or the hand prosthesis is controlled accordingly.

[0037] Examples of commands can be: 1. Movement from lower right corner to upper left corner → prosthesis opens hand completely. 2. The viewer's line of sight only partially passes through the command field from the upper left corner to the lower right corner → the prosthesis closes the hand halfway. 3. Partial movement in the opposite direction as in 2. → Prosthesis opens hand halfway. 4. Circular movement of the direction of gaze → move thumb.

[0038] Of course, the commands can be customized according to the functions of the hand prosthesis 1. For hand prostheses that automatically switch off when a certain contact pressure is applied, for example, only the on / off command is sufficient.

[0039] If drawing certain patterns by moving the head within the command field 3 is not sufficient, the control of the hand prosthesis 1 can be carried out as shown in Fig. 3, can be enhanced by eye tracking. Eye tracking can be used to determine additional independent information, namely the eye orientation 23, for example, of the user's left eye 21 (shown only for clarity using a direction to an object 25), as well as the blinking of eye 21. This also makes it possible to derive commands from the combination of the position of the prosthesis 1 and the eye orientation 23.

[0040] If the prosthesis is to be used as a replacement for a computer mouse, for example, the movements of prosthesis 1 can be directly translated to the movements of the cursor. In this example of the computer mouse, the cursor position can be derived from the position of prosthesis 1, and selection ("clicking") can be achieved by blinking or focusing the mouse pointer. Of course, a combination for controlling only prosthesis 1 is also conceivable. For example, camera tracking could be used to move prosthesis 1, and blinking could open a closed hand, for example to release an object held in the hand.

[0041] Eye tracking can also be used to support camera tracking in controlling a prosthesis 1 in certain patient groups (e.g. stroke patients or patients with amyotrophic lateral sclerosis, in whom head movements are only possible to a limited extent).

[0042] As in Fig. As shown in Figure 3, the AR glasses 7 have an additional eye tracker 19 or an eye tracking camera 19 directed at the left eye 21 to implement the additional visual control, in order to detect the eye alignment 23 and a blink of the eye 21. The prosthesis 1 also carries the marker 5 in order to be able to determine its position relative to the glasses 7. The overlays near the prosthesis are not absolutely necessary in this approach, but are not excluded and depend on the specific application. List of reference symbols 1 hand prosthesis 3 virtual, displayed command field (display) 5 markers 7 AR glasses 9 Field of view of the AR glasses 11 Camera 13 Camera field of view 15 Direction of view of the user or operator (line of sight) 17 Movement of the line of sight on command field 3 19 Eye Tracker (Camera) 21 left eye 23 Eye alignment 25 objects

Claims

[1] Method for controlling a device, in particular a hand prosthesis (1) or a robot arm, characterized by , that a) at least one marker (5) positioned on or in relation to the device is detected by a camera (11) arranged on an operator, b) wherein the camera (11) arranged on the operator's head is arranged in such a way as to be moved when the head is moved, c) from the detection of the at least one marker (5) a predefined movement of the head of the operator is detected together with the camera (11), d) wherein the predefined movement of the operator’s head in the form of a line of sight (15) is detected by the camera (11) by means of camera tracking and e) is used to trigger a corresponding action of the device. [2] Method according to claim 1, characterized bythat a feedback event (3) is output on an output unit, which indicates the recognition of the at least one marker (5) to the operator. [3] Method according to one of the preceding claims, characterized by that the operator is a wearer of AR glasses (7). [4] Method according to claim 3, characterized by that a camera of AR glasses (7) is used as the camera (11) and a display of AR glasses (7) is used as the output unit. [5] Method according to claim 3 or 4, characterized by that as a feedback event in the AR glasses (7) a window (3) is displayed, which is displayed in a fixed position relative to the position of the device. [6] Method according to claim 5, characterized by that information about the device is displayed in the window (3). [7] Method according to claim 5 or 6, characterized bythat the predefined movement of the line of sight (15) must occur within the window (3) in order to be recognized. [8] Method according to one of the preceding claims, characterized by that the eye alignment and / or blinking is additionally detected by means of a further eye-tracking camera (19) in order to trigger a corresponding action of the device according to predefined eye movements and / or blinking or to detail existing actions. [9] Arrangement comprising a device, in particular a hand prosthesis (1) or a robot arm, and AR glasses (7) for carrying out a method according to one of the preceding claims, wherein a) at least one marker (5) is positioned on or in relation to the device in order to be recognized by a camera (11) of the AR glasses (7) worn by an operator, b) wherein the camera (11) arranged on the operator's head is arranged in such a way as to be moved when the head is moved, c) the camera (11) or the AR glasses (7) is designed such that the at least one marker (5) is recognized and, in the event of recognition, a feedback event (3) is output on a display of the AR glasses (7), d) the camera (11) or the AR glasses (7) are designed such that, upon detection of the at least one marker (5), a predefined movement of the operator's head is detected together with the camera (11), e) the camera (11) or the AR glasses (7) are designed such that the predefined movement of the operator's head in the form of a line of sight (15) is detected by means of camera tracking by the camera (11) or by the AR glasses (7) and is used to trigger a corresponding action of the device.

Images