Method for operating an operating system, operating system and device with an operating system

Abstract

A method for operating an operating system (12), in particular of a motor vehicle (10), which comprises an input field (14) with several touch-sensitive control surfaces (16), comprising the steps of: - detecting an area in which a user (20) touches the input field (14); - determining, based on the detected area, which of the control surfaces (16) the user (20) is touching; - detecting the gaze direction (26) of the user (20); - determining, based on the detected gaze direction (26), which of the control surfaces (16) the user (20) is looking at; - selecting one of the control surfaces (16) of the input field (14) taking into account the information about which of the control surfaces (16) the user (20) is touching and which of the control surfaces (16) the user (20) is looking at; - executing a function assigned to the selected control surface (16), characterized in that - if it is detected that none of the control surfaces (16) are being touched by the user (20) is touched,the control surface (16) is selected for which it has been determined that the user (20) has directed their gaze, and – if it is detected that the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze differ, – the control surface (16) on which the user (20) has directed their gaze is selected only if the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze are no further apart than a predetermined distance (28), and – the control surface (16) touched by the user (20) is selected if the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze are further apart than the predetermined distance (28).

Description

The invention relates to a method for operating a control system, in particular of a motor vehicle, which comprises an input field with several touch-sensitive control surfaces. The invention further relates to a control system and a device with such a control system. When using touchscreens, especially with complex menus and small selection elements, errors frequently occur. For example, if a user touches an area between two touch-sensitive control areas on the touchscreen, the control area where a slight overhang of the touching finger is detected will be activated. This often leads to the wrong control area being activated—particularly due to perspective distortion caused by the touchscreen being offset from the user's line of sight—and consequently, to a function assigned to that control area being triggered that the user did not intend. DE 10 2007 057 208 A1 discloses a method for displaying image objects in a virtual image space. This method takes into account the detected viewing direction of a user when displaying pixels in a display field. German patent DE 10 302 387 A1 discloses a head-up display for use in motor vehicles. The parallax that occurs when displaying an information in the head-up display is modified so that the display appears at a comfortable distance for the viewer. EP 1 997 667 A1 discloses an information transmission device and a method for transmitting information. In this device, information displayed on a display field is adjusted depending on the detected gaze direction of a user. US 2006 / 0279528 A1 shows a display with multiple different views. The information displayed is adjusted depending on the detected user position. DE 69 715 816 T2 discloses a display system for presenting a virtual, three-dimensional image display. A parallax effect is generated depending on the detected head position of a user. From US patent 2013 / 0169560A1, a method and a system with gaze direction and gesture control are known. A user of a computer system can control a graphical user interface depending on gaze direction and by means of gestures. From US patent 2005 / 0243054A1, a system and a method for selecting and activating a target object by means of a combination of gaze direction control and key operation are known. DE 10 2012 219 958 A1 describes a method for determining an input position on a touch-sensitive display by a user. The method detects the position of the user's eyes and the position of a touch on the touch-sensitive display. Furthermore, the input position is determined based on the detected position of the eyes in relation to at least a part of the touch-sensitive display. US Patent 2010 / 0283722A1 describes a device. The device includes a coordinate input area. The coordinate input area can be positioned above a display and may form part of a touchscreen. The device further includes a first position detection unit configured to detect the first position of a finger on the display. The device also includes a second position detection unit configured to generate an estimate of a second position from which a user is looking at the coordinate input area. The first and second positions are used in combination to control the device. German patent application DE 10 2009 057 081 A1 describes a method for providing a user interface. This method detects an input gesture on a touch-sensitive surface. Furthermore, the method employs a device for tracking a user's eye position, thereby determining how frequently, within a specific time interval, the user focuses their gaze on an unchanging display context on the user interface's display area. The method also automatically detects a help situation. If the user views the display area more frequently or for a longer period than a certain threshold, this is interpreted as a help situation. EP 3 063 602 B1 describes a computing system with gaze-assisted touchscreen input. The computing system includes input registration logic to receive a touch signal indicating a user's touch on the system's touchscreen and a gaze location signal indicating the user's gaze area on the touchscreen. Furthermore, the input registration logic is used, at least partially, to determine, based on the gaze location signal and the touch signal, whether the touch occurred outside the user's gaze area.Using the input registration logic, a touch type of the touch signal is selected. The selected touch type is a non-gaze-related touch type if it is determined that the touch is outside the user's field of vision, and a non-gaze-related or gaze-related touch type if it is determined that the touch is within the user's field of vision. Based on the selected touch type of the touch signal, an input signal is then generated for the computing system using the input registration logic. The object of the present invention is to reduce the probability of incorrect operation of an operating system which comprises an input field with several touch-sensitive operating surfaces. This problem is solved by a method for operating a control system, by a control system, and by a device with a control system having the features of the independent claims. Advantageous embodiments with expedient and non-trivial further developments of the invention are specified in the dependent claims. The inventive method for operating a control system, in particular of a motor vehicle, which comprises an input field with several touch-sensitive control surfaces, has the following steps: - Detecting an area in which a user touches the input field; - Determining, based on the detected area, which of the control surfaces the user is touching; - Detecting the user's gaze direction; - Determining, based on the detected gaze direction, which of the control surfaces the user is looking at; - Selecting one of the control surfaces of the input field, taking into account the information about which of the control surfaces the user is touching and which of the control surfaces the user is looking at; - Executing a function assigned to the selected control surface. According to the invention, it is therefore provided not only to take into account the touch of touch-sensitive control surfaces of an input field, but also to detect the direction of view of the user concerned, whereby these two pieces of information are evaluated in combination before one of the control surfaces of the input field is selected and a corresponding function assigned to the selected control surface is executed. The input field itself can be, for example, part of a touchscreen or touchpad of the operating system. In the case of a touchscreen, the touch-sensitive controls can be different icons displayed within the input field. A function assigned to a selected control could be, for example, a program call linked to that specific touch-sensitive control, such as a corresponding icon. Depending on the design of the currently displayed menu interface of the input field, the controls can also be other types of controls, such as adjusting the volume of audio playback or influencing other functions.The method according to the invention now enables improved safeguarding to determine which of the touch-sensitive control surfaces a user actually intends to select, since, in addition to the touch detection of the input field, the user's gaze direction is also evaluated and taken into account. This significantly reduces incorrect operation resulting from the selection of control surfaces that the user did not intend to select. In an advantageous embodiment of the invention, if it is detected that the user is touching more than one of the control surfaces, the system selects the control surface that the user has determined to be looking at. For example, the control surfaces may be arranged relatively close together or displayed closely together on the input field, so that a user, especially one with somewhat wider fingers, might touch two or more control surfaces simultaneously. In this case, it is particularly useful to detect the user's gaze direction in order to make an additional comparison to determine which control surface the user actually intended to operate. This significantly reduces user safety and the likelihood of incorrect operation when using the control system. Additionally, the invention provides that if it is detected that none of the user interfaces are being touched, the interface on which the user's gaze has been determined to be directed is selected. This situation can occur particularly when the input field is relatively large and the touch-sensitive interfaces are relatively far apart. In such cases, the user might, for example, place their finger in an area of ​​the input field that lies between two touch-sensitive interfaces, meaning that the user is not actually touching either of them. By comparing this with the detected gaze direction of the user, it is nevertheless possible to select the interface that the user most likely intended to activate or select. Additionally, the invention provides that if it is detected that the user-touched control surface and the control surface on which the user is looking differ, the control surface on which the user is looking is selected. In other words, it can be provided that the gaze direction detection effectively overrides a selection of one of the control surfaces that would have been made without gaze direction detection, i.e., solely by touch detection of the input field. It is assumed that the user will typically focus their gaze more accurately on the control surface they intend to select, whereas accidentally touching the input field in an unintended location, and thus selecting a touch-sensitive control surface, is more likely to occur. Furthermore, it is additionally provided that the control surface on which the user is looking is only selected if the control surface touched by the user and the control surface on which the user is looking are no more than a predetermined distance apart. This predetermined distance can preferably be freely selected by the user of the operating system. Because this type of alignment via gaze direction detection only occurs up to a certain distance between the control surfaces in question, the continued intuitive use of the operating system can be ensured, since beyond a certain distance between the control surfaces it is relatively unlikely that the user would actually want the control surface on which they are looking to be selected instead of the control surface on which they have just placed their finger.This could be the case, for example, if the user only glanced over the input field without focusing on the specific control area they actually wanted to select. In addition, the invention provides that if the user-touched control surface and the control surface towards which the user is looking are further apart than the predetermined distance, the control surface touched by the user is selected. Thus, a gaze-direction-controlled agreement via the touch-controlled selection of the control surface is prevented beyond a certain distance between the respective control surfaces. Another advantageous embodiment of the invention provides that the user's gaze direction is detected and, based on this direction, it is determined which of the control surfaces the user is looking at, before and / or during the touch of the input field. Ideally, the user's gaze direction is continuously detected so that the period before the actual touch of the input field can be recorded, allowing for a particularly reliable estimation, based on the detected gaze direction, of which of the control elements the user actually intended to touch. The operating system according to the invention comprises an input field with several touch-sensitive control surfaces. Furthermore, the operating system comprises a touch detection device designed to detect the area in which a user touches the input field and, based on the detected area, to determine which of the control surfaces the user is touching. The operating system also comprises a gaze detection device designed to detect the user's gaze direction and, based on this direction, to determine which of the control surfaces the user is looking at. Finally, the operating system comprises a control unit designed to select one of the control surfaces of the input field, taking into account the information about which control surface the user is touching and which control surface the user is looking at, and to execute a function assigned to the selected control surface.In addition, the control device is designed to select, if it is detected that none of the operating surfaces are being touched by the user, the operating surface that has been determined to be the one the user is looking at.Additionally, the control device is designed to select, if it detects that the user-touched control surface and the control surface towards which the user is looking differ, the control device to select the control surface towards which the user is looking. If the user-touched control surface and the control surface towards which the user is looking are no more than a predetermined distance apart, the control device is designed to select only the control surface towards which the user is looking. If the user-touched control surface and the control surface towards which the user is looking are more than a predetermined distance apart, the control device is designed to select the user-touched control surface. Advantageous embodiments of the method according to the invention are to be regarded as advantageous embodiments of the operating system according to the invention, wherein the operating system in particular comprises means for carrying out the method steps. In an advantageous embodiment of the operating system according to the invention, it is provided that the input field is part of a touchscreen or a touchpad of the operating system. Another advantageous embodiment of the operating system provides that the eye-tracking device includes a stereo camera for detecting the direction of gaze. This allows the gaze direction of the user in question to be detected with particular accuracy and assigned to the relevant operating surface. The device according to the invention comprises the operating system according to the invention or an advantageous embodiment of the operating system, wherein the device is characterized in that it is a motor vehicle, a smartphone, a tablet computer, a notebook or a desktop computer. In other words, the operating system can be integrated into essentially any device in which touch-sensitive operating surfaces, for example touchpads or touchscreens, are usefully employed. Further advantages, features, and details of the invention will become apparent from the following description of preferred embodiments and from the drawings. The features and combinations of features mentioned above in the description, as well as those mentioned below in the figure description and / or shown in the figures alone, can be used not only in the combinations specified, but also in other combinations or individually, without departing from the scope of the invention. The drawing shows: Fig. 1 a schematic representation of a motor vehicle with an operating system comprising an input field with several touch-sensitive control surfaces, a touch detection device, a gaze detection device, and a control device; Fig. 2 a schematic detail view of the input field with several touch-sensitive control surfaces; Fig. 3 another schematic view of the input field; and Fig. 4 another schematic view of the input field. In the figures, identical or functionally equivalent elements are given the same reference symbols. A motor vehicle 10 with an operating system 12 is shown schematically in Fig. 1. The operating system 12 comprises an input field 14, which has several touch-sensitive control surfaces 14 (not shown in detail here). The input field 14 can, for example, be part of a touchscreen or touchpad of the operating system 12 (not shown here). The operating system 12 further comprises a touch detection device 18, which is coupled to the input field 14 and is designed to detect an area in which a user 20 touches the input field 14, for example, as in the present case with their index finger 22, and to determine, based on the detected area, which of the operating surfaces 16 the user 20 is currently touching. The touch detection device can, for example, operate capacitively and / or pressure-sensitively. The operating system 12 further includes a gaze detection device 24, which is designed to detect a gaze direction 26 of the user 20 marked by the dashed line and to determine, based on the detected gaze direction 26, which of the operating surfaces 16 the user 20 is currently looking at. Finally, the operating system 12 also includes a control device 28, which is designed to select one of the operating surfaces 16 of the input field 14, taking into account the information which of the operating surfaces 16 the user 20 is currently touching and on which of the operating surfaces 16 the user 20 has directed his gaze, and to execute one function assigned to the selected operating surface 16. The eye-tracking device 24 can also include a stereo camera for eye-tracking, by means of which the eye-tracking direction 26 of the user 20 can be recorded particularly precisely and on the basis of which it can be determined which of the operating surfaces 16 the user 20 is currently looking at. The following explains a method for operating the operating system 12 with reference to Fig. 2, Fig. 3 to Fig. 4, in which the input field 14 is schematically represented in different operating situations. The eye-tracking device 24 continuously detects the gaze direction 26 of the user 20 and determines which of the control surfaces 16 the user 20 is currently looking at. As soon as the user 20 touches the input field 14, in this case with their index finger 22, the area that the user 20 is currently touching is detected. Based on the detected area, it is determined which of the control surfaces 16 the user 20 is currently touching. In the case shown in Fig. 2, the user 20 touches two control surfaces 16 simultaneously with their index finger 22, since they have placed their finger 22 centrally between the two control surfaces 16, with the distance between the control surfaces 16 being so small that their finger 22 touches both of them. Based on the continuously recorded gaze direction 26 of the user 20, it is also determined which of the control surfaces 16 the user is looking at, in this case, the upper left control surface 16. In the case shown here, it is therefore detected that more than one of the control surfaces 16 is being touched by the user 20, whereby in this case the control surface 16 is selected for which it has been determined that the user 20 is looking, i.e., the uppermost and leftmost control surface 16. The case shown in Fig. 2 can occur in particular if the control surfaces 16 are arranged relatively close together and at the same time the user 20 has relatively wide fingers 22 in relation to the spacing of the control surfaces 16.By detecting the direction of gaze, it is possible to determine with relatively high certainty which of the control surfaces 16 the user 20 actually intended to touch, even though, as in the case shown here, he touched two of the control surfaces 16 simultaneously with his finger 22. Figure 3 shows another alternative scenario that can occur when operating the control system 12. In the case shown, the user 20 has placed their finger 22 in an unspecified gap between two of the control surfaces 16, so that their finger 22 is not touching either of the control surfaces 16. If, in this case, it is detected that none of the control surfaces 16 are being touched by the user 20, the control surface 16 is selected that has been determined to be the one the user 20 is looking at, in this case, again the uppermost left control surface 16. The scenario shown in Figure 3 can occur particularly if the control surfaces 16 are arranged relatively far apart compared to the width of the user 20's finger, so that the user 20 accidentally presses into a gap between the two control surfaces 16 without actually touching either of them. Figure 4 shows another possible scenario for operating the control system 12, in which the user 20 touches a control surface 16 while looking at a completely different control surface 16. If, as in the present case, it is detected that the control surface 16 touched by the user 20 and the control surface 16 on which the user 20 is looking differ, the control surface 16 on which the user 20 is looking is selected, in this case again the upper left control surface 16. A further boundary condition can be specified such that the control surface 16, on which the user 20 has directed their gaze, is only selected if the control surface 16 touched by the user 20 and the control surface 16 on which the user 20 has directed their gaze are no more than a predetermined distance 28 apart. Preferably, the operating system 12 can be designed such that the user 20 can set this predetermined distance 28 themselves. Thus, if the predetermined distance 28 between the respective control surfaces 16 is greater than the predetermined distance 28, the control surface 16 that the user 20 is actually touching with their finger 22 is selected instead of the control surface 16 on which the user 20 has directed their gaze. The described method and the use of the described operating system 12 are not limited to motor vehicles alone. Rather, it is possible for such an operating system 12 to also be integrated into a smartphone, a tablet computer, a notebook or a desktop computer, whereby the method can be carried out analogously in these devices.

Claims

Method for operating an operating system (12), in particular of a motor vehicle (10), which comprises an input field (14) with several touch-sensitive control surfaces (16), comprising the steps of: - detecting an area in which a user (20) touches the input field (14); - determining, based on the detected area, which of the control surfaces (16) the user (20) is touching; - detecting the gaze direction (26) of the user (20); - determining, based on the detected gaze direction (26), which of the control surfaces (16) the user (20) is looking at; - selecting one of the control surfaces (16) of the input field (14) taking into account the information about which of the control surfaces (16) the user (20) is touching and which of the control surfaces (16) the user (20) is looking at; - executing a function assigned to the selected control surface (16), characterized in that - if it is detected that none of the control surfaces (16) are being used by the User (20) is touched,the control surface (16) is selected for which it has been determined that the user (20) has directed their gaze, and – if it is detected that the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze differ, – the control surface (16) on which the user (20) has directed their gaze is selected only if the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze are no further apart than a predetermined distance (28), and – the control surface (16) touched by the user (20) is selected if the control surface (16) touched by the user (20) and the control surface (16) on which the user (20) has directed their gaze are further apart than the predetermined distance (28). Method according to claim 1, characterized in that if it is detected that more than one of the operating surfaces (16) is being touched by the user (20), the operating surface (16) is selected for which it has been determined that the user (20) has directed his gaze towards it. Method according to one of the preceding claims, characterized in that the detection of the gaze direction (26) of the user (20) and the determination, based on the detected gaze direction (26), of which of the operating surfaces (16) the user (20) has directed his gaze, takes place before and / or during the touching of the input field (14). Operating system (12) comprising: - an input field (14) comprising several touch-sensitive control surfaces (16); - a touch detection device (18) designed to detect an area in which a user (20) touches the input field (14) and, based on the detected area, to determine which of the control surfaces (16) the user (20) is touching; - a gaze detection device (24) designed to detect a gaze direction (26) of the user (20) and, based on the detected gaze direction (26), to determine which of the control surfaces (16) the user (20) is looking at; - a control device (28) designed to select one of the control surfaces (16) of the input field (14) taking into account the information about which of the control surfaces (16) the user (20) is touching and which of the control surfaces (16) the user (20) is looking at, and to execute a function assigned to the selected control surface (16).characterized in that the control device (28) is designed: - if it is detected that none of the operating surfaces (16) is being touched by the user (20), to select the operating surface (16) for which it has been determined that the user (20) has directed their gaze, and - if it is detected that the operating surface (16) touched by the user (20) and the operating surface (16) on which the user (20) has directed their gaze differ, - to select the operating surface (16) on which the user (20) has directed their gaze only if the operating surface (16) touched by the user (20) and the operating surface (16) on which the user (20) has directed their gaze are no further apart than a predetermined distance (28), and - to select the operating surface (16) touched by the user (20) if the operating surface (16) touched by the user (20) and the operating surface (16) on which the user (20) has directed their gaze differ.are further apart than the specified distance (28). Operating system according to claim 4, characterized in that the input field (14) is part of a touchscreen or touchpad of the operating system (12). Operating system according to claim 4 or 5, characterized in that the eye-tracking device (24) comprises a stereo camera for eye-tracking direction detection. Device with an operating system (12) according to one of claims 4 to 6, characterized in that the device is a motor vehicle (10), a smartphone, a tablet computer, a notebook or a desktop computer.

Images