Means of transportation, user interface and method for operating a user interface

Abstract

Procedure for operating a user interface (1) comprising the steps: • Detect (100) a user input using an input device (2) via a touch-sensitive surface (3), • Sensor-based detection (200) of a force acting on the input means (2) during user input in a direction (P1) perpendicular to a surface normal (P2) of the touch-sensitive surface (3), and • Determine (300) based on the user input and the force applied that a surface area of ​​the surface actuated by the user input has not been addressed by the user (5), • wherein the sensor-based determination of the force effect is carried out using a steering angle sensor, particularly in conjunction with a travel speed.

Description

[0001] The present invention relates to a user interface, a means of locomotion, a smart wearable, a wireless communication terminal, and a method for operating a user interface. In particular, the present invention relates to facilitating the use of a user interface while in motion.

[0002] Operating portable, touch-sensitive devices while in motion can present significant challenges for the user. Vibrations and forces occurring during movement can cause the input device to deviate from its intended trajectory for addressing a predetermined surface area, instead activating a surface area adjacent to it. While touchscreens and other solid-state touch surfaces (touchpads) have been the most common form factor to date, flexible fabric surfaces and smart wearables (intelligent clothing and accessories) are increasingly being discussed.

[0003] https: / / www.derstandard.de / story / 2000115846371 / touch-stoffe-sorgen-fuer-gefuehlvolle- elektronik showcases flexible fabric surfaces that function much like a smartphone screen. These surfaces typically have a capacitive field, which is influenced by touch with fingers or another input device. These changes are detected and then translated into input gestures.

[0004] Unintended input errors frequently occur when operating touch displays and other smart wearables, for example, if sudden accelerations occur near the display during gesture execution. Driving over a pothole is one example of this. Similarly, switching from one screen view to another can cause a user input to unintentionally refer to a new button, where the button the user intended to press was previously located. Since screen changes can sometimes be masked by light reflections, changes in lighting conditions while moving the user interface can make operation more difficult.Furthermore, noises occurring during movement can mask a sound output that is usually provided by the design of the user interface when switching from a first to a second screen, so that the user misses the usual indication of a change in screen views.

[0005] US 2014 / 0 160 048 A1 discloses a user interface which, in response to a mechanical disturbance of the user, does not allow a gesture performed by the user to be recognized as such, so that an erroneous input does not lead to a function call.

[0006] US 2017 / 0 364 207 A1 discloses a user interface for an aircraft in which a vibration-induced input error is automatically corrected.

[0007] US patent 2014 / 0062893A1 discloses a system and method for reducing the probability of accidental user input on a touchscreen. This involves using information about the terrain, map data, and sensors to distinguish between intentional and unintentional input.

[0008] US Patent 2013 / 0176264A1 discloses a system and a method for reducing unintended inputs in a portable electronic user device. An accelerometer detects impacts or shocks, and user inputs received in close temporal proximity to these impacts are, if necessary, discarded as accidental.

[0009] US 2021 / 0 382 602 A1 discloses a method for automatically correcting unintentional inputs in connection with a means of transportation. According to one embodiment, an acceleration signal is received from electronic user devices carried by occupants of the means of transportation and compared to an acceleration signal determined by the means of transportation itself. Depending on the relationship between the two signals, a user input on a touchscreen of the means of transportation is either rejected as potentially unintentional or corrected.

[0010] Based on the aforementioned prior art, it is an object of the present invention to facilitate user input under unfavorable conditions, in particular during non-uniform movements. Disclosure of the invention

[0011] The aforementioned problem is solved according to the invention by a method for operating a user interface with the features according to claim 1.

[0012] The dependent claims describe preferred embodiments of the invention.

[0013] Accordingly, a user interface is understood, for example, as an input device by means of which the user can make gesture-based inputs. In particular, the input can be detected using a touch-sensitive surface. The first step involves detecting user input via an input device. The input device can be, for example, a finger, a thumb, a hand, a stylus, a ballpoint pen, or similar. A touch-sensitive surface is used to detect the user input, which itself can be deflected from a resting position by operational shocks and / or vibrations. This does not preclude the possibility that the touch-sensitive surface is immune to the aforementioned disturbances and that the disturbances may only affect the user and / or the input device.In a second step, a force is detected by sensors that acts on the input device during user input in a direction perpendicular to a surface normal of the touch-sensitive surface. At a minimum, the force has a component oriented perpendicular to the surface normal of the touch-sensitive surface. Alternatively or additionally, a sensor-based detection of movement between the touch-sensitive surface and the input device can be performed, such that the input device moves, at least partially, in a direction perpendicular to a surface normal of the touch-sensitive surface. In other words, the sensor automatically detects that, due to the force exerted by movement or operation, the input device misses a target on the touch-sensitive surface (presumably intended by the user).In a third step, the system determines whether a surface area activated by user input was actually addressed by the user. Sometimes, there is a functional difference between the surface area actually activated and the area of ​​the touch-sensitive surface that the user intended to address. This is determined based on the user input and the force applied. This detection is sensor-based and, in particular, automatic. As a result, the user input can be ignored, conditionally accepted, or corrected by the user interface. Conditional acceptance, for example, might involve sending a message to the user, requiring them to confirm the user input.Since the message can display a larger button than the one the user previously (perhaps accidentally) pressed, the user interface can be assisted. Alternatively or additionally, a comprehensive timer message can be displayed, informing the user that if a timer expires without result (e.g., one second, two seconds, three seconds, four seconds, or five seconds), the user's input will be interpreted as intentional. The same can apply to the user interface that the user (presumably) addressed but did not activate. In this way, the operation of a user interface during movement and under the influence of forces unpredictable by the user can be facilitated. The force is determined using a steering angle sensor.If the user interface is integrated into a vehicle, the vehicle's steering angle during driving can indicate the extent to which the user's input was taken into account. For example, in a manually driven vehicle, the steering angle can be assumed to be very familiar to the user, and the user input and the resulting forces can be assumed to be well understood. However, in an automatically controlled driving situation, the steering angle may have been less familiar to the user and therefore less accurately considered. For instance, the user may not have been looking at the steering wheel at all during the input, or there may be no steering wheel or other indication of the current steering angle in the vehicle.In conjunction with determining the steering angle, the steering angle and / or the speed of the vehicle can also be changed to determine the resulting lateral forces. These forces can then be used to determine the forces acting on the input device and / or the touch-sensitive surface, depending on how the touch-sensitive surface and the input device are connected to the vehicle.

[0014] User input can be achieved, for example, via a touch-sensitive surface. This surface can be arranged on a touchscreen, for instance. Alternatively, the touch-sensitive surface can also be designed as a touchpad (not necessarily transparent). Furthermore, it should be noted that the touch-sensitive surface does not necessarily have to be flat, so that materials such as fabrics, films, textiles, etc., designed as touch-sensitive surfaces can also be used within the scope of the present invention.

[0015] The touch-sensitive surfaces can, for example, be designed as part of a smartphone, a tablet, a smart wearable (ring), a smartwatch, a smartcloth, or similar devices. Surfaces of a means of transportation, in particular screens or interior surfaces, can also be used in accordance with the present invention.

[0016] User input can preferably be monitored using a camera. This allows the camera to analyze the user input for sudden movements of the input device relative to the touch-sensitive surface and, if a movement is detected, to treat it as potentially erroneous. The camera data can also detect, for example, a bump in the road, a pothole, or similar obstruction, which provides an indirect indication that the user input may have missed its intended target.

[0017] Naturally, the force can also be determined using an accelerometer, force transducer, or similar device. Rotation rate sensors, gyroscopes, and similar devices can also be used to detect a force that negatively impacts input precision. In particular, multiple accelerometers can be used, their signals evaluated together with respect to the same user input. For example, the first acceleration signal could represent an acceleration of the touch-sensitive surface, while a second acceleration signal could represent an acceleration of the input device. If both accelerations are measured in approximately the same direction and at approximately the same magnitude, it can be concluded that the user input is not being distorted by the identical force.Otherwise, a deviation in the direction and magnitude of the force (or at least one of the two) may indicate that the input made by the user did not correspond to their intention.

[0018] The procedure can optionally include further steps, the first of which involves automatically (sensorially) determining the user's gaze direction during input. This gaze direction can, for example, provide information about where the user's attention was directed at the time of input. This can be used as an additional criterion to determine whether the user input was distorted by the force applied, or whether the user anticipated the force and was able to compensate for it successfully. The user's gaze direction at the time of input can thus be seen as an additional criterion for assessing whether the user input was distorted.

[0019] According to a second aspect of the present invention, a user interface is proposed which can be a component of a smartphone, a means of transportation, a smart wearable, a wireless communication device, or the like. The user interface has a data input, an evaluation unit, and a data output. The user interface is therefore configured to detect user input via an input device on a touch-sensitive surface using the data input. Additionally, the user interface is configured via the data input to detect, using sensors, a force and / or movement acting on the input device in a direction perpendicular to a surface normal of the touch-sensitive surface during user input. In other words, the force or movement can be detected by means of the data input.The movement is responsible for the unpredictable disruption of the input device's planned trajectory with respect to the touch-sensitive surfaces. The user interface is therefore configured to determine, based on the user input and the force applied, that a surface area of ​​the touch-sensitive surface activated by the user input was not addressed by the user. In other words, the system determines the fact or probability that the user did not actually intend to operate the button being pressed, but rather intended to manipulate and / or touch a different segment of the touch-sensitive surface. Thus, a signal can be output via the data output indicating that the actual user interface interaction is not being evaluated, not initially evaluated, or evaluated in a predefined alternative manner, and that a correspondingly different function is being executed.Thus, the user interface is set up to clearly explain the features, combinations of features and the resulting advantages of the method according to the invention in such a way that reference is made to the above explanations to avoid repetition.

[0020] According to a third aspect of the present invention, a means of transportation, smart wearable, or wireless communication terminal is proposed which has a user interface according to the second aspect of the invention. Thus, the means of transportation, smart wearable, or wireless communication terminal can also perform the features, combinations of features, and advantages mentioned above without the need for a separate discussion of the features mentioned above.

[0021] According to one aspect of the present invention, the aim is to assess whether a current input to a control element was actually intended by a person or not. If an input is assessed as unintended with a sufficiently high probability, the input is either not implemented, not implemented immediately (timer until the end of the implementation period with the possibility of user intervention), or the system even determines what the user's original, actual intended input was and then implements it, if necessary, immediately or indirectly (also after the end of a timer with the possibility of user intervention).The probability of an unintentional activation of a control element can be determined, for example, based on measurable measurements in the vicinity of the control element (input device) and / or on measurable measurements at the activated surface area, or alternatively or additionally, by the fact that certain distracting events have been detected by sensors in the context. For example, camera and / or microphone recordings can be made in the user's environment to determine the potential for distraction on the one hand and the focus of the user's attention on the other.

[0022] A control element can be, for example, an interaction element of a graphical user interface (GUI) or a mechanical operating device (e.g., a mechanical switch). The idea is that, for instance, an input made on a touch surface or via a (hardware) button should be assigned a probability indicating whether it actually corresponded to the user's intention.

[0023] Gesture / motion recognition on the user can detect that the user's movement may have been uncontrolled at the time of input. Therefore, the input, for example on a touchscreen or hardware buttons, should be ignored or interpreted differently because the probability of an unintentional input exceeds a predefined threshold. This probability can be determined using: - Wearables or smart wearables on the person / user; - Motion and / or location sensors in the room on the user's body, such as on the hand (smart ring, smartwatch, bracelet) or in the clothing (e.g. in the sleeve of a garment or in a glove; - Camera recordings of the user before and / or during operation using a camera including image processing plus object recognition plus gestures and / or motion interpretation and / or gaze direction recognition.

[0024] In this way, it can be determined, for example, that - For example, if acceleration thresholds were exceeded on a touchscreen within a specified period before or at the time of input (e.g., this could have been the effect of driving over a pothole or bump, and the user could only execute an input uncontrollably). In such a case, there is a sufficient probability that an unintended input was made by the user; - the touch of the display occurred within a specified time window (response time plus a predefined amount of X seconds) after switching from an old to a new menu with different or differently sorted control buttons.

[0025] After switching to a new user interface, the user could not have reacted quickly enough to make a correct input. This input would then most likely have been unintentional and, according to the invention, can be automatically ignored or even corrected. If it is recognized that the user's input was with a sufficiently high degree of probability unintentional, it can be handled as follows: - the user's input may, for example, be deemed invalid or ignored, and the user will be informed that the input would have to be repeated if necessary; - Upon determining a sufficiently high level of certainty regarding the user's actual intention, a message can be displayed to provide enlarged buttons allowing the user to choose between the two options. Alternatively or additionally, a timer can be started, which triggers a preset function if the user does not make a different selection before the timer expires; - User input can be automatically corrected. For example, it can be determined that the user could not have reacted quickly enough to enter the correct input on the newly displayed user interface when switching from an old to a new one. In this case, it is determined that the input belonged to the previous user interface. Even after switching to a new interface, the input can still be interpreted as belonging to the old interface, allowing, for example, the desired function to be executed. Alternatively, based on movements (e.g., using motion sensors, camera footage, or an accelerometer signal), it can be determined with sufficient accuracy what input the user actually intended.

[0026] The context can be considered when determining the probability of whether a user input is genuinely intended or an erroneous input (i.e., the extent to which the input was disrupted by the environment, the user's state, and / or the driving situation). Possible influencing factors include the following: - Environmental conditions: noise level inside and outside the vehicle, light intensity, lighting conditions, air temperature (outside and / or in a vehicle cabin), weather; - User states: posture, movement, gestures, health status, fatigue level, stress level, bio-vital data, conversation situation in a vehicle cabin, telephone conversation; - Calendar information regarding planned or past appointments, which provides an indication of the user's comprehension and reaction time; - Time of day / day of the week / season: for example, the user may be more relaxed in the morning than in the evening, or more tense on weekdays than on weekends. The season and pollen count can also characterize the user in terms of their senses and perception; - Where appropriate, the number of passengers in the means of transport and / or the identity / age (especially children) of the identified passengers may be taken into account; - Mode of transport (on foot, by bicycle, by car, by commuter train, etc.). In particular, a change in the mode of transport can be automatically detected, and the evaluation can be modified accordingly so that the sensors no longer used are no longer employed. Instead, the sensors of the newly used mode of transport are included in the determination of force and user input. This applies especially to acceleration sensors, which become available due to the new mode of transport. The type of control element itself, the object on which the control element is located, and how frequently incorrect user inputs have been made in the past can be stored.For example, such frequency may refer to a predefined touch surface, a specific button, a specific control element on a smartphone or smartwatch, a surface area on smart clothing (smartcloth), or vehicles of any kind. • The touchscreen is touched within a predefined time window (e.g., within the reaction time) after the user interface switches from an old to a new menu with different or differently arranged buttons. After switching to a new user interface, the user could not have reacted quickly enough to make an intended, correct input on the new interface. This input would then most likely have been unintended. Therefore, the system determines that the input belongs to the old user interface. Even after switching to a new user interface, the input (which occurred, for example, within the reaction time) can still be interpreted as belonging to the old interface, thus allowing a desired function to be executed. • The probability determination of whether a function to be executed is actually desired by the user's input or whether an incorrect input was executed can take into account the environment and the user's state (i.e., to what extent they were disturbed by the environment or their own state during the input): ◯ User's environmental conditions: noise level inside and outside the vehicle, light intensity, lighting conditions (glare), visibility, air temperature (outside and / or inside a vehicle cabin), weather (e.g., whether the user is wearing wet clothing) ◯ User states: health status, stress level, thermal state (e.g., the user is hypothermic), bio-vital data, fatigue level, conversation situation in a vehicle cabin or telephone conversation, posture, movements, gestures ◯ Driving situation (e.g. turning situation in city traffic) ◯ Calendar information (scheduled or past appointments) ◯ Time of day / Day of the week / Season ◯ The type of control element itself, the object on which it is located, and how frequently incorrect inputs have been made by the user in the past. ◯ (if applicable) Number of passengers in a vehicle, identified individual passengers ◯ Wearables on the person themselves detect uncoordinated movements of the person (smart ring, smartwatch, smart bracelet, smart clothing with sensors)

[0027] Inputs that could be particularly disruptive to the user as the driver, and which could, for example, significantly distract them while driving (based on experience), or be so disruptive that they are unable to concentrate on driving for a certain period of time with a probability exceeding a certain threshold, require a higher probability of being considered an intended input than inputs that have less impact on concentration or physical condition. For example, adjusting the climate control system with an airflow towards the eyes could cause them to close their eyes, or suddenly turning up the music loudly or opening the windows at high speeds could cause a startle response. In such cases, the calculated probability of an intended input should be higher than in other cases, such as simply changing the radio station.

[0028] The invention will now be explained in more detail using exemplary embodiments. These will be shown below: Fig. 1 a schematic representation of an embodiment of a means of locomotion designed according to the invention with a user who wears smart trousers as an embodiment of a smart wearable and operates a touch screen of the means of locomotion as an embodiment of a wireless communication terminal; Fig. 2 a schematic representation of a data input of an embodiment of a user interface according to the invention; and Fig. 3 A flowchart illustrating the steps of an embodiment of a method according to the invention for operating a user interface. Embodiments of the invention

[0029] Fig. Figure 1 shows a passenger car 10 as an embodiment of a means of transport according to the invention, in which a driver 5 is seated as the user. The driver 5 is within the detection range of an interior camera 6, which is connected to a data input 14 of an electronic control unit 12. The data input 14 is also connected to a touch-sensitive surface 3 of the touchscreen 4. The touchscreen 4 itself is connected to a data output 13 of the electronic control unit 12. An acceleration sensor 7 is arranged within the electronic control unit 12, which detects the passenger car 10 traveling on the road 9 over a bump 8.The signal from the acceleration sensor 7 indicates to the user interface 1 that the car 10 is executing an upward movement in the direction of arrow P1, which has a component perpendicular to a surface normal P2 on the touchscreen 4 and its touch-sensitive surface 3. Due to this movement, the driver's 5 input device 2 (hand) deviates from its intended trajectory and misses its target on the touch-sensitive surface 3. The same applies to an area of ​​the touch-sensitive surface 3 of the Smart Cloth 11, which the driver 5 attempts to operate while driving.Knowing the upward movement in the direction of arrow P1, the user interface 1 can correct the position actually operated by the driver 5 on the touch-sensitive surface 3 such that a surface area which would most likely have been touched without the upward movement of the car 10 is assumed to be operated and the function associated with it is triggered.

[0030] Fig. Figure 2 shows input variables of a user interface of a second embodiment of the present invention. The evaluation unit 12 has a data input 14 which receives a steering angle φ, a travel speed V, a first acceleration a1, and a second acceleration a2. The first acceleration a1 is the acceleration that the user interface undergoes or experiences. The second acceleration a2 is the acceleration that the smart wearable experiences on the user's hand. By calculating the difference, the differential acceleration, and thus the influence on the trajectory between the touch-sensitive surface and the input device, can be determined. By computationally reversing this influence, the position originally addressed by the user can be determined, and the function associated with it can be triggered.

[0031] Fig.Figure 3 shows steps of an embodiment of a method according to the invention for operating a user interface. In a first step 100, user input is detected by means of an input device in the form of a user's hand via a touch-sensitive surface. This can be done via the capacitive or otherwise touch-sensitive surface of the user interface. In step 200, a force and / or movement is detected by sensor, which acts on the input device in a direction perpendicular to a surface normal of the touch-sensitive surface during the user input. In other words, a temporal relationship between the force / movement and the operation of the touch-sensitive surface is determined such that the force / movement can be classified as unpredictable for the user.In step 300, based on the user input and knowledge of the force applied, it is determined that a surface area activated by the user input was not actually addressed by the user. In other words, the user input is classified as erroneous or invalid, and the function associated with the actually touched button / actuated control is not triggered. Instead, the user interface may display a message indicating that after a (displayed) timer expires, the button or surface area presumably addressed by the user will be evaluated as activated unless the user intervenes in time. Reference symbol list 1 User interface 2 Hand / Input device 3 touch-sensitive surfaces 4 Touch-Screen 5 drivers / users 6 Interior camera 7 Accelerometer 8. Unevenness of the ground 9 lanes 10 cars 11 Smartcloth 12 electronic control unit 13 Data output 14 Data input 100 to 300 process steps a1 first acceleration a2 second acceleration P1 vertical direction P2 surface normal V cruising speed φ Steering angle

Claims

[1] Method for operating a user interface (1) comprising the steps: • Detect (100) a user input using an input device (2) via a touch-sensitive surface (3), • Sensor-based detection (200) of a force acting on the input means (2) during user input in a direction (P1) perpendicular to a surface normal (P2) of the touch-sensitive surface (3), and • Determine (300) based on the user input and the force applied that a surface area of ​​the surface actuated by the user input has not been addressed by the user (5), • wherein the sensor-based determination of the force effect is carried out using a steering angle sensor, particularly in conjunction with a travel speed. [2] Method according to claim 1, wherein the user input is determined by means of the touch-sensitive surface (3). [3] Method according to claim 1 or 2, wherein the touch-sensitive surface (3) is part of a - Smartphones and / or - Tablets and / or - Smart Wearables (11) and / or - means of transport (10) is. [4] Method according to one of the preceding claims, wherein the user input is monitored, in particular determined, by means of a camera (6). [5] Method according to one of the preceding claims, wherein the force effect is determined by means of an acceleration sensor (7). [6] Method according to one of the preceding claims, wherein the force effect is determined by means of a smart wearable (11). [7] Method according to claim 6, wherein the force effect is determined as a difference between a signal recorded by means of an accelerometer of the smart wearable (11) and a signal recorded by means of a further accelerometer (7). [8] Method according to any of the foregoing claims further comprising: - Determining the user's gaze direction (5) during user input and depending on the gaze direction - Determine that a surface area of ​​the surface actuated by the input means (2) has not been addressed by the user (5). [9] User interface comprehensive: • a data input (14), • an evaluation unit (12) and • a data output (13), wherein the user interface (1) is set up via the data input (14), • to detect user input via an input device (2) via a touch-sensitive surface (3), • to determine, by means of a steering angle sensor, in particular in conjunction with a travel speed, a force acting on the input means (2) during user input in a direction (P1) perpendicular to a surface normal (P2) of the touch-sensitive surface, and • to determine, based on the user input and the force applied, that a surface area of ​​the touch-sensitive surface (3) actuated by the user input has not been addressed by the user (5). [10] Means of transport (10) or smart wearable (11) or wireless communication terminal comprising a user interface (1) according to claim 9.

Images