Method for operating an operator control system, and operator control system

Abstract

In the method for operating an operator control system, a controllable function of the operator control system is detected, and at least one potential control action type for the controllable function is determined according to the controllable function. A control history for the potential control action type of the controllable function is detected, and an output is generated using the control history and is output, said output comprising an indication of the execution of the potential control action type. The operator control system comprises a control unit (8) and an output unit (3) which is coupled to the control unit (8). The control unit (8) is configured to detect a controllable function of the operator control system, to determine at least one potential control action type for the controllable function according to the controllable function, and to detect a control history for the potential control action type of the controllable function. The control unit (8) is also configured to activate the output unit (3) such that an output comprising an indication of the execution of the potential control action type is output.

Description

[0001] The present invention relates to a method for operating an operating system and an operating system, in particular in a vehicle.

[0002] Modern everyday life is characterized by a multitude of devices, the operation of which can be carried out in different ways depending on the context. Various operating concepts have been proposed for this purpose, such as control via keyboards, gestures, voice, or pointing devices, each with specific advantages and disadvantages depending on the application. In some cases, different operating modes may be required to achieve the same system response, which the user must be aware of and choose between.

[0003] In familiar systems, it is often intended that a user learns the available operating procedures using a manual or an introductory video, for example, by playing a video during the initial setup of the system and / or making it available for later viewing. The user is often confronted with a larger amount of information than they can process and must invest considerable time learning how to operate the system.

[0004] US patent 2015 / 0175172 A1 describes a system for gesture-based input with haptic feedback. A gesture is detected and corresponding feedback is output. Voice control is also possible. Various gestures are described that can be captured using a camera and are assigned to different controllable actions within the system.

[0005] US patent 2004 / 0189720 A1 discloses an architecture for controlling a computer using gestures, in which gestures can be used in combination with voice commands.

[0006] German patent DE 10 2009 039 114 A1 proposes an operating device for a vehicle in which a user's body part is detected in one of three spatial areas and a corresponding object is displayed for a graphical object. For example, a number and a scale can be displayed.

[0007] US patent 2018 / 0307347 A1 describes a method for detecting gestures using a capacitive sensor. The user can define gestures for specific commands in a database and also learn predefined gestures by consulting a guide.

[0008] German patent application DE 10 2006 037 156 A1 discloses a method for operating an interactive control device in which a control element is displayed and a user's approach to the control element is detected. The control element is then displayed in an optimized manner, for example by enlarging it. The adjustment of the display can also be controlled by means of a gesture.

[0009] A method for adapting user interfaces is known from US Patent 2018 / 365025 A1. This method determines the user experience level based on the use of a software application and other recorded factors. Based on this user experience level, the user interface is adapted to provide the user with an enhanced experience.

[0010] The present invention is based on the objective of providing a method and an operating system in which a user is supported in recognizing and applying different operating options.

[0011] According to the invention, this problem is solved by a method with the features of claim 1, an operating system with the features of claim 8, and a vehicle with the features of claim 12. Advantageous embodiments and further developments are described in the dependent claims.

[0012] In the method according to the invention, among other things, an operable function of the operating system is detected, and depending on the operable function, at least one potential operating action type for the operable function is determined. An operating history for the potential operating action type of the operable function is recorded, and an output is generated and displayed based on this operating history. This output includes an indication of the execution of the potential operating action type.

[0013] This method advantageously allows for particularly flexible, targeted, and efficient support of users when operating a system using specific types of user actions. To this end, the user's operating history is analyzed, which can be used to determine, in particular, which types of user actions require improvement or training for the user.

[0014] The process begins by detecting an operable function. Specifically, it identifies which function or functions are currently operable. This could, for example, be the state of a device in which user input or other interaction with the device can be detected. At least one operating action is required for operation, allowing the user to interact with the device. When a user interacts with the device, the operating system detects the user's action and generates a control signal, which is then transmitted to the device to activate or deactivate a function, input an adjustable parameter, or send an instruction.

[0015] When operating a function, a specific type of operating action can be assigned to each action by which a function is currently operated. This means, in particular, that the concrete operating action actually performed by the user is assigned to an abstract model of an operating action.

[0016] In one embodiment of the method according to the invention, the potential operating action type comprises a gesture in three-dimensional space, a gesture on a touch-sensitive surface, voice input, or the operation of a push, slide, or rotary switch. An operating action type can further include, for example, pressing a button, rotating a control element, or otherwise acting upon it. This advantageously allows the user to be particularly well supported in performing suitable operating actions.

[0017] For example, a spoken utterance is captured using a microphone and assigned to a specific voice command. This takes into account that the actually captured utterance typically does not correspond exactly to the predefined, idealized model of the operating action type for the specific voice input, but rather that deviations occur, such as noise during capture, background sounds, and inconsistent sound production when pronouncing the voice command. Similarly, a gesture in a three-dimensional space can be captured and assigned to, for example, a swiping gesture in a specific direction, whereby a model of the corresponding operating action type is determined for the actually captured gesture.

[0018] Furthermore, operating modalities are defined in particular, which group together a set of specific operating action types, especially through common characteristics of the generation, execution, and / or recording of the operating action types of an operating modality. For example, an operating modality "voice input" can encompass various operating action types, especially different individual voice commands. In another example, an operating modality "gesture control" for three-dimensional space can encompass various gestures, such as swiping gestures right and left, swiping gestures up and down, a confirmation gesture, and / or a point-and-move gesture, which can be defined, for example, as a volume gesture. Other operating modalities can be, for example, gestures on a touch-sensitive surface or the activation of physical operating devices.

[0019] To perform a gesture, an object of action can be used, in particular the user's hand. In the following explanations, statements relating to a user's hand as an object of action should be generalized to other objects of action, such as a pen.

[0020] A "gesture" is understood to mean, in particular, a specific position and orientation, that is, a pose of an object being actuated, such as a user's hand, or a specific movement performed with the object being actuated. Gestures can be designed in a manner known per se. They include, in particular, pointing gestures, swiping gestures, and gestures used in everyday life, for example, hand rotations, grasping gestures, and combinations of several such gestures, possibly performed in rapid succession. The gestures are performed within a detection space. This space can be spatially related to a specific device, such as a dispensing device, and may, for example, be adjacent to it. A gesture can also involve approaching a specific spatial position, such as a device or a position on a display surface.Gesture control provides the user with a particularly simple and intuitive operating option.

[0021] Furthermore, a gesture can be used to determine a direction in space, and depending on this direction, a position can be determined for which a pointing gesture is defined. For example, it can be determined which point in a display area a user is pointing at, and this information can be used in various ways for operation. A selection can be made, or the pointing at a specific area can be evaluated as input. Selecting a graphic object in a display area can be visualized, which has the advantage that the user receives immediate feedback on their action. The point in the display area can also be visualized even if no object has yet been identified in the surrounding area. This prevents the user from having to search among empty objects until they find a specific area or the desired graphic object.This can further accelerate the selection process.

[0022] Gestures can be captured contactlessly using a camera system that records time-resolved video data from the detection area and, with the help of a connected analysis unit, assigns the captured user movements to specific gestures. Alternatively or additionally, gestures can be captured using resistive and / or capacitive surfaces.

[0023] The procedure determines at least one potential operating action type for the operable function, depending on the operable function; in particular, several potential operating action types can be determined. That is, after first determining which functions are currently operable, it is also determined which types of operation are possible and in which ways—that is, by means of which operating modalities and / or by means of which operating action types—this operation can be carried out. For example, in a current operating situation and a state of the operable function, certain gestures, voice commands, or other types of operating actions may be usable, while other types of input cannot currently be used for operation. For example, in a certain state of the operable function, vertical scrolling may be performed, but not lateral scrolling, or vice versa.Accordingly, certain commands or operating procedures may or may not be usable in this situation.

[0024] The output generated by the process includes a hint regarding the execution of the potential operating action type. This hint can, for example, indicate a particularly suitable or preferred operating action type, especially a particularly suitable operating gesture. Furthermore, it can provide guidance on how to execute an operating action of a given operating action type, or how to improve the execution of a user-performed operating action, for instance, to recognize the action with greater accuracy and reliability, or to enable operation using a specific operating action type with less physical effort.

[0025] During training, the output is perceptible visually, audibly, and / or tactilely. The output can be achieved in a familiar manner, such as via a display or loudspeaker. Furthermore, a three-dimensional representation can be displayed, for example, using a holographic or autostereoscopic display.

[0026] According to the invention, the output comprises a schematic graphic representation of a hand, wherein the indication of the execution of the potential operating action type is conveyed by means of the representation of the hand. This advantageously allows for a particularly clear indication of the execution of a gesture.

[0027] The output, particularly with an indication of how to execute a gesture in three-dimensional space, can, for example, include a graphic representation of an actuating object, such as a human hand. The representation is primarily schematic, showing, for instance, only the outline of the actuating object. The graphic representation can be displayed on a screen of a display unit, for example, superimposed on another display. Specifically, a user interface can be created on the screen, overlaid with the graphic representation of the actuating action using the actuating object. In addition to a three-dimensional gesture, a touch or gesture on a touch-sensitive surface and / or a voice command can also be provided as potential actuating actions.In particular, a schematic hand representation is generated in such a way that it has the size of an actual hand and shows a projection of the arrangement of a hand relative to the output surface during a spatial gesture.

[0028] During further training, at least two potential operating action types for the operable function are determined, and one of these potential operating action types is designated as the preferred operating action type. In particular, a preferred operating modality can be determined alternatively or additionally. This advantageously signals to the user which type of operating action is currently best suited for operation.

[0029] For example, a function can be operated using various types of actions or modalities, such as voice input, three-dimensional gestures, and gesture control on a touchscreen. However, depending on the current situation and / or the current state of the function, it may be determined that a particular type of action or modality is preferred, such as voice input or gesture control. This might be because the user is using their hands for something else, because a control element or detection area is more or less accessible for executing a gesture, because background noise interferes with voice input, or due to the user's personal preferences.

[0030] In further training, a user's position relative to an input device is recorded, and the preferred operating action type, depending on the user's position, comprises either an actuation of the input device or a spatial gesture. This advantageously allows preferred operating action types and / or operating modalities to be determined particularly easily.

[0031] For example, a user's seating position in an interior space, such as a vehicle, can be detected. If the seat is positioned far back in the interior, i.e., at a considerable distance from an input device such as a touchscreen, the preferred operating action may involve a spatial gesture, while operation by activating or touching the touchscreen or other control element is not preferred. The user's reach can also be detected, such as arm's length, or a predefined reach value can be assumed, whereby activating or touching a touchscreen or control element is only the preferred operating action if it is within the user's reach.

[0032] Furthermore, the user's orientation within the interior, particularly relative to a specific preferred direction, such as a direction of travel or a forward or backward direction in the longitudinal direction of a vehicle, can be detected and taken into account when determining the preferred operating action type. For example, a user's line of sight or the orientation of a seat can be determined. Depending on the user's orientation, a preferred operating action type can then be determined. For example, this can be used to determine whether a control element is positioned in an ergonomically accessible direction for the user, such as in front of the user rather than behind them.

[0033] The output can also depend on the user's orientation within the interior space, in particular a visually perceptible output via an output device in the user's field of vision.

[0034] The process also includes a user history for the potential user action type, based on which an output is generated.

[0035] In further development of the procedure, the operating history for the potential operating action type includes information about the frequency of use and / or the execution quality of the potential operating action type. This advantageously allows for a particularly good evaluation of the specific use of a particular operating action type.

[0036] The operating history includes, for example, stored data about the use of operating actions of a specific type, possibly in connection with an operable function, particularly regarding the frequency of use if the operating action type is preferred, and / or a certain degree of recognition reliability during execution. The operating history includes, for example, information about how often an input was made using an operating action of a specific type for an operable function. It also includes, for example, how frequently an operating action type is used if that operating action type has been designated as the preferred type.

[0037] During further training, a user identification is recorded, and the operating history is captured for the potential operating action type for the identified user. This can advantageously improve the use of a specific operating action type for that user.

[0038] User identification is captured in a known manner. For example, a personal item such as a mobile device, a vehicle key, an identification card or other device, user input, or an optical user identification method can be used.

[0039] The user history is generated specifically for an identified user, so that the user's skills and / or habits can be advantageously assessed. For example, the user history can determine whether and / or how often a user performs actions of a specific type when available for input or to call up a function, and how often they use an action of a different type instead, particularly a less preferred action type. The user history can be stored, for example, on a mobile user device or retrieved using user identification, such as by accessing an external storage unit like a server or an internal storage unit, such as in a vehicle.

[0040] Furthermore, it is possible to determine how a user performs an operation; for example, the certainty with which an operation of a certain type is recognized can be determined, and / or the execution quality for an operation can be determined. For example, in the case of a gesture, it can be determined with what certainty or confidence the gesture can be assigned to a specific type of operation, particularly to distinguish the operation from other types of operations. Furthermore, an execution parameter can be determined, such as clarity, speed, or execution space.

[0041] If the operating action type includes a gesture performed in space, it's possible to determine, for example, the detection area in space where the gesture was performed, how pronounced a movement or pose of an object is, the amplitude or speed of the movement, or the duration for which a specific pose is held. Furthermore, various voice inputs can be available as operating action types. The operating history can then include how clearly a voice input was recognized and / or the input quality of the voice input, for example, depending on volume, direction of speech, speaking speed, emphasis, or speech articulation. The output can then provide the user with guidance on how to better execute the potential operating action type.

[0042] During training, each user action is recorded and assigned to the corresponding action type. The user history for that action type is then updated based on the recorded action. The user history can also be updated if it is detected that a user does not perform any actions of a potential action type, particularly if it is a preferred action type. This ensures that the user history is always up-to-date and that the output can be tailored to the user's needs.

[0043] In this case, the user history can be supplemented with additional information, or existing information can be updated. For example, the frequency with which a user performs a particular type of operation can be updated. Furthermore, information relating to the execution quality of the user actions can be added or updated; for example, the quality and / or clarity with which a user performs a specific type of operation may improve, and this can be recorded and stored in the user history.

[0044] In training, the user history encompasses a predefined period or a predefined number of user actions, or for a specific function. The user history can also cover a variable period, with the process first determining the scope of the history to be considered. For example, a shorter period might be considered when a user is learning the system for the first time, as significant changes in user behavior are to be expected at this stage. Conversely, after a longer period during which the user has become familiar with the system, a longer period of the user history can be considered to generate the output based on a larger data set and to provide the user with more targeted support.

[0045] Furthermore, it can be detected that the user performs an action that cannot be assigned to any potential action type, for example, because the user is using an action type that is currently unavailable, or because the action cannot be assigned to any known action type. For instance, the user performs a swipe gesture even though no corresponding input is currently possible, or they perform a gesture unclearly, for example, too small, too quickly, or inaccurately. It can then be recognized that the user intends to input data but apparently does not know which action types are available and / or how to correctly execute potential actions.

[0046] During further training, an operating action is recorded that cannot be assigned to any potential operating action type, and an intention hypothesis is determined based on the operable function and the recorded operating action. The output with the corresponding message is generated based on this intention hypothesis. This allows a more suitable operating option to be suggested to the user. In particular, it identifies which operating action type the user likely intended to use.

[0047] When determining the intention hypothesis, probabilities for different potential user action types can be determined and compared, and the action type with the highest probability can be used as the intention hypothesis. These probabilities can be determined based on similarity to the actually recorded user action; in this case, for example, an action type can be identified even if the action was not performed clearly enough. Furthermore, the user history can be used to determine an action type that the user employs most frequently for the function in question. In combination, weighting can also be used to determine both which action types are most frequently used for the function according to the user history and which of these action types the actually recorded action is most similar to.Furthermore, it is possible to consider which potential operating action types are available and to choose the most probable of these operating action types as the intention hypothesis. In particular, the user can be asked to confirm that the intention hypothesis is indeed the intended operating action type.

[0048] During training, following the output with the instruction, an operating action of the potential operating action type is recorded, and a learning process is carried out based on the recorded operating action to recognize the potential operating action type. In particular, a control signal for operating the function of the operating system is generated depending on the recorded operating action. This advantageously allows the recognition of operating actions of the operating action type to be improved if necessary.

[0049] For example, the user history might reveal that actions of a specific type are being recorded with low reliability or certainty, particularly for a particular user. This could be due to a user-specific way of executing a gesture or pronouncing a voice command. A learning process can then be implemented, perhaps using established machine learning methods, to improve the recognition of these actions. In this case, the system not only encourages users to employ a potential action type or to perform corresponding actions more effectively, but also improves the recognition of the actually recorded actions.

[0050] During training, a selection / operating action is recorded, and based on this action, a corresponding control device is identified. The control function is then determined based on the identified control device. Specifically, the selection / operating action is designed as a pointing gesture. This allows the control function to be selected particularly easily and precisely.

[0051] For example, a user can point an object, particularly a finger, at a feature in the vehicle, especially using a specific pointing gesture, and this feature is then selected for operation. Functions of the feature that can be operated in the current situation are identified as operable functions. For example, pointing at a sliding door can select it, and a sliding or swiping gesture can be defined as a potential operating action type, allowing the door to be opened or closed. Similarly, windows, flaps, lighting, and / or seat adjustments can be controlled.

[0052] The operating system according to the invention comprises a control unit and an output unit coupled to the control unit. The control unit is configured to detect an operable function of the operating system, to determine at least one potential operating action type for the operable function depending on the operable function, and to record an operating history for the potential operating action type of the operable function. Furthermore, the control unit is configured to control the output unit in such a way that an output is produced which includes an indication of the execution of the potential operating action type.

[0053] The operating system according to the invention is particularly designed to implement the method described above according to the invention. The operating system thus has the same advantages as the method according to the invention for its operation.

[0054] The output unit can be configured in various ways. For example, it can include a display surface on which a graphic representation can be shown during output. It can also include a device for outputting acoustically perceptible output, in particular speech output. Furthermore, the output unit can be capable of generating an output with a three-dimensional representation, such as a holographic or autostereoscopic output.

[0055] One embodiment of the operating system according to the invention comprises an identification device configured to capture a user's identification. The control unit is further configured to record the operating history for the potential operating action type for the identified user. This advantageously allows the output to be tailored particularly precisely to the needs of the respective user.

[0056] In a further development, the operating system includes a detection unit for recording an operating action, in particular with a camera for capturing a gesture within a detection area, a touch-sensitive surface, and / or a speech recognition unit. Specifically, the operating system includes detection units for multiple operating modalities. This advantageously allows different operating modalities to be used to record operating actions.

[0057] During further training, the control unit is configured to perform a learning process based on the recorded operating action to recognize the potential operating action type. This advantageously improves the recognition of operating actions of a specific operating action type. The learning process is carried out, in particular, using known machine learning methods.

[0058] The vehicle according to the invention comprises an operating system as described in the present description.

[0059] The invention will now be explained using exemplary embodiments with reference to the drawings. Figure 1 shows a vehicle with an embodiment of the operating system according to the invention, Figure 2 shows an embodiment of the method according to the invention and Figure 3 shows an embodiment of an output generated by the method according to the invention.

[0060] With reference to Figure 1 A vehicle is described with an embodiment of the operating system according to the invention.

[0061] The vehicle 1 comprises a touchscreen 2, which is designed in a manner known per se and includes a display unit 3 and a touch-sensitive surface 4 arranged above it. The vehicle 1 further comprises a detection unit 5, which is designed as a camera 5 and by which gestures can be detected in a detection area 7 in the interior of the vehicle 1, and an output unit 6 designed as a loudspeaker 6. The touchscreen 2, the camera 5 and the loudspeaker 6 are coupled to a control unit 8, to which a multimedia unit 9 is also coupled.

[0062] With reference to the Figures 2 and 3 An embodiment of the method according to the invention and an output generated thereby are explained. This refers to the above-mentioned method. Figure 1 The illustrated embodiment of the operating system according to the invention is taken as a starting point, which is further specified by the description of the method.

[0063] In step S1, the currently available functions are determined. This involves identifying which devices can currently be controlled by the control unit 8 and are in a state where user input can be processed. In this embodiment, functions of the multimedia unit 9 are available, such as those for outputting audiovisual content, text, and images, controlling a navigation device, accessing other vehicle 1 devices, or controlling a telecommunications device, like a telephone. A user can also navigate through hierarchically organized data, such as music tracks or address data, or menu structures. In other embodiments, alternatively or additionally, other functions and / or devices of the vehicle 1 may be available.

[0064] In this step S1, the user's identity is also recorded in the exemplary embodiment. In this embodiment, this is done using a mobile device assigned to the user, in particular a vehicle key. In other embodiments, identification can be carried out using a mobile phone whose position in the area of ​​a specific vehicle seat, such as the driver's seat, is determined within the vehicle 1 and which is assigned to a specific user. Identification can also be carried out using user input, for example, by logging in via the touchscreen 2.

[0065] Depending on the operable functions, potential operating action types are determined in a further step S2; that is, it is determined which types of operating actions a user can perform to make specific inputs for operating the operable functions. In the exemplary embodiment, various operating modalities are first determined, such as three-dimensional gestures in the detection area 7, gestures on the touch-sensitive surface 4, voice commands, and / or actuations of physical operating elements such as pushbuttons, switches, rotary switches, sliders, rotary controls, or buttons.

[0066] In a further step S3, preferred operating action types are determined, whereby in the exemplary embodiment, the operating action types of a specific preferred operating action type of a preferred operating modality are determined as preferred. That is, it is determined which operating action types are particularly well suited in the current situation or for specific operable functions.

[0067] According to the invention, the position of a vehicle seat on which a user is located is determined. For this purpose, it can first be determined which vehicle seat the user is actually located on, for example by means of a seat mat or a detection device. Furthermore, a specific vehicle seat can be assigned to the operating system or a specific operable device 9 of the vehicle 1. In the exemplary embodiment, it is assumed that the driver of the vehicle 1 is considered the user and is located in the driver's seat. Different operating modalities can be preferably assigned to certain positions of the vehicle seat.For example, if the driver's seat is in a rearward position, such as during automatic driving, and the controls on the dashboard of the vehicle 1 are far away, then operating modalities that do not require the operation of controls located on the dashboard may be preferred; for example, in this case, control by means of voice commands or three-dimensional gestures may be preferred.

[0068] In a further embodiment, it can also be determined which types of user actions require minimal physical effort or allow for the fastest possible operation for specific inputs. For example, scrolling through a list may require multiple presses of a physical or virtually represented button, while analog operation using a swipe gesture along the touch-sensitive surface 4 or within the detection area 7 can be performed much faster and with less effort. In another example, jumping to a specific entry in a list using a voice command may be particularly fast, while entering the entry using a keyboard requires multiple presses.

[0069] In further embodiments, additional parameters can be used to determine preferred operating action types or operating modalities, such as a manual or at least partially automatic driving mode, a type of road traveled, a geographical environment, or settings, inputs, and preferences for the identified user or driver of vehicle 1.

[0070] In a further step, S4, an operating history is recorded for the identified user. In this example, the operating history includes information about which types of operating actions the user has used in the past to input data for specific operable functions. It also includes information about how the user performed these actions, for example, how well the inputs were recognized based on the operating actions of specific types and to what extent a change in the execution of the operating actions would improve recognition. In other words, the operating history includes information about whether the user has used specific types of operating actions in the past to make certain inputs and how they performed these actions.

[0071] For example, in the past, the user performed certain types of actions inaccurately, such as swiping in three-dimensional space too quickly, over too short a distance, and / or not in a sufficiently straight line. This resulted in low confidence in swipe gesture recognition, with, for instance, the amplitude or direction of the swipe being determined with only limited accuracy. In another example, the user uttered a voice command unclearly, too quietly, or too quickly. In yet another example, when activating a virtual control on touchscreen 2, the user touched an area at the edge of the control, making it difficult to clearly distinguish the input from the activation of an adjacent control.

[0072] In this embodiment, the user history is recorded in such a way that it includes a predefined number of user inputs. In other embodiments, a specific time period can be predefined, or the number of past user actions to be considered can be variably determined during the execution of the procedure.

[0073] If, based on the recorded operating history, it is determined that the user normally uses a preferred operating action type for operation and performs the corresponding operating action in a clearly recognizable manner, then in a further step S5 an operating action is recorded, a corresponding control signal is generated and the device of a vehicle 1 is controlled.

[0074] If, on the other hand, it is determined that the user has rarely or never used the preferred operating action type, or that the user does not optimally execute the operating action of the preferred operating action type, then in step S6 an output is generated and displayed with a reference to the execution of the potential operating action type.

[0075] The user should be made aware that they are not performing potential gestures at all, are performing them incorrectly, or that they could perform the operation with less physical effort. The output is now generated individually in the respective situation to indicate a user action type that is not being performed, is being performed incorrectly, or is not being performed optimally, such as a specific gesture.

[0076] In this example, if the user does not use certain gestures at specific points in the operation of a human-machine interface, a schematic hand is displayed at those points, representing the execution of the unused gesture type. If a gesture of a particular type is executed incorrectly or suboptimally, the intended gesture is recognized, and a schematic hand is again displayed, indicating the correct or more efficient gesture execution in motion.

[0077] In other examples, the correct gesture execution is displayed differently. For instance, holographic or autostereoscopic displays can show a three-dimensional hand in space in front of a screen or within another detection area, illustrating the correct gesture movement. A possible extension of the method involves supplementing the visual display with speech output. For example, a voice prompt can be provided for gestures that are never performed in a particular operating situation but which the user executes correctly and is familiar with elsewhere; such as, "You can also perform a swipe gesture here." If a gesture is executed incorrectly or suboptimally, a voice prompt for execution and improvement can be provided; for example, "You need to hold the gesture very briefly at the end" or "You don't need to move your arm so far."This allows for a more targeted approach to the problem of executing a specific type of operating action and the imitation of a natural interaction, such as with a natural counterpart.

[0078] Furthermore, when the user's identity is captured, either via a key or biometric characteristics, the system can address the gesture control and usage patterns of different users individually. For example, the system can recognize, based on the user's operating history, that they do not perform a particular gesture in certain situations, even upon entering the vehicle. It can also identify that the user has previously experienced difficulties executing a specific gesture correctly or with minimal physical effort. Individualized support for optimal gesture control can then be provided to the user.

[0079] Figure 3Figure 1 shows an example of output with a hint regarding the execution of the potential user action type. In the example shown, the preferred user action type is a swipe gesture in three-dimensional space, which allows various graphical objects 22 to 25 to be moved within a user interface 20, in particular to bring an element to the foreground, select it, and then operate it further. The hint regarding such a swipe gesture is represented by a schematic outline of a hand 21. The output is dynamically generated; that is, the schematically represented hand 21 is displayed moving in the manner in which the swipe gesture is to be executed in three-dimensional space.However, in the user interface 20 of the exemplary embodiment, operation in other ways is also possible using non-preferred operating actions, such as by pressing arrow keys, by a swiping gesture along the touch-sensitive surface 4, by a voice command or by operating a physical control element, such as a rotary push-button controller.

[0080] In this embodiment, the output is further tailored to how the user can improve their execution of preferred operating actions; that is, those aspects of the execution where improvement is needed are highlighted. For example, the hand position during the gesture can be highlighted, as can the speed or amplitude of the movement, perhaps over a particularly long distance.

[0081] In another example not according to the invention, the indication of the execution of the potential operating action type is provided by issuing a written instruction. In further embodiments, the output can be provided by means of a holographic or autostereoscopic display, particularly to represent three-dimensional gestures. In further embodiments, the output can alternatively or additionally be audibly perceptible, for example by a spoken instruction on the execution of the preferred operating action type. In particular, the output can be provided by means of an output modality that indicates a preferred operating modality, for example by issuing an audibly perceptible instruction on operation by means of voice commands or by highlighting an operable physical or virtually represented control element, for example by brightness, color, or a dynamic effect such as flashing.

[0082] In a further embodiment, after output, a user action can be recorded as input in step S5 and a corresponding control signal generated to operate the device 9 of the vehicle 1. Operation can also be performed by means of an action of a non-preferred type.

[0083] The operating history is updated based on the recorded operating action.

[0084] In a further embodiment of the method according to the invention, a user action is detected, such as a gesture in the detection area 7 or a voice command, but the action is not assigned to any currently available operation of a device 9 of the vehicle 1. This may be because the action was not correctly recognized and assigned to the wrong type of action, or because the user performed an action of an unavailable type, for example, because they are unaware of which types of actions are currently available or how to execute them. An intention hypothesis is determined; that is, the system forms a hypothesis about which type of action was probably intended by the user. The output is then generated with a reference to the type of action intended according to the intention hypothesis.

[0085] For example, the intention hypothesis can be determined by identifying the type of user action that most closely resembles the recorded action. This also takes into account which potential types of user actions are actually currently available. Furthermore, the user history can be used to identify those types of user actions that the user prefers, or it can be determined whether the recorded action shows particular similarities to a previously recorded action in which the user did not optimally execute a specific type of action.

[0086] For example, a voice command can be recorded as an operating action if it has a phonetic or semantic similarity to a potential operating action type, which is then determined as the intention hypothesis. In another example, a three-dimensional gesture with a currently unintended direction, such as a vertical direction instead of a horizontal direction, can be recorded as an operating action, with an operating action type with a horizontal direction then being determined as the intention hypothesis.

[0087] In a further embodiment, a learning process is performed based on the operating history and / or a recorded operating action, thereby improving the system's recognition of the operating action type. Known methods, such as machine learning, can be applied for this purpose. In this way, the method identifies both a learning need for the user, enabling them to optimally apply a preferred operating action type, and a learning need for the system, improving its recognition of a specific operating action type for the user.

[0088] In another embodiment, a pointing gesture by the user is detected in the detection area 7, and a direction is determined based on this gesture. The system determines which device 9 of the vehicle 1 the user is pointing at and selects that device 9. Any subsequent operating action detected is treated as operation of the selected device 9; that is, a control signal for the selected device 9 is generated based on this action. For example, the user points to a sliding door of the vehicle 1 and initiates automatic opening or closing of the sliding door by means of a sliding or swiping gesture in a specific direction. In another example, the user points to a display of the vehicle 1 on which image data or user interfaces can be output, and any subsequent operating action is treated as operation of the output or the user interface.In this way, the number of available operating mode types can be limited to relevant operating options for a specific facility 9. Reference symbol list

[0089] 1 Vehicle 2 Touchscreen 3 Output unit; Display unit 4 Detection unit; Touch-sensitive surface 5 Detection unit; Camera 6 Output unit; Speaker 7 Detection area 8 Control unit 9 Device; Multimedia unit 20 User interface 21 Schematic representation of a hand 22 to 25 Graphic object S1 to S6 Step

Claims

1. Method for operating an operating system, in which an operable function of the operating system is detected; at least one potential operating action type is determined for the operable function depending on the operable function, wherein one of the potential operating action types is determined as a preferred operating action type; an operating history is acquired for the preferred operating action type of the operable function, wherein the operating history for the preferred operating action type includes information about whether a user has in the past used operating actions of the preferred operating action type to make certain inputs, and about how the user executed these operating actions; and an output is generated and output on the basis of the operating history, wherein the output includes a suggestion for the execution of the preferred operating action type, wherein the operating actions of the user are further detected as an input and assigned to the operating action types, and the operating history is updated on the basis of the detected operating actions, characterized in that the preferred operating action type is determined on the basis of the position of a vehicle seat on which the user is located and the output comprises a schematic graphic representation of a hand (21), wherein the suggestion for the execution of the preferred operating action type is output by means of the representation of the hand (21), wherein the preferred operating action type comprises a gesture in three-dimensional space or on a touch-sensitive surface, wherein, if it is determined, on the basis of the acquired operating history, that the user has rarely or never used the preferred operating action type, the representation of the hand (21) represents the execution of this gesture type, or if it is determined, on the basis of the acquired operating history, that the user has not optimally executed the operating action of the preferred operating action type, the gesture of the operating action type is recognized and a schematic hand (21) that shows the correct or more economical execution of the gesture in motion is displayed.

2. Method according to claim 1, characterized in that at least two potential operating action types for the operable function are determined; and one of the potential operating action types is determined as the preferred operating action type.

3. Method according to either of the preceding claims, characterized in that the operating history for the preferred operating action type includes information about a frequency of use and / or a quality of execution of the preferred operating action type.

4. Method according to any of the preceding claims, characterized in that a user identification of the user is detected; and the operating history for the preferred operating action type is acquired for the identified user.

5. Method according to any of the preceding claims, characterized in that an operating action is detected which cannot be assigned to any potential operating action type; an intention hypothesis is determined on the basis of the operable function and the detected operating action; and the output with the suggestion is generated on the basis of the intention hypothesis.

6. Method according to any of the preceding claims, characterized in that following the output with the suggestion, an operating action of the preferred operating action type is detected; and a learning process to recognize the preferred operating action type is executed on the basis of the detected operating action.

7. Method according to any of the preceding claims, characterized in that a selection operating action is detected; an operable device is determined on the basis of the detected selection operating action; and the operable function is determined on the basis of the determined operable device.

8. Operating system comprising a control unit (8); and a display unit (3) coupled to the control unit (8); wherein the control unit (8) is designed to detect an operable function of the operating system; to determine at least one potential operating action type for the operable function depending on the operable function, wherein one of the potential operating action types is determined as the preferred operating action type; and to acquire an operating history for the preferred operating action type of the operable function, wherein the operating history for the preferred operating action type includes information about whether a user has in the past used operating actions of the preferred operating action type to make certain inputs, and about how the user executed these operating actions; wherein the control unit (8) is further designed to control the display unit (3) in such a way that an output is generated and output on the basis of the operating history, which output comprises a suggestion for the execution of the potential operating action type, wherein the operating actions of the user are further detected as an input and assigned to the operating action types, and the operating history is updated on the basis of the detected operating actions, characterized in that the control unit (8) is further designed such that the preferred operating action type is determined on the basis of the position of a vehicle seat on which the user is located, the output comprises a schematic graphic representation of a hand (21), wherein the suggestion for the execution of the preferred operating action type is output by means of the representation of the hand (21), wherein the preferred operating action type comprises a gesture in three-dimensional space or on a touch-sensitive surface, wherein, if it is determined, on the basis of the acquired operating history, that the user has rarely or never used the preferred operating action type, the representation of the hand (21) represents the execution of this gesture type, or if it is determined, on the basis of the acquired operating history, that the user has not optimally executed the operating action of the preferred operating action type, the gesture of the operating action type is recognized and a schematic hand (21) that shows the correct or more economical execution of the gesture in motion is displayed.

9. Operating system according to claim 8, characterized by an identification apparatus designed to detect a user identification of the user; wherein the control unit (8) is further designed to acquire the operating history for the preferred operating action type for the identified user.

10. Operating system according to claim 8 or claim 9, characterized by a detection unit for detecting the operating action, in particular having a camera (5) for detecting a gesture in a detection region (7) and / or having a touch-sensitive surface (4) and in particular additionally having a speech detection unit.

11. Operating system according to claim 11, characterized in that the control unit (8) is designed to execute a learning process to recognize the potential operating action type on the basis of the detected operating action.

12. Vehicle (1) having an operating system according to any of claims 8 to 11.

Images