System and procedure for processing fuzzy user input

Abstract

Method (200) for processing inputs from a user (60) of a vehicle (80), the method comprising: Capturing (202) a first input from the user (60), wherein the first input has a first modality and is explicitly taken as an active input from the user (60); Capturing (204) a second input from the user (60), wherein the second input has a second modality that differs from the first modality, wherein the second input is implicitly a passive input from the user (60); Determine (208) a combined user input (60) based on the first input and the second input; and Triggering (210) a function based on the combined input further encompassing: Determine (206) a context that describes a use of the vehicle (80) by the user (60); wherein Determining (208) the combined input of the user (60) is further based on the context; where the context describes one or more of the following: a current or previous driving situation of the vehicle (80); a use of the vehicle (80); a use of a component in the vehicle (80), preferably a navigation system, an infotainment system, or a communication device; a condition of the user (60) and / or of one or more occupants in the vehicle (80); a condition of the vehicle (80); one or more objects in the vicinity of the vehicle (80), preferably including one or more road users, one or more structural features of a road travelled by the vehicle, and / or weather data; and general parameters including date and / or time.

Description

[0001] The disclosure relates to systems and methods for processing user input. In particular, the disclosure relates to systems and methods for processing fuzzy, multimodal user input in vehicles. State of the art

[0002] Systems with user interfaces that process user input are known in the prior art. Classic, deterministic input methods are known, such as the operation of switches, buttons, and the like (also in the form of touchscreens), or the selection of individual predefined menu items in a graphical user interface.

[0003] User input typically needs to conform to a predetermined format to be uniquely recognized and trigger corresponding functions. This also applies essentially to multimodal and / or more modern input methods, such as speech recognition or gesture control, where predefined voice commands or gestures can be used to trigger specific functions, for example, the command word "navigation" to use the navigation system or a circular finger gesture to increase or decrease the volume of an infotainment system.

[0004] Regardless of the input method, systems and user interfaces typically require explicit, unambiguous, and precise user input or utterances. If an input cannot be assigned (e.g., no clear recognition of a command word or gesture), no function is triggered, thus preventing incorrect operation of even non-driving-related functions in the context of vehicle use.

[0005] Familiar systems and user interfaces typically rely on explicit, unambiguous, and precise user input, meaning that in cases of doubt, if no clear assignment can be made, the system cannot be used. This can significantly impair the user experience if a user interface cannot precisely receive and process input.

[0006] In particular, imprecise user input, which is not unambiguous but based on probabilities, for example, often cannot be captured or processed at all. This especially affects modern input methods, such as voice input or gesture control. A voice input like "Should I turn here?" or a pointing gesture to the right or left cannot currently be processed meaningfully.

[0007] Furthermore, well-known systems and user interfaces are mostly monomodal (or based on a few modes) and thus limited to one or a few specific modalities, such as speech input and output, without the possibility of considering and integrating other modalities (e.g., gestures, facial expressions, gaze, or other forms of expression).

[0008] Therefore, there is a need for methods and systems that can capture and process fuzzy user input.

[0009] There remains a need for methods and systems that improve the accuracy of capturing and processing user input, especially fuzzy user input, by combining input from different modalities.

[0010] Dumas et al. (Multimodal Interfaces: A Survey of Principles, Models and Frameworks. In: Human Machine Interaction: Springer Berlin Heidelberg, 2009. Pages 3-26. - ISBN 978-3-642-00437-7) describe multimodal interactions and in particular describe theoretical principles, time-critical software architectures and challenges of fusion and fission of multimodal data.

[0011] Celikyilmaz et al. (Modeling uncertainty with fuzzy logic. Studies in fuzziness and soft computing, 2009, Vol. 240, No. 1, pp. 149-215. - ISBN 978-3-540-89923-5) describe a modeling of inaccuracies using fuzzy methods or fuzzy logics.

[0012] Exposito et al. (Anomalies Detection by self-observation of Situations within a Connected Autonomous Vehicle, 2018) describes the detection of anomalies through self-monitoring of a vehicle.

[0013] Van Brummelen et al. (Autonomous vehicle perception: The technology of today and tomorrow. Transportation research part C: emerging technologies, 2018, Vol. 89, pp. 384-406. - ISSN: 0968-090X) provides an overview of the state of the art in relation to vehicle self-perception. Disclosure of the invention

[0014] It is the purpose of this disclosure to provide systems and methods for processing user input that avoid one or more of the aforementioned disadvantages and / or enable one or more of the described advantages.

[0015] This problem is solved by the subject matter of the independent claims. Advantageous embodiments are specified in the dependent claims.

[0016] In a first aspect according to embodiments of the present disclosure, a method for processing inputs from a user of a vehicle according to one of claims 1 to 6 is specified.

[0017] In a second aspect, a system for processing inputs from a user of a vehicle according to claim 7 is specified.

[0018] In a third aspect, a vehicle according to claim 8 is specified.

[0019] The methods and systems disclosed herein offer one or more of the advantages listed below.

[0020] Fuzzy user input can be captured and processed more effectively. The accuracy of capturing and processing user input, especially fuzzy input, can be improved by combining input from different modalities.

[0021] This can lead in particular to improved user attention and / or reduced user distraction. The road safety of a user-driven vehicle can thus be improved. Short description of the drawings

[0022] Exemplary embodiments of the disclosure are shown in the figures and are described in more detail below. Unless otherwise noted, the same reference symbols are used for identical and equivalent elements. Fig. Figure 1 shows a schematic representation of a system for processing user input according to embodiments of the present disclosure for use in a vehicle; and Fig. Figure 2 shows a flowchart of a method according to embodiments of the present disclosure. Embodiments of the disclosure

[0023] In the following, unless otherwise stated, the same reference symbols are used for identical and equivalent elements.

[0024] Fig. Figure 1 shows a schematic representation of a system 100 for processing user input according to embodiments of the present disclosure for use in a vehicle 80. The system can essentially be executed on a control unit 120 of the vehicle 80. In addition to the control unit 120, the vehicle 80 further comprises a communication unit 130, which is configured for data communication with components external to the vehicle 80 (e.g., mobile devices 70 and / or a backend 150), and a user interface 110.

[0025] Vehicle 80 also includes a sensor 115 configured to record user behavior. The sensor 115 is in Fig. Figure 1 is merely an example and can include several components, in particular components installed at various locations in or on the vehicle. The sensor system 115 can, for example, include one or more cameras configured to capture the vehicle's interior, especially a user or occupant, and thus record their behavior, particularly gestures, facial expressions, posture, gaze direction, and the like. To capture a user's gestures, further components can be provided, such as sensors that implement gesture control (e.g., pointing, indicating, swiping on the central information display (CID) without having to use a touchscreen). The sensor system 115 can also include one or more cameras that capture the area around the vehicle.Based on this, various contexts can be determined and / or further enriched, for example, by detecting roadways, buildings, other road users, and the like. The sensor system can include existing or special vehicle sensors, such as ultrasonic sensors, radar sensors, lidar sensors, or similar devices, which typically provide information for the vehicle's operation (e.g., lateral and longitudinal guidance, parking assist, stop & go assist). Data from these sensors can also be used to determine and / or enrich various contexts.

[0026] The user interface 110 includes one or more multimodal user interfaces, in particular user interfaces configured for operating the vehicle 80 (e.g., climate control, seat adjustment, infotainment, vehicle settings). The user interface 110 enables the multimodal acquisition of input from a user 60 (not in Fig. 1 shown), for example via a graphical user interface (e.g., touchscreen), via classic vehicle controls 80 (e.g., buttons, switches, iDrive controller), via voice control (e.g., using sensors 115), and the like. The user interface 110 further enables the multimodal output of information to a user 60, for example via graphical display elements (e.g., touchscreen, head-up display, instrument cluster, central information display or CID), via tactile elements (e.g., vibration of the steering wheel or parts of the seat), via voice output through a speaker system present in the vehicle (e.g., infotainment system) or acoustic signal generators (e.g., gong, beeper), and the like.The user interface can implement a graphical user interface based on relevant configuration data, displaying indicators and controls that the user can use to operate the vehicle. Additionally or alternatively, the user interface can include (further) indicators and controls, such as switches, buttons, and displays.

[0027] The control unit 120 can communicate with external components and services via the communication unit 130, for example, to communicate with backend servers and / or services 150. Alternatively or additionally, the control unit can communicate with a user's mobile device 70 via the communication interface 130 and use data and / or sensors provided by the mobile device 70.

[0028] Furthermore, the system 100 can include an external backend component 150 or infrastructure that provides one or more resources (e.g., servers, services). The backend component 150 can communicate with the vehicle's control unit 120 80 temporarily or permanently. Preferably, resource-intensive processing steps (e.g., natural language processing and / or image / video data processing, speech recognition, processing of large amounts of external data) can be outsourced to the external backend component 150, as these steps would be difficult or impossible for the control unit 120 in the vehicle 80 to perform. Consideration can also be given to potential requirements regarding computing power, storage capacity, available bandwidth, connectivity to external data sources, and the like.

[0029] A software component running on the control unit 120 can establish a data connection to one or more services 150. Such a connection can preferably be established directly if a service or platform (or the underlying protocol) is directly supported by the software component. Alternatively or additionally, such a connection can be established via a mobile device 70 connected to the vehicle 80 by the user 60. In this case, the software component can then use all services, platforms, and protocols supported by the mobile device.

[0030] In an illustrative example, user 60 is driving vehicle 80 and following a route provided by a navigation system. Vehicle 80 approaches an intersection where, due to a complex intersection geometry (e.g., with two left and two right turns), the navigation instructions "Turn right" are not clear to user 60. User 60 looks approximately to the right and asks via voice control: "This one?"

[0031] A standard navigation system does not initially provide any prompts, neither monomodal nor multimodal. At best, a user can repeat a previously given navigation instruction by pressing a button. In this case, it must be assumed that the time until the turn is too short for such a repetition and that simply repeating the last navigation instruction will not resolve the user's confusion.

[0032] According to the present disclosure, both the voice input of user 60 via the usual microphone input and the gestures or body posture of user 60 can be captured by means of an interior camera. The system now has at least two inputs of different modalities. The voice input can be analyzed and processed by the system, for example using natural language processing (NLP), whereby the voice input made by user 60 exhibits a relative imprecision. The voice input "This one?" can, for example, be interpreted as a question about an indeterminate or relative proximity (cf. "here") of a junction (cf. "The"), whereby the context of a turning situation is assumed (e.g., by a navigation system). Based on this, a prioritization of the turning options can be carried out, in ascending distance to vehicle 80.Assuming four turning options left and right, this would result in several closely spaced junctions that could be meant (e.g., associated with probabilities of 29%, 27%, 23% and 21% from the nearest left turn to the furthest right turn).

[0033] If two junctions are located at a similar distance, resulting in a similar probability that one or the other is intended, a sufficiently clear result cannot be achieved based solely on voice input. Ultimately, the junction intended by user 60 could be further away (or at the same distance) as an unintended junction on the other side of the road. Therefore, a navigation system response based solely on voice input is unlikely to produce the desired result—namely, that user 60 takes the correct turn.

[0034] In addition to the voice input from user 60, system 100 in this example also receives input in the form of a gesture or body posture from user 60. At the time of the voice input, the system can detect user 60's posture, in particular their head position and / or gaze direction. From the orientation of the head or gaze direction, the system can then narrow down the area relative to user 60 or vehicle 80 to which the user's voice input refers. In this example, user 60 has turned their head and gaze to the right, which system 100 can detect. Here, too, a corresponding probability (e.g., 80% to 20%) is sufficient. Together with user 60's voice input, the number of possible turns can thus be restricted not only based on distance (see above) but also based on the direction "right".Of the four assumed turning options (left and right), only those on the right remain available (e.g., associated with probabilities of 55% and 45% from the nearest to the furthest turn), as the turnings on the left are no longer possible. Therefore, adding another modality to the user input can significantly increase the probability of triggering the correct function (here: turning).

[0035] Additionally, a plausibility check can be performed by, for example, ignoring or processing inputs that conflict with other inputs and / or the context. In the present example, if the user looks left even though the navigation system has indicated a right turn, the navigation system can repeat the announcement and / or notify the user of this, for example, via a voice output and / or display. The same applies to conflicts between different inputs, of which those with the lowest probabilities or plausibility can be ignored.

[0036] In this case, system 100 can consider input based on an additional modality. For example, the system can also detect a gesture from driver 60 (e.g., via camera or gesture control). In addition to the inputs described above via speech (explicit) and body posture (implicit), user 60 can provide further input by pointing or indicating with their hand. A user pointing in the direction of a specific junction can thus further increase the probability that the user input has been correctly detected. In the present example, user 60's gesture of pointing to the right in the direction of one of the two junctions (e.g., detected with probabilities of 70% to 30%) can, together with the speech input and body posture, result in a very high probability (e.g., 90% or 95%) that the user is indicating the correct (or incorrect) junction.Based on this, a confirmation (or correction) can be made, prompting the user to perform the turning maneuver (or to delay until the correct turn).

[0037] The aforementioned confirmation or correction can, in turn, be communicated by System 100 in a multimodal manner, for example, as a voice prompt such as "Yes, please take the first turn" and a visual indication (e.g., in the head-up display and / or on the map in the CID). A correction can also be communicated through other means, such as a vibration in the steering wheel when turning into the wrong direction.

[0038] Fig. Figure 2 shows a flowchart of a method 200 according to embodiments of the present disclosure. The method 200 for processing inputs from a user 60 of a vehicle 80 begins at step 201.

[0039] In step 202, the first input from user 60 is recorded. This first input has a first modality, for example, voice input (see the example described above). The first input can preferably be explicit, as an active input from user 60 (e.g., voice input), as opposed to implicit input (e.g., recording the gestures, facial expressions, etc., of user 60).

[0040] In step 204, a second input from user 60 is recorded. This second input has a different modality than the first, such as body posture or gestures (see the example described above). The second input can preferably be implicit, as a passive input from user 60 (e.g., recording gestures, facial expressions, etc.), as opposed to an explicit input (e.g., speech input).

[0041] Regarding steps 202 and 204, it should be noted that sequential processing is not required. Steps 202 and 204 can be performed in any order or essentially simultaneously or in parallel. For example, input via gesture control can occur essentially in parallel with input via voice control. The two inputs can therefore overlap completely or partially, one input can last longer than the other, etc.

[0042] In a preferred embodiment, a third input from the user 60 is captured in optional step 205. The third (explicit or implicit) input has a third modality that differs from the first and / or second modality. The third input can then be taken into account in determining the combined input (see step 208).

[0043] In a preferred embodiment, in optional step 206, a context is determined that describes the use of the vehicle 80 by the user 60. By way of example, and not exhaustively, a context may include: a current or previous driving situation of the vehicle 80, use of the vehicle 80, use of a component in the vehicle 80, preferably a navigation system, an infotainment system, or a communication device, the state of the user 60 and / or one or more occupants in the vehicle 80, the state of the vehicle 80, one or more objects in the environment around the vehicle 80, preferably including one or more road users, one or more structural features of a road traveled by the vehicle, and / or weather data and general parameters including date and / or time. The context can then be taken into account in determining the combined input (see step 208).

[0044] In a preferred embodiment, in optional step 207, the (explicit and / or implicit) inputs of user 60 are processed using suitable techniques (see the example described above). Fuzzy techniques are particularly suitable here, as they can assign probabilities to fuzzy inputs, enabling the assignment, analysis, assessment, or evaluation of the inputs. The processed inputs can then be considered in determining the combined input (see step 208). In addition to fuzzy logic techniques, machine learning (ML) techniques can also be used. As an alternative or additional step to the multi-stage combination described above, input signals from various sensors can be processed separately or essentially simultaneously by an ML model.In one example, an audio signal can be processed by automatic speech recognition (ASR) and natural language understanding (NLU) to determine the intent (e.g., "ask for turn location") and entity (e.g., "this one"). This may involve anaphora, which needs to be resolved. In other examples, video signals from a camera used for gesture recognition can be processed by a machine learning (ML) model into a vector indicating the pointing direction. This vector, along with a geometric environment model, is then processed to determine a point (e.g., "turn location"). Further examples from the fields of fuzzy logic and machine learning are conceivable, such as early and late sensor fusion.

[0045] In step 208, a combined input from user 60 is determined based on user inputs. This determination can be based on the first and second inputs, preferably with additional inputs based on the context (see step 206) and / or a third input (see step 205), as well as on the processed inputs (see step 207).

[0046] In a preferred embodiment, detecting a combined input (see step 208) further involves detecting a third input (see step 205) using a third modality, wherein a combined input is first detected (e.g., based on the detection of a speech input and a gesture control input), which is then further combined with the input using the third modality (e.g., gaze control). A combination can therefore be performed directly (1st, 2nd, and 3rd modalities in parallel) or in a suitable sequence (e.g., first combining the inputs using the 1st and 2nd modalities, then further combining the combined input with the input using the 3rd modality).

[0047] In step 210, a function is triggered based on the combined input, preferably if the combined input has a predetermined reliability. The procedure ends in step 212.

[0048] Although the invention has been illustrated and explained in detail by means of preferred embodiments, the invention is not limited by the disclosed examples, and other variations may be derived therefrom by those skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that the embodiments mentioned by way of example are truly only examples and should not be construed as limiting the scope, possible applications, or configuration of the invention in any way.Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make various changes, for example with regard to the function or the arrangement of individual elements mentioned in an exemplary embodiment, without leaving the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

Claims

[1] A method (200) for processing inputs from a user (60) of a vehicle (80), the method comprising: detecting (202) a first input of the user (60), wherein the first input has a first modality and is explicitly made as an active input of the user (60); detecting (204) a second input from the user (60), the second input having a second modality different from the first modality, the second input being implicit as a passive input from the user (60); Determining (208) a combined input of the user (60) based on the first input and the second input; and Triggering (210) a function based on the combined input, further comprising: Determining (206) a context that describes a use of the vehicle (80) by the user (60); wherein Determining (208) the combined input of the user (60) further based on the context; where the context describes one or more of the following: a current or previous driving situation of the vehicle (80); use of the vehicle (80); a use of a component in the vehicle (80), preferably a navigation system, an infotainment system, or a communication device; a state of the user (60) and / or one or more passengers in the vehicle (80); a condition of the vehicle (80); one or more objects in the environment around the vehicle (80), preferably including one or more road users, one or more structural features of a road traveled by the vehicle, and / or weather data; and general parameters including date and / or time. [2] Method (200) according to the preceding claim 1, further comprising: detecting (205) a third input from the user (60), wherein the third input has a third modality that is different from the first and / or second modalities; wherein Determining (208) the combined input of the user (60) is further carried out based on the third input. [3] The method (200) of claim 2, wherein the third input is made by the user (60) explicitly or implicitly. [4] Method (200) according to one of the preceding claims 1 to 3, wherein the first, second and optionally third modalities are selected from the group comprising: Voice input; Inputs based on a user’s gesture (60); Inputs based on a posture of the user (60), in particular on a head position of the user (60); Inputs based on a tactile input from the user (60); Inputs based on a user’s gaze guidance and / or gaze direction (60); and Inputs based on the user’s facial expressions (60). [5] Method (200) according to one of the preceding claims 1 to 4, further comprising: Processing (207) the first, second and / or optionally third input by means of fuzzy techniques if the respective first, second and / or optionally third inputs have a degree of uncertainty, preferably wherein the processing (207) provides for an assignment of corresponding probabilities for a plurality of alternatives; wherein Determining (208) the combined input of the user (60) is further based on the processed first, second and / or optionally third inputs. [6] Method (200) according to one of the preceding claims 1 to 5, further comprising: Determining the reliability of the combined input; and Triggering (210) the function based on the combined input if the reliability is greater than or equal to a predetermined value. [7] System (100) for processing inputs of a user (60) of a vehicle (80), the system comprising a control unit (120) configured to carry out the method (200) according to any one of the preceding claims 1 to 6. [8] Vehicle (80) comprising a system (100) according to the preceding claim 7.

Images