Method for the application of a user interface and system

Abstract

A method for utilizing an automated assistant (12) for an application in an electronic computing device (14) in a motor vehicle (16). Comprising: capturing an input from a user to perform an operation

Description

FIELD OF THE INVENTION

[0001] The invention relates to the field of automobiles. More specifically, the present invention relates to a method for the application of a user interface on a display device for a vehicle according to claim 1. Furthermore, the present invention relates to a system, a corresponding computer program product, and a corresponding non-transitory computer-readable storage medium. BACKGROUND INFORMATION

[0002] Users typically interact with devices featuring visual user interface elements. These elements can change over time via an electronic display. When a user interface element is displayed and the user intends to interact with it, they usually initiate some form of motion, such as moving their hand, finger, or mouse towards the interface element.

[0003] In dynamic user interfaces, the position of the element might change, or the function of the element might be altered. If such a change occurs in the time between the user deciding to interact with the element and the user being able to perceive the change, the user ends up performing an action that does not align with their intent.

[0004] For instance, humans have visual reaction times of approximately 300 milliseconds from the moment a visual change occurs to when a corresponding physical movement takes place. This implies that any alteration to the user interface that happens 300 milliseconds prior to a user interface interaction results in an unintended action.

[0005] A prime example of this is seen with cookie banners, which often appear a few seconds after a website is loaded. In attempting to click on an element of the website, one might accidentally click on the appearing banner instead. Another example is a slow or frozen user interface, where the user intends to confirm a click but ends up inadvertently clicking an element on the next screen because the screen updates at that moment.

[0006] The US20220236857A1 discloses implementations related to an automated assistant hat can perform operations to revert various applications to prior states that the applications may have arrived at via certain user inputs. The user can provide a spoken utterance such as, “undo,” in order to cause the automated assistant to identify a particular application that the user may want to affect with the “undo” command. Once the specific application is identified, the automated assistant can identify one or more operations recently performed using that particular application.

[0007] Thus, the assistant is used to perform an interaction based on a signal from the user and to undo the operations.

[0008] The objective of the invention is to provide a method through which an application of a user interface may be used particularly easy and intuitive.

[0009] This objective is achieved by means of a method with the features of claim 1 and by means of a system according to the invention. Advantageous embodiments and further developments can be inferred from the dependent claims and the description. SUMMARY OF THE INVENTION

[0010] An aspect of the inventions relates to a method for utilizing an automated assistant for an application in an electronic computing device in a motor vehicle, implemented by a system in the vehicle, comprising the following steps:

[0011] Capturing an input (voice interaction, commands, manual input on user interface UI) from a user to perform an operation by the application, using a capturing device such as a driver assistance system with microphone and speaker for intercommunication. Additionally, however, a display device with a user interface may also be used, which facilitates interactive communication with a user.The capturing device is specifically implemented to capture a command and forward it to the electronic computing device for further processing.

[0012] Determining by the automated assistant that the input is intended to undo a prior operation in the application, previously captured by the automated assistant.The assistant distinguishes between a command to execute an operation and a command that considers the last operation as misleading, thus necessitating its undoing. This type of command is therefore captured for further processing.

[0013] Identifying at least one such misleading operation from a set of previous operations depending on the input, which may have influenced the current state of the application. In this step, it is intended to identify from the set of previous operations the one that the user likely considers as incorrect.

[0014] Undoing at least one identified misleading operation. This means that after selecting one or more misleading operations, the assistant is designed to undo them to revert to a previous state. This allows the user to reissue the commands to perform the desired operation.

[0015] To solve the objective of the invention, it is envisioned that the application recognizes which misleading operation the user has undone. Data of the misleading operation and data leading to the undoing of the misleading operation are stored. The automated assistant is then trained with this data to avoid future repeat errors regarding at least this one identified misleading operation. Therefore this Inputs for the training are selected, and at least one improvement is made, allowing better later interactions with this command for this operation. Algorithms or Al can be used for this purpose, with a predictive model being preferred. This configuration of the method aims to enable particularly intuitive communication, allowing for increased driving safety and faster command inputs, for example.

[0016] Such a predictive model is typically trained on a dataset containing user interface transitions and human interactions that undo the immediate preceding action. It is now proposed to use data indicating or containing information that a human action was performed as an interaction with the application and immediately (within a predefined time interval) undone, as an indication that the interaction was unintentional.

[0017] For example, in a browser setting, the unintended action could be accidentally clicking on a link due to a time delay in loading the page. (For instance, an element appears that the user intends to click on, but in the meantime, a pop-up appears on the page, and the user inadvertently clicks on it in the area where the element was previously displayed. Now the user needs to undo the action.) The undone action would be returning to the previous page by clicking on the back button. With this combination of data, a predictor can be trained to forecast how long ago a user interface element should have been presented when a user interaction occurs before it changes

[0018] Alternatively, a scenario where a user is skimming through information and clicks on a "Next" button, but perhaps clicks too quickly, may be considered. The too-short time interval between the first "Next" click and the excessively quick "Next" click can be compared, for example, to the timing of other consecutive "Next" clicks, and viewed as a deviation that can be used for training. However, this "too short" time interval could also be compared with the content of the information on the page and set accordingly, so that a relationship between the time intervals between "Next" clicks and the content on the page can be trained afterward.

[0019] If it is probable based on the interaction pattern that the user intended a different action and mistakenly clicked due to the change in the user interface, the action could be suppressed, or an easily accessible undo option could be offered to the user. This is particularly useful for interfaces in vehicles, where mis-clicks may occur more frequently, and navigating UI menus can be distracting and therefore dangerous. Additionally, special error patterns can be integrated with this method. For example, if a person typically interacts with the interface from a certain position relative to the display, secondary issues such as a hand blocking the view of a UI element may arise, making it impossible for the user to see the UI change.

[0020] In another example where interactions with the assistant are performed via voice, initiating a request for street navigation may be considered. However, the user forgets the address momentarily and hesitates. The assistant's voice control would immediately start responding by providing a prompt to ask questions. Then, the user simultaneously recalls the address, and the assistant does not understand it. The user would then have to cancel and restart the command from the beginning. However, with the new method, by learning the user's delay, the assistant would not respond immediately but would instead introduce a time interval to give the user time to remember and provide the address.

[0021] Predicting whether a user interaction was unintentional based on the timing of assistant or UI changes, which generalizes to all types of interfaces (voice, hand, finger, mouse, etc.), is particularly useful for vehicle interfaces and can be extended to secondary issues.

[0022] In an advantageous embodiment of the invention, it is provided that the identified misleading operations are communicated to the user and presented for selection. Thus, a list of possible options is presented to the user to make an accurate selection, which can be displayed on the display device or spoken aloud, for example.

[0023] In another advantageous embodiment of the invention, it is provided that the selection is based on the frequency with which the undoing of the misleading operation has already been performed, with the most frequent one being presented first. The list is constructed accordingly, and created based on frequency, allowing for efficient identification of the misleading operation.

[0024] In another advantageous embodiment of the invention, it is provided that data leading to the misleading of the operation (e.g., too short time interval) is also taken into account for other training the assistant for other operations. This means that known errors such as a too short time interval for reactions will be utilized for other operations as well, not just for the one misleading operation.

[0025] In another advantageous embodiment of the invention, it is provided that the assistant is trained with this data depending on different users. Since users can always vary, the training data and all settings, as well as any modified settings, are adjusted according to the user, ensuring that not all users have to work with the same settings. This allows for a flexible configuration of settings and the use of the assistant.

[0026] Further advantages, features, and details of the invention derive from the following description of preferred embodiments as well as from the drawing. The features and feature combinations previously mentioned in the description as well as the features and feature combinations mentioned in the following description of the figure and / or shown in the figure alone can be employed not only in the respectively indicated combination but also in any other combination or taken alone without leaving the scope of the invention. BRIEF DESCRIPTION OF THE DRAWING

[0027] The novel features and characteristic of the disclosure are set forth in the appended claims. The accompanying drawings, which are incorporated in and constitute a part of this disclosure, illustrate exemplary embodiments and together with the description, serve to explain the disclosed principles. In the figures, the left-most digit(s) of a reference number identifies the figure in which the reference number first appears. The same numbers are used throughout the figures to reference like features and components. Some embodiments of system and / or methods in accordance with embodiments of the present subject matter are now described below, by way of example only, and with reference to the accompanying figures.

[0028] The drawing shows in:

[0029] Fig. 1 a possible implementation of a system to execute a method for utilizing an automated assistant for an application in an electronic computing device in a motor vehicle. DETAILED DESCRIPTION

[0030] In the present document, the word "exemplary" is used herein to mean "serving as an example, instance, or illustration". Any embodiment or implementation of the present subject matter described herein as "exemplary" is not necessarily to be construed as preferred or advantageous over other embodiments.

[0031] While the disclosure is susceptible to various modifications and alternative forms, specific embodiments thereof have been shown by way of example in the drawing and will be described in detail below. It should be understood, however, that it is not intended to limit the disclosure to the particular forms disclosed, but on the contrary, the disclosure is to cover all modifications, equivalents, and alternatives falling within the scope of the disclosure.

[0032] The terms “comprises”, “comprising”, or any other variations thereof, are intended to cover a non-exclusive inclusion so that a setup, device or method that comprises a list of components or steps does not include only those components or steps but may include other components or steps not expressly listed or inherent to such setup or device or method. In other words, one or more elements in a system or apparatus preceded by “comprises” or “comprise” does not or do not, without more constraints, preclude the existence of other elements or additional elements in the system or method.

[0033] In the following detailed description of the embodiment of the disclosure, reference is made to the accompanying drawing that forms part hereof, and in which is shown by way of illustration a specific embodiment in which the disclosure may be practiced. This embodiment is described in sufficient detail to enable those skilled in the art to practice the disclosure, and it is to be understood that other embodiments may be utilized and that changes may be made without departing from the scope of the present disclosure. The following description is, therefore, not to be taken in a limiting sense.

[0034] Fig.1 shows a possible implementation of a system 10 to execute a method for utilizing an automated assistant 12 for an application in an electronic computing device 14 in a motor vehicle 16.

[0035] It comprises the following components: a driver assistance system 18 with a microphone 20 and speaker 22 for intercommunication and / or a display device 24 with a user interface 26 as detection devices 28 for user inputs of a driver / user 30, such as voice interactions, commands, and manual inputs on the driver / user 30 interface 26. Furthermore, the system 10 includes the automated assistant 12 for processing user inputs and performing operations in the application. The driver assistance system 18 facilitates intercommunication between the driver / user 30 and the system 10. The display device 24 represents the user interface 26 through which the driver / user 30 interacts with the system 10, alternatively through verbal communication via the microphone 20 and speaker 22. The detection devices 28 capture user inputs, while the automated assistant 14 processes the incoming inputs and executes operations in the application

[0036] The following steps are carried out using the system 10:

[0037] Capturing an input (voice interaction, commands, manual input on UI) from the driver / user 30 to perform an operation by the application by the detection devices 28.

[0038] Determining by the automated assistant 12 that the input is intended to undo a prior operation in the application, previously captured by the automated assistant 14.

[0039] Identifying at least one such misleading operation from a set of previous operations depending on the input, which may have influenced the current state of the application.

[0040] Undoing at least one identified misleading operation.

[0041] The application recognizes which misleading operation the user has undone, and data of the misleading operation and data leading to the undoing of the misleading operation are stored, and the automated assistant 12 is trained with this data to avoid future repeat errors regarding at least this one identified misleading operation.

[0042] In summary, the invention proposes a reinterpretation of user interface input based on changes in interface appearance. List of reference sings 10 system 12 automated assistant 14 electronic computing device 16 vehicle 18 assistance system 20 microphone 22 speaker 24 display device 26 user interface 28 detection devices 30 driver / user

Claims

1. Method for utilizing an automated assistant (12) for an application in an electronic computing device (14) in a vehicle (16), comprising the following steps:- Capturing an input from a user to perform an operation by the application;- Determining by the automated assistant (12) that the input is intended to undo a prior operation in the application, previously captured by the automated assistant (12);- Identifying at least one such misleading operation from a set of previous operations depending on the input, which may have influenced the current state of the application;- Undoing at least one identified misleading operation, characterized in thatthe application recognizes which misleading operation the user has undone, and data of the misleading operation and data leading to the undoing of the misleading operation are stored,and the automated assistant (12) is trained with this data to avoid future repeat errors regarding at least this one identified misleading operation.

2. Method according to claim 1, characterized in thatthe identified misleading operations are communicated to the user and presented for selection.

3. Method according to claim 2, characterized in thatthe selection is based on the frequency with which the undoing of the misleading operation has already been performed.

4. Method according to any of the preceding claims, characterized in thatdata leading to the misleading of the operation is also taken into account for other training the automated assistant 12) for other operations.

5. Method according to any of the preceding claims, characterized in thatthe automated assistant (12) is trained with this data depending on different users.

6. System (10) to execute a Method for utilizing an automated assistant (12) for an application in an electronic computing device (14) in a vehicle (16) according to any of the preceding claims.

7. A computer program product comprising program code means for performing a method according to claim 1 to 5.

8. A non-transitory computer-readable storage medium comprising at least the computer program product according to claim 7.12

Images