Computer-implemented method for improving the performance of a cockpit user interface

Abstract

Computer-implemented method for improving the performance of a cockpit user interface (1), comprising at least the following components: - a first processor structure for operating a human-machine interface (2) for the graphical representation of control elements (3), which are prepared for operating functions in a tree structure (4); - a second processor structure for executing a machine learning algorithm; - a database (5) for storing and providing information on the controls (3) of the human-machine interface (2) and on user behavior, wherein the method comprises at least the following steps in the order mentioned: a. by means of the first processor structure, providing information from the database (5) for the human-machine interface (2) for the graphical representation of operating elements (3); b. when interacting with at least one of the controls (3), using the first processor structure to record this interaction and input this recording into the machine learning algorithm executed on the second processor structure; c. using the machine learning algorithm executed on the second processor structure, comparing the performance of the input interaction with the respective performance of previously known interactions stored in the database (5), and thus finding a similar interaction with better performance; d. by means of the machine learning algorithm executed on the second processor structure, providing information to the first processor structure for the graphical representation of operating elements (3) on the human-machine interface (2) based on the found similar interaction with the better performance, wherein, in the case of a repeated identical interaction with information provided in step a., steps b. to d. are repeated, characterized in that The interaction is performed by a simulation model of a user.

Description

[0001] The invention relates to a computer-implemented method for improving the performance of a cockpit user interface. The invention further relates to a computer program comprising computer program code for executing such a method, and to a computer program product on which the computer program code is stored.

[0002] It is common practice to equip modern vehicles with an entertainment system that gives occupants access to software related to vehicle comfort (such as climate control or music), vehicle settings (such as comfort or sport mode), and auxiliary systems like navigation or hands-free calling. The number of selectable functions and their corresponding controls is immense, and a problem arises: operating such a user interface can distract the driver, thus compromising driving safety. Therefore, it is desirable to offer the user the shortest possible task completion time. This is achieved through the use of a customizable cockpit user interface.

[0003] DE 10 2018 206 653 A1 relates to a method for dynamically adapting an operating device in a motor vehicle, as well as an operating device and a motor vehicle.The operating device is designed to operate at least one vehicle function of the motor vehicle, which provides that, in at least one driving situation, the operating device records observation data describing the current driving situation and usage data describing which of the vehicle functions is currently activated via the operating device, and, based on the observation data and the usage data, generates an assignment rule using a machine learning method, which assigns a state of the operating device adapted to the observation data according to the usage data, and in at least one further driving situation, the state of the operating device is adapted by means of the assignment rule depending on observation data determined in the further driving situation.

[0004] Based on this, the present invention aims to overcome, at least partially, the disadvantages known from the prior art. The features of the invention are defined in the independent claims, for which advantageous embodiments are shown in the dependent claims. The features of the claims can be combined in any technically meaningful way, whereby the explanations in the following description and features from the figures, which comprise supplementary embodiments of the invention, can also be used.

[0005] The invention relates to a computer-implemented method for improving the performance of a cockpit user interface, comprising at least the following components: - a first processor structure for operating a human-machine interface for the graphical representation of control elements, which are prepared in a tree structure for operating functions; - a second processor structure for executing a machine learning algorithm; - a database for storing and providing information on the controls of the human-machine interface and on user behavior, the procedure includes at least the following steps in the order mentioned: a. by means of the first processor structure, providing information from the database for the human-machine interface for the graphical representation of operating elements; b. when interacting with at least one of the controls, the first processor structure records this interaction and inputs this recording into the machine learning algorithm executed on the second processor structure; c. by means of the machine learning algorithm executed on the second processor structure, comparing the performance of the input interaction with the respective performance of previously known interactions stored in the database, and thus finding a similar interaction with better performance; d. by means of the machine learning algorithm executed on the second processor structure, providing information to the first processor structure for the graphical representation of controls on the human-machine interface based on the identified similar interaction with better performance, where, in the case of a repeated identical interaction with information provided in step a., steps b. to d. are repeated.

[0006] Unless explicitly stated otherwise, ordinal numbers used in the preceding and following descriptions serve solely for unambiguous differentiation and do not indicate any order or ranking of the components referred to. An ordinal number greater than one does not necessarily imply the presence of another such component.

[0007] The computer-implemented method proposed here is designed to make a cockpit user interface adaptive in such a way that a user is distracted as little or as briefly as possible from their other tasks (for example, driving a vehicle). It should be noted that the method can be executed locally on a motor vehicle, but preferably the motor vehicle is primarily configured to receive the user's commands and monitor their behavior, while the actual changes to the cockpit user interface are calculated externally, for example, in a cloud or in a proprietary backend.Furthermore, it should be noted that the procedure is primarily designed as training, and in one embodiment a user or vehicle only receives the result of a training session, i.e., for example, after defining a user profile, it receives a fixed software architecture for the cockpit user interface.

[0008] In a preferred embodiment, the method is continuously improved and / or the user is continuously monitored, and a correspondingly improved cockpit user interface is provided. When we refer to an improved cockpit user interface, we mean the display of controls on this cockpit user interface through which a user gains access to functions of their vehicle, such as the infotainment system.

[0009] It should be further noted that the first processor structure is designed to operate a human-machine interface, but is, for example, merely a virtual representation of an actual such processor, which is used in the execution of the method proposed here. In a pure training procedure, the second processor structure is preferably located outside of a motor vehicle and communicates with a cloud via a local backend. From a purely structural point of view, the first and second processor structures do not necessarily have to be separate components. It should also be noted that a processor structure can be formed, for example, by a classic CPU (Central Processing Unit) or by an architecture of multiple interconnected processors.The first processor structure includes, for example, a processor optimized for graphical representation, such as a GPU (Graphical Processing Unit). The graphical representation of controls on the cockpit user interface is based on a tree structure, from which the desired specific function can be selected. For example, if the user wants to listen to music, they select a parent control called "Media" and are then presented with a display of controls offering a selection of sources, such as radio, Bluetooth, locally stored media, and / or others. If the user then wants to play music from their smartphone, they select the "Bluetooth" control and are then, if necessary, directed to their smartphone's control structure, where they are presented with a new range of options.

[0010] If a user frequently wants to play music on their smartphone using the same app, it makes sense to modify this tree structure accordingly, for example, by moving the app directly to the first level. The database is now set up to store such information about the controls, as well as information about the behavior of a single user or multiple users. Based on this, steps a. to d. are executed, thereby improving the interaction with the human-machine interface with regard to distractions and user satisfaction. Alternatively or additionally, the information provided in step b. also includes further characteristics of the controls that simplify their location for the user, such as the color and size of a control, or synonyms for voice or gesture control.

[0011] It is proposed here that this improvement process be repeated frequently, preferably every time the same interaction occurs. The tree structure, for example, is organized using a Monte Carlo algorithm and then improved. Performance is preferably measured by the time required, but various aspects should be considered to achieve a short time, namely the speed at which a control is located, the accuracy (i.e., that the desired control is selected), the color representation, and the size of the control. The interaction is not limited to tapping but is also accessible via gesture control or voice control, for example.

[0012] The method is characterized primarily by the fact that the interaction is performed by a simulation model of a user.

[0013] It is proposed here that instead of relying solely on the inputs or interactions of a real user, a user simulation model should be used. This simulation model learns or estimates user behavior, thereby accelerating the learning loop and significantly increasing the number of iterations in a shorter timeframe by handling the repetition of interactions. This substantially increases the amount of data available for a machine learning algorithm, thus improving learning progress and enabling it to be executed much more quickly.

[0014] In an advantageous embodiment of the computer-implemented method, it is further proposed that steps b. to d. are repeated in a loop until a predetermined target performance is achieved, preferably also recording interactions from at least one real user.

[0015] It is proposed here that a specific target performance is defined, and a loop consisting of steps b. to d. is repeated until this target performance is achieved. It is advantageous to use a user simulation model, as explained above.

[0016] In an advantageous embodiment, interactions of real users are recorded and used, for example, to improve a user's simulation model.

[0017] In an advantageous embodiment of the computer-implemented method, it is further proposed that in step d. the tree structure of the functions operable via the human-machine interface is changed.

[0018] As mentioned above, it is now proposed that the underlying tree structure for accessing certain functions (controls) be modified to improve performance during interaction. It should be noted that this is only one of many ways to improve performance during interaction.

[0019] In a further advantageous embodiment of the computer-implemented method, it is proposed that the frequency of identical operations is recorded over a multitude of individual interactions, and that in step d. the tree structure is restructured according to this frequency.

[0020] It is proposed here to consider the frequency of interactions, for example, as input for the machine learning algorithm on the second processor structure. Thus, frequently used functions / controls are placed higher up in the hierarchy of the tree structure, contrary to a logical tree structure. In contrast, a rarely or never used function is placed lower in the hierarchy or below a less specific control and / or one that includes a safety warning, advising the driver not to use the following controls while driving, or even restricting access to such controls while driving.

[0021] It is further developed in an advantageous embodiment of the A computer-implemented method has been proposed in which a group of different interactions are summarized in the database to form a user profile. where, if a better-performing interaction is found, the user profile that includes the found better interaction is used. in step d. is provided by the second processor structure as a replacement for the current user profile.

[0022] The proposal here is to group various interactions into a user profile, so that a user is immediately provided with suitable interactions for other functions based on their behavior during a particular interaction. For example, it might be determined that a user frequently travels for business and uses a navigation system with constantly changing destinations or with known and saved destinations, and also frequently makes hands-free calls while driving. Therefore, they would be offered particularly short operating times for this interaction. Another user, for instance, might primarily travel in their free time or while commuting between home and work, and enjoys listening to music and testing different modes of their vehicle to optimize fuel consumption and / or achieve a sportier driving experience.

[0023] Business travelers, on the other hand, have little time or patience for selecting music and typically set their vehicle preferences once and leave them unchanged. Therefore, such functions do not require optimized interaction for business travelers, and a corresponding control element remains, for example, at a lower hierarchical level and / or in a less visible or accessible location within the human-machine interface (e.g., in a section of a screen accessible by scrolling or flipping, or less prominent due to size or color).

[0024] It is proposed here that when an interaction resulting in improved performance is detected, the entire user profile should be replaced. This is advantageous when the user is well-known and / or a good first approach when the user is unknown (e.g., a new user), so that they are highly likely to already have an improved user profile.

[0025] It is further developed in an advantageous embodiment of the A computer-implemented method has been proposed in which a group of different interactions are summarized in the database to form a user profile. where a new user profile is created when an interaction with better performance is found, where the new user profile includes at least one interaction of the current user profile and includes at least the specific interaction found with better performance as a replacement for the corresponding current interaction.

[0026] It is proposed here that a new user profile be created when improved performance is found for a single interaction. In one embodiment, for example, only that one interaction within a user profile is replaced, while the other interactions are retained. This is advantageous if the other interactions already perform very well and only the single interaction can be improved. It is also beneficial if a user does not want to relearn certain interactions and would be dissatisfied with replacing the entire user profile.

[0027] In an advantageous embodiment, a user is notified when an interaction is changed and, if necessary, given the option to choose whether the entire user profile should be modified or only a single interaction, with this being triggered by performing an interaction. In a training situation, the choice between replacing the entire user profile or only a single interaction is made based on performance, preferably also using test users to verify the training results.

[0028] It should be noted again that a motor vehicle and / or a motor vehicle user is equipped with one or more fixed user profile(s), for example, after an initial survey or an initial or repeated test phase, whereby only one or more user profile(s) are selected. For example, a motor vehicle may be assigned several user profiles for different drivers, or a driver may be assigned several user profiles selectable at the start of the journey (for example, a work profile and a private profile).

[0029] In another embodiment, the display of the controls is subject to a continuous process, whereby new functions are integrated and new user behaviors are observed, for example, due to new trends and / or offerings, such as a new app. In one embodiment, a user is asked whether they want to allow a change or whether they want to perform an update.

[0030] It is further developed in an advantageous embodiment of the A computer-implemented method has been proposed that evaluates performance based on a user's service time.

[0031] It is further developed in an advantageous embodiment of the A computer-implemented method has been proposed that information about the controls for the human-machine interface includes at least one of the following properties: - Arrangement and type of display of the controls; - Arrangement and type of presentation of functional information; - A user requirement for a specific operating mode; and - Type of feedback to an operator during an interaction.

[0032] It is proposed here that, for example, the arrangement and display of the controls can be changed. For instance, a control can be displayed larger or smaller. For example, many or few controls can be displayed simultaneously. For example, one screen can be selected from a plurality of screens, such as in the classic speedometer area behind the steering wheel, in the center console, or on a head-up display (on and / or in the line of sight through the windshield). For example, it can be selected whether a control is displayed purely graphically or also audibly.

[0033] It is further or alternatively proposed that, for example, the arrangement and manner of display of functional information be changeable. Such functional information could include, for example, a speedometer (speed indicator), a telephone, a thermometer (i.e., a temperature reading), tire pressure, and / or other information. This is not a control element with which interaction is possible, but rather an interaction with a control element is potentially triggered and executed based on this information.

[0034] It is further suggested, or alternatively, that the user can modify the required input method. An input method could be, for example, a double-click, swipe, gesture control, voice command, eye movement, key press, and / or other methods. For instance, multiple input methods could be used simultaneously, making the requirement somewhat flexible.

[0035] It is further suggested, or alternatively, that the type of feedback provided during an interaction should be changeable. This feedback could include, for example, a pop-up of a symbol or control element or functional information, a symbol becoming larger or smaller, a color change, an acoustic signal, a vibration, and / or other feedback.

[0036] According to another aspect, a computer program is proposed, comprehensive a computer program code, wherein the computer program code is executable on at least one computer such that the at least one computer is caused to execute one of the methods according to an embodiment as described above, wherein at least one unit of the computer: - is located in the motor vehicle, preferably as an on-board computer; and / or - is set up for communication with a cloud or backend, on which preferably at least part of the computer program code is provided.

[0037] The method described here is implemented in a computer according to this embodiment. The computer-implemented method is stored as computer program code, wherein the computer program code, when executed on a computer, causes the computer to execute the method according to an embodiment as described above.

[0038] The computer-implemented method is realized, for example, by a computer program, wherein the computer program comprises the computer program code, and wherein the computer program code, when executed on a computer, causes the computer to execute the method according to an embodiment as described above. Computer program code is synonymously defined as one or more instructions or commands that cause a computer or a processor to perform a series of operations, which, for example, represent an algorithm and / or other processing methods.

[0039] The computer program is preferably executable, either partially or completely, on a server or server unit of a cloud system and / or on at least one unit of the computer. The term "server" or "server unit" here refers to a computer that provides data and / or operational services or services for one or more other computer-based devices or computers, thus forming the cloud system. The at least one unit of the computer in the motor vehicle (for example, the on-board computer) is, for instance, conventionally configured and comprises a memory unit and a processor. Alternatively, the at least one unit of the computer is configured for communication with a motor vehicle, for example, as part of a server and / or a cloud, with the server and / or the cloud being located, for example, on-site at a computer manufacturer's premises.

[0040] The terms cloud system or computer are used here synonymously with devices known from the prior art. A computer therefore comprises one or more general-purpose processors (CPUs) or microprocessors, RISC processors, GPUs, and / or DSPs. The computer also includes additional elements such as memory interfaces or communication interfaces. Alternatively or additionally, the terms refer to a device capable of executing a provided or integrated program, preferably using a standardized programming language (such as C++, JavaScript, or Python), and / or controlling and / or accessing data storage devices and / or other devices such as input and output interfaces.The term "computer" also refers to a multitude of processors or a multitude of (sub)computers that are interconnected and / or linked and / or otherwise communicate with each other and may share one or more other resources, such as memory. A (data) storage device is, for example, a hard disk drive (HDD) or a (non-volatile) solid-state memory, such as ROM or flash memory (Flash EEPROM). The storage often comprises multiple individual physical units or is distributed across a multitude of separate devices, allowing access via data communication, such as a package data service. The latter is a decentralized solution, where the memory and processors of multiple separate computers are used instead of, or in addition to, a central server.

[0041] According to another aspect, a computer program product is proposed, on which a computer program code is stored, wherein the computer program code is executable on at least one computer such that the at least one computer is caused to execute at least one of the methods according to an embodiment as described above, wherein at least one unit of the computer: - is located in the motor vehicle, preferably as an on-board computer; and / or - is set up for communication with a cloud or backend, on which preferably at least part of the computer program code is provided.

[0042] As a computer program product, comprising the computer program code described above, it is stored, for example, on a medium such as RAM, ROM, an SD card, a memory card, a flash memory card, or a disc, or on a server and can be downloaded. Once the computer program is made readable via a read device, such as a drive and / or an installation, the contained computer program code and the procedure contained therein can be executed by a computer or in communication with a plurality of server units, for example, as described above.

[0043] The invention described above is explained in detail below against the relevant technical background with reference to the accompanying drawings, which show preferred embodiments. The invention is in no way limited by the purely schematic drawings, although it should be noted that the drawings are not dimensionally accurate and are not suitable for defining size relationships. It is illustrated in Fig. 1. A cockpit user interface; Fig. 2. A combination of a Monte Carlo tree structure and a self-reinforcing machine learning algorithm in a schematic view; and Fig. 3. A flowchart of a procedure for improving the performance of a cockpit user interface.

[0044] In Fig. 1 is In Fig. Figure 1 shows an example of a cockpit user interface 1 with a touchscreen as a human-machine interface 2, on which a large number of control elements 3 are displayed. It can be seen, for example, that the symbols representing the control elements 3 are significantly larger in the top row than in the two lower rows. These are control elements 3 that are selected more frequently and, due to their size, are firstly quicker to find and recognize, and secondly offer a larger operating area for greater accuracy.The symbols shown represent, for example (from left to right and from top to bottom), a navigation system, media, hands-free system, general settings, video playback, battery status, temperature, vehicle statistics, weather data, connection to a smartphone, a stopwatch or alarm clock, an email account, an update, access to a smart home, user profile input, a calendar, and finally, a power button.

[0045] In Fig. Figure 2 is a schematic representation of a combination of a tree structure (e.g., Monte Carlo) 4 and a self-reinforcing machine learning algorithm. In step i, a (real or virtual) user interacts with an existing tree structure 4, performing the interaction across three levels: a first level where an initial selection is made, a second level where a second selection is made, and a third level where a third selection is made. The algorithm then determines which of the selections is the user's most frequent choice. (Here, a solid line represents the more frequent selection, and a dashed line the less frequent selection.)

[0046] In step ii, the machine learning algorithm expands and evaluates the input, and in step iii, the selection is processed backward to the top level. These three steps are executed in a loop, i.e., starting again from step i after step iii. Once satisfactory performance is achieved, a corresponding tree structure 4, designed based on the frequency of interactions, is offered in step iv.

[0047] In Fig. Figure 3 shows a flowchart of a procedure for improving the performance of a cockpit user interface 1. In step a., information from the database 5 is provided for a human-machine interface 2 for the graphical representation of controls 3, for example as shown in Fig. 1 shown. In step b., an interaction is recorded and this recording is fed into the machine learning algorithm, for example as in step i. in Fig. 2 shown. In step c, the performance of the entered interaction is compared with the respective performance of previously known interactions. The previously known interactions and their respective performance are stored in a database 5. This allows a similar interaction with better performance to be found, if one exists. This process is carried out, for example, as in step ii and step iii. Fig. 2 ab. In step d., the machine learning algorithm provides information for the graphical representation of operating elements 3 on the human-machine interface 2 (compare Fig. 1) Based on the identified similar interaction with the better performance.

[0048] This sequence of steps b. to d. is repeated whenever the same interaction with information provided in step a. is repeated. In an embodiment with a simulation model of a user, this loop is artificially repeated frequently, thus achieving a significant improvement in the performance of the cockpit user interface 1 in a short time, for example, by reducing interaction time. However, input from a real user, especially in a real-world usage environment (i.e., a driving situation), is possible and desirable for improving performance in a preferred embodiment. It should be noted that, given the current performance of an on-board computer system in a motor vehicle, an unchanged tree structure 4 is preferably used, and the data is intended for future vehicles and / or announced updates to the vehicle.

[0049] The invention relates to a computer-implemented method for improving the performance of a cockpit user interface, the procedure includes at least the following steps in the order mentioned: a. Providing information from the database for the human-machine interface for the graphical representation of controls; b. when interacting with at least one of the controls, recording this interaction and inputting this recording into the machine learning algorithm executed on the second processor structure; c. using the executed machine learning algorithm, comparing the performance of the input interaction with the respective performance of previously known interactions stored in the database, and thus finding a similar interaction with better performance; d. Using a machine learning algorithm, information is provided to the first processor structure for the graphical representation of user interface elements based on the identified similar interaction with better performance. The method is characterized primarily by the fact that, upon repeated identical interactions using the information provided in step a., steps b. through d. are repeated.

[0050] The method proposed here makes it possible to achieve a short interaction time for a cockpit user interface. Reference symbol list 1 Cockpit user interface 2 Human-Machine Interface 3 Control element 4 Tree structure 5 Database

Claims

[1] Computer-implemented method for improving the performance of a cockpit user interface (1), comprising at least the following components: - a first processor structure for operating a human-machine interface (2) for the graphical representation of control elements (3), which are prepared for operating functions in a tree structure (4); - a second processor structure for executing a machine learning algorithm; - a database (5) for storing and providing information on the controls (3) of the human-machine interface (2) and on user behavior, wherein the method comprises at least the following steps in the order mentioned: a. by means of the first processor structure, providing information from the database (5) for the human-machine interface (2) for the graphical representation of operating elements (3); b. when interacting with at least one of the controls (3), using the first processor structure to record this interaction and input this recording into the machine learning algorithm executed on the second processor structure; c. using the machine learning algorithm executed on the second processor structure, comparing the performance of the input interaction with the respective performance of previously known interactions stored in the database (5), and thus finding a similar interaction with better performance; d. by means of the machine learning algorithm executed on the second processor structure, providing information to the first processor structure for the graphical representation of operating elements (3) on the human-machine interface (2) based on the found similar interaction with the better performance, where, in the case of a repeated identical interaction with information provided in step a., steps b. to d. are repeated. characterized by , that The interaction is performed by a simulation model of a user. [2] Computer-implemented method according to claim 1, wherein step b. to step d. are repeated in a loop until a predetermined target performance is achieved. [3] Computer-implemented method according to claim 2, wherein interactions of at least one real user are additionally recorded. [4] Computer-implemented method according to one of the preceding claims, wherein in step d. the tree structure (4) of the functions operable via the human-machine interface (2) is modified. [5] Computer-implemented method according to claim 4, wherein The frequency of identical operations is recorded through a large number of individual interactions. in step d. the tree structure (4) is restructured according to this frequency. [6] Computer-implemented method according to any one of the preceding claims, wherein in the database (5) a group of different interactions are summarized for a user profile, where, if a better-performing interaction is found, the user profile that includes the found better interaction is used. in step d. is provided by the second processor structure as a replacement for the current user profile. [7] Computer-implemented method according to any one of the preceding claims, wherein in the database (5) a group of different interactions are summarized for a user profile, where a new user profile is created when an interaction with better performance is found, where the new user profile includes at least one interaction of the current user profile and includes at least the specific interaction found with better performance as a replacement for the corresponding current interaction. [8] Computer-implemented method according to one of the preceding claims, wherein performance is evaluated after a period of operation for a user. [9] Computer-implemented method according to one of the preceding claims, wherein information about the control elements (3) for the human-machine interface (2) comprises at least one of the following features: - Arrangement and type of presentation of the controls (3); - Arrangement and type of presentation of functional information; - A user requirement for a specific operating mode; and - Type of feedback to an operator during an interaction.

Images