Adaptive human-computer interface
By integrating sensors and machine learning systems into the vehicle, the human-machine interface of the touchscreen display is automatically adjusted to adapt to the user's interaction pattern, solving the problem of poor user experience in existing technologies and realizing a personalized user interaction experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- GM GLOBAL TECHNOLOGY OPERATIONS LLC
- Filing Date
- 2025-01-20
- Publication Date
- 2026-05-26
AI Technical Summary
The existing human-machine interface system of vehicle touch screen displays cannot adapt to the personalized interaction needs of different users, resulting in a poor user experience.
By detecting user interactions with the vehicle through vehicle sensors, and accumulating training datasets using machine learning systems, the human-machine interface of the touchscreen display is automatically adjusted to adapt to the specific user's interaction patterns and preferences, including voice, touch, eye tracking, and gesture interaction. Combined with vehicle conditions such as speed, seat position, and weather, the LSTM system is used for personalized adjustments.
The touchscreen display human-machine interface system can automatically adjust according to user characteristics and vehicle conditions, improving the convenience and personalized experience of user interaction.
Smart Images
Figure CN122086281A_ABST
Abstract
Description
Technical Field
[0001] This topic relates to vehicles, and more specifically to human-machine interface systems that include the ability to adapt based on specific user interactions. Background Technology
[0002] Vehicles typically include multiple human-machine interface (HMI) systems that allow users to interact with and control one or more corresponding vehicle systems. A typical HMI is a display presented on a touchscreen. Touchscreen displays allow buttons and other interactive elements to be displayed and interacted with by the user. To ensure maximum usability, existing HMIs for touchscreen displays are typically designed for ordinary users and / or common use cases.
[0003] However, every user is different and will have their own interaction patterns and preferences. In most cases, an HMI design for the average user provides a sufficiently good interface. However, such an interface is not ideal or optimized for a particular user if the user does not have to expend a great deal of effort to identify and modify the HMI. Furthermore, in some cases, users may have narrow uses or specific preferences that are not suited to designs for the average user.
[0004] Therefore, it is desirable to provide an HMI system for vehicle touchscreen displays that adaptively changes to meet the interaction types and preferences of specific users, without requiring user involvement in a time-consuming and difficult manual HMI modification process. Summary of the Invention
[0005] In one exemplary embodiment, the vehicle includes vehicle sensors that communicate with a controller, such that the output signal of each sensor is provided to the controller. The sensors are configured to detect at least one vehicle condition. A human-machine interface system for the vehicle includes a touchscreen display that communicates with the controller. The controller includes a vehicle operator recognition module and a human-machine interface module. The human-machine interface module is configured to cause the touchscreen display to display the human-machine interface, accumulate a machine learning training dataset based on user interactions with the human-machine interface, train a machine learning system using the machine learning training dataset, and automatically adapt the displayed human-machine interface based on the trained machine learning system.
[0006] In addition to one or more features described herein, the controller also includes at least one data connection to at least one remote data source.
[0007] In addition to one or more features described herein, the remote data source includes a set of processors configured to at least partially implement training a machine learning system using a machine learning training dataset, and to automatically adapt the trained machine learning system to the displayed human-computer interface.
[0008] In addition to one or more features described herein, at least one remote data source includes at least one data store for vehicle conditions.
[0009] In addition to one or more features described in this paper, accumulating machine learning training datasets based on user-machine interface interactions includes monitoring vehicle operator interactions with the human-machine interface and extracting at least one machine learning feature from each interaction.
[0010] In addition to one or more features described in this article, vehicle operator interactions include voice interaction, touch interaction, eye-tracking interaction, and gesture interaction.
[0011] In addition to one or more features described in this paper, each feature defines a single interaction and a set of conditions associated with the interaction.
[0012] In addition to one or more features described herein, a set of conditions includes at least one preceding interaction.
[0013] In addition to one or more features described herein, a set of conditions includes at least one immediately following interaction.
[0014] Apart from one or more features described in this article, each feature is in the same data format as every other feature.
[0015] In addition to one or more features described herein, a set of conditions includes multiple features that occur concurrently with the interaction, including vehicle speed, seat position, driver position, weather conditions, ambient lighting, time of day, direction of travel, and traffic conditions.
[0016] In addition to one or more features described herein, the human-computer interface module is also configured to continuously update the displayed human-computer interface after monitoring interactions and conditions and updating the machine learning training dataset using features that define subsequent interactions to adapt the displayed human-computer interface.
[0017] In addition to one or more features described herein, continuously updating the displayed human-machine interface after adaptation also includes identifying at least one target of the adapted displayed human-machine interface and gaps in recorded interactions with the adapted displayed human-machine interface.
[0018] In addition to one or more features described herein, a training-based machine learning system automatically adapts to the displayed human-computer interface by at least one of the following: changing the spacing between icons, changing the size of one or more icons, changing the brightness of one or more icons, changing the size of the touch area, and changing the options in the menu selection system.
[0019] In addition to one or more features described herein, automatically adapting the displayed human-machine interface includes changing options in a menu selection system, and wherein changing options in a menu selection system includes hiding at least one option.
[0020] In addition to one or more features described in this paper, the machine learning system is a Long Short-Term Memory (LSTM) system.
[0021] In another exemplary embodiment, a method for adapting a human-computer interface based on user interaction records user interactions with the interface. Each recorded user interaction is converted into a corresponding machine learning feature, and each machine learning feature is stored in a feature set. The feature set is provided as a training dataset to a machine learning system, and the machine learning system is trained using the training dataset. At least one element of the human-computer interface is modified based on the output of at least one trained machine learning system.
[0022] In addition to one or more features described in this paper, each feature defines a single interaction and a set of conditions associated with the interaction.
[0023] In addition to one or more features described herein, a set of conditions includes multiple features that occur concurrently with the interaction, including vehicle speed, seat position, driver position, weather conditions, ambient lighting, time of day, direction of travel, and traffic conditions.
[0024] In addition to one or more features described herein, changing at least one element of the human-computer interface includes at least one of the following: changing the spacing between icons, changing the size of one or more icons, changing the brightness of one or more icons, changing the size of the touch area, and changing the options in the menu selection system.
[0025] The above-described features and advantages, as well as other features and advantages, of this disclosure will become apparent when taken in conjunction with the accompanying drawings and the following detailed description. Attached Figure Description
[0026] Other features, advantages, and details appear only as examples in the following detailed description, which is described in detail with reference to the accompanying drawings, in which:
[0027] Figure 1 An exemplary vehicle includes an adaptive human-machine interface (HMI) system;
[0028] Figure 2 This is an example touchscreen HMI;
[0029] Figure 3 It is used to automatically adapt using machine learning-based processes. Figure 2 Example of a touchscreen HMI process;
[0030] Figure 4 Is with Figure 3 Example machine learning architectures used in the process; and
[0031] Figure 5 It is possible to be Figure 3 The process achieves a set of HMI adaptations. Detailed Implementation
[0032] The following description is exemplary in nature only and is not intended to limit this disclosure, its application, or use. It should be understood that throughout the drawings, corresponding reference numerals denote the same or corresponding parts and features. As used herein, the term "module" refers to processing circuitry that may include application-specific integrated circuits (ASICs), electronic circuitry, processor (shared, dedicated, or group) and memory executing one or more software or firmware programs, combinational logic circuitry, and / or other suitable components that provide the described functionality.
[0033] As used herein, the term controller refers to a system that includes a processor and memory. This system can be a dedicated controller, a general-purpose controller including subprocesses and modules for implementing the described control functions, a network of local processors and memory configured to work collaboratively to implement the described control functions, a combination of local and remote processors and memory configured to work collaboratively to implement the described functions, a cloud computing system, or any similar system in which at least one processor and memory is configured to implement the described control functions.
[0034] In a general example of the system described in this paper, a controller is used to track interactions between the user and the vehicle. In addition to the type of interaction, various conditions occurring concurrently with the interaction (e.g., vehicle speed, weather conditions, lighting, traffic, etc.) are tracked and associated with specific interactions. Features are extracted from the tracked interactions and associated conditions, and then used to train a machine learning (ML) model. The output of the ML model is used to generate modifications to the default HMI, thereby creating a personalized graphical user interface (GUI) for the user. The personalized GUI is provided to the human-machine interface system (HMI), which is then adapted to utilize the personalized GUI.
[0035] In some examples, after implementing an adapted HMI, interaction patterns with the adapted HMI are continuously monitored to identify gaps in the adapted HMI where users are unlikely to interact in the expected way. Events related to these gaps are then used to extract features, and these features are used to retrain the ML model. This, in turn, allows for the generation and provision of new, updated, adapted HMIs to the HMI system.
[0036] According to an exemplary embodiment, Figure 1A schematic diagram of a vehicle 10 is shown, which includes a body 12 having an interior passenger compartment 14. Figure 2 Human-machine interface 50 is shown. Vehicle operator 20 is located in the driver's seat and operates vehicle 10. Vehicle operator 20 interacts with some vehicle functions using a touchscreen display 30 connected to controller 40. Touchscreen display 30 displays human-machine interface 50, and vehicle operator 20 can interact with controller 40 by touching various elements on human-machine interface 50.
[0037] The controller 40 includes a driver identification module 42 capable of identifying a unique vehicle operator 20. In one example, the unique vehicle operator 20 is identified via visual recognition provided by a camera 60, where the camera 60 defines a field of view 62 that includes the vehicle operator 20. In alternative examples, the driver identification module 42 may use other processes including driver login, driving style recognition, token recognition (e.g., connection to the driver's mobile phone), or any similar methods to identify the unique vehicle operator 20.
[0038] Vehicle 10 also includes sensors 70, which monitor vehicle operation and conditions, including speed, revolutions per minute, ambient temperature, ambient lighting, and external temperature. Any conventional sensor suitable for detecting the corresponding characteristics can be used, and the sensor values are provided to controller 40.
[0039] The controller 40 includes a wireless connection 43 for connecting to the remote computing system 80. The remote computing system 80 may include data sources storing vehicle-related data, such as weather tracking databases, mapping systems, etc., and network processing accessed via one or more remote processing centers. The one or more remote processing centers allow computationally intensive processes or portions of processes to be performed outside the vehicle 10, thereby allowing processors within the controller 40 or other vehicle elements to have lower requirements.
[0040] In some examples, controller 40 includes a Global Navigation Satellite System (GNSS) 44. GNSS 44 uses satellite positioning to identify the global location of vehicle 10.
[0041] The human-machine interface 50 displayed on the touchscreen display 30 is generated and controlled by the HMI module 46 in the controller 40. The HMI module 46 initially provides the touchscreen display 30 with a default human-machine interface 50, which defines multiple icons 52, application selectors 54, and a main display portion 56 (collectively referred to as elements 52, 54, and 56). Each of elements 52, 54, and 56 has a defined height and width, as well as a defined placement on the HMI 50. As used herein, height refers to the vertical axis on the HMI 50, and width refers to the horizontal axis on the HMI 50. In one example, the default height, width, and placement are optimized for a typical user.
[0042] In alternative examples, elements 52, 54, and 56 may include any number of additional elements and / or element types, including but not limited to scrollbars, the main part of a partition 56, drop-down menus, or any other graphical user interface (GUI) element.
[0043] In addition to the default configuration of HMI 50, HMI module 46 also includes a process for adapting HMI 50 to a specific user (vehicle operator 20). See also... Figure 1 and Figure 2 , Figure 3 The process 300 for modifying the default human-machine interface 50 to a specific user identified by the driver recognition module 42 is shown.
[0044] User interaction 302 occurs each time a user interacts with HMI 50. Controller 40 recognizes user interaction 302 and records the interaction in log interaction mode step 304. The recorded interaction includes a specific interaction (e.g., touching a first element) and any available conditions that occur concurrently with the specific interaction. For example, conditions may include current weather, current traffic conditions, current ambient lighting, the vehicle's current geographic location, the vehicle's current speed, time of day, direction of travel, driver's position (e.g., distance between the driver and the steering unit or display), seat position, and any other available relevant conditions.
[0045] Furthermore, in some examples, conditions may include precursor interactions and / or subsequent interactions. As an example, when vehicle operator 20 cancels a previous selection and immediately performs the current interaction, a precursor interaction condition would define that interaction as a correction of the immediately preceding interaction. Similarly, when vehicle operator 20 immediately follows an interaction with a subsequent interaction, such as selecting an element from a submenu, that subsequent selection can be stored as a condition.
[0046] In addition to direct conditions, each interaction may also include one or more secondary conditions, such as age-based driver preferences and driving habits, occupation-based driving habits, and the methods used by the vehicle operator to perform the interaction (each finger number used for tapping, the angle of the finger, the pressure applied during tapping, etc.).
[0047] In the initial iteration, before proceeding to the process log and feature extraction step 306, process 300 records a threshold number of interactions in the log interaction mode step 304. The specific threshold depends on the particular model being implemented and the number of interactions required to exceed the confidence threshold used to train the model on the dataset. Interactions can be voice interactions, touch interactions, eye-tracking interactions, gesture interactions, or any other type of interaction with the HMI 50.
[0048] Once the threshold is reached, process 300 extracts features from the interaction pattern in step 306. A feature is a single, defined data point that identifies an interaction and the conditions associated with that interaction. All features are presented in the same format, effectively normalizing all recorded interactions into a single data format that can be provided as a training set for a machine learning system. In one example, features may match those defined in the following feature representations:
[0049] Input layer :
[0050] {
[0051] "interaction id":"####",
[0052] “screen width”:“640”,
[0053] "screen height":"480",
[0054] "user id": "#####",
[0055] "timestamp":""######",
[0056] “driver”:“y”,
[0057] “right_handed”:“n”,
[0058] "hand(or)finger tracking":{
[0059]
[0060] The input layer refers to the feature set. Within each feature, the interaction ID identifies the type of interaction (e.g., screen tap). For example, the screen width condition identifies the width of the HMI 50 in pixels, the screen height condition identifies the height of the HMI 50 in pixels, the user ID identifies the specific vehicle operator 20 performing the interaction, the timestamp identifies when the interaction occurred, the driver condition identifies whether the interaction was performed by the driver or someone else, the right-hand condition identifies whether the driver is right-handed, the hand or finger tracking condition identifies the position of the finger touch on the HMI 50 on the vertical (x) axis / height and the horizontal (y) axis / width, the touch detection condition identifies whether the touch was determined, the speed condition identifies the speed of the touch in milliseconds, and the eye gaze tracking condition identifies which part of the HMI 50 the vehicle operator 20 is looking at, the diameter of the vehicle operator's pupil, and how low the vehicle operator is looking at the HMI 50.
[0061] The pseudocode features provided are a single example implementation, and actual implementations may include any number of additional conditions that quantify the conditions applied to the interaction type.
[0062] After processing all interactions into features, the features are combined into a machine learning training dataset, which is used to train the machine learning system in step 308 of training the ML system. In one example, the machine learning system is a Long Short-Term Memory (LSTM) machine learning algorithm. LSTM algorithms are particularly well-suited for process 300 due to their ability to accurately predict the next action in a time series. In this particular implementation, the next action is the possible next interaction with the human-machine interface 50, and this probability of the next action is used to generate changes to the human-machine interface 50 in step 310 of generating personalized GUI asset attributes.
[0063] In a typical implementation, the output of the machine learning algorithm is either a probability vector for selecting specific elements located adjacent to each other, or a heatmap for the next interaction with a specific part of the human-machine interface 50. This output is then applied to one or more rules to generate changes indicated by the output.
[0064] The personalized GUI includes changes to elements 52, 54, and 56 that constitute the HMI 50. Changes may include size changes, position changes, menu reordering, etc. In the asset attribute update step 312, the resulting defined GUI for the HMI 50 replaces the existing HMI 50. Once updated, the new HMI 50 is presented to the vehicle operator 20, and interaction with the updated HMI 50 is monitored in the monitor update HMI step 314.
[0065] During step 314, interactions are recorded in the same manner as in the initial log interaction mode step 304. Furthermore, interactions are analyzed for gaps in the gap quantification step 316. A gap occurs when one or more elements of the HMI 50 are adapted to achieve a target but fail to do so. As an example, when the target is to reduce the number of taps required to engage vehicle features to 1, and the vehicle operator 20 typically still performs 2 taps, a gap of 1 tap exists even though reducing to 2 taps is still an improvement over the default HMI 50.
[0066] After a threshold number of interactions, the difference between the recorded interactions and the quantization is provided to step 304, where process 300 is repeated. The threshold number can vary depending on the specific implementation and conditions, and is set to the number of patterns appearing in the logged interactions. Second and subsequent iterations do not need to establish enough interactions to generate a complete training set, as the initially developed training set is supplemented with subsequent interactions.
[0067] Continue to refer to Figure 1-3 , Figure 4 The visual diagram illustrates process 300, in which default HMIs 50 and 402 undergo multiple user interactions 404. User interactions 404 are recorded and processed into features in the user interaction layer 406, and these features are provided to the machine learning system 408. The machine learning system 408 generates a new adapted HMI 410, which replaces the default HMIs 402 and 50. In some examples, the new adapted HMI 410 can be further deployed to a central data system and / or other users, thereby providing more information to the training set and allowing for further refinement of the default HMI.
[0068] Continue to refer to Figure 1-4 , Figure 5 Several possible adaptations 510, 520, 530, and 540 can be made to HMI 50, and in particular to specific elements of HMI 50.
[0069] In the first adaptation 510, a set of icons is included in row 514, where each icon has a depiction portion with a set width 512. In the base HMI 50, each icon contains a segment with the same width 512 as each of the other icons. In the transformed segments, the width 512 of the segments varies, and the order of the icons has been changed to emphasize more frequently used icons (squares and triangles) and de-emphasize less frequently used icons (circles and diamonds). This adaptation can be performed to increase the visibility of more frequently used icons and reduce the number of corrections required due to users accidentally touching the wrong icon or accidentally touching multiple icons.
[0070] In the second adaptation 520, a single icon is contained within a defined space, and the size of the icons within that defined space 522 can be increased or decreased. This modification increases the visibility and / or readability of the icons, and can be achieved when the machine learning output aims to reduce the number of incorrect selections.
[0071] In the third adaptation 530, the element includes a defined touch area 532. The touch area is the region of the HMI 50 registered as a touch corresponding to elements 52, 54, and 56 when the vehicle operator 20 touches the screen. In the third adaptation 530, the size of one or more of the touch areas 532 is increased, thereby increasing the real estate of the HMI 50, which can be touched to correspond to any selection of the included elements 52 and 54.
[0072] In the fourth adaptation 540, the element includes multiple selection boxes 542, each of which includes an additional submenu. Furthermore, a scroll bar 544 allows the vehicle operator 20 to scroll up and down and change which selection box 542 is displayed on the screen. When the machine learning output indicates that certain selections are not used, these selection boxes 542 can be hidden or included within other selection boxes, and the nesting order of the selection boxes 542 can be changed. This, in turn, allows the complete list of selection boxes 542 to appear without scrolling, and allows more frequently used selections to be placed at a higher level within the selection boxes 542.
[0073] Figure 5 The adaptations 510, 520, 530, and 540 described and illustrated are exemplary in nature and are not intended to be exhaustive. In one practical implementation, the machine learning output defines the target of the adaptation, and process 300 may apply rules to translate the target into an actual adaptation. In some examples, further adaptations may include increasing or decreasing the brightness of elements, removing or hiding elements, or any similar type of adaptation.
[0074] The terms “a” and “an” do not indicate a limitation of quantity, but rather that at least one of the referenced items is present. Unless the context clearly indicates otherwise, the term “or” means “and / or”. Throughout the specification, the reference to “aspect” means that a particular element described in connection with that aspect (e.g., a feature, structure, step, or characteristic) is included in at least one aspect described herein, and may or may not be present in other aspects. Furthermore, it should be understood that the described elements may be combined in any suitable manner in the aspects.
[0075] When an element such as a layer, film, region, or substrate is referred to as being "on" another element, it can be directly on the other element, or there may be intermediate elements present. Conversely, when an element is referred to as being "directly" on another element, there are no intermediate elements present.
[0076] Unless otherwise stated herein, all test standards are the most recent standards in force up to the date of filing of this application, or, if priority is claimed, the date of filing of the earliest priority application in which a test standard appears.
[0077] Unless otherwise defined, the technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure pertains.
[0078] While the foregoing disclosure has been described with reference to exemplary embodiments, those skilled in the art will understand that various changes can be made and elements can be substituted with equivalents without departing from its scope. Furthermore, many modifications can be made to adapt particular situations or materials to the teachings of this disclosure without departing from the basic scope of this disclosure. Therefore, it is intended that this disclosure be limited to the specific embodiments disclosed, but will include all embodiments falling within its scope.
Claims
1. A vehicle comprising: Vehicle sensors that communicate with a controller, such that the output signal of each vehicle sensor is provided to the controller, the vehicle sensors being configured to detect at least one vehicle condition; A human-machine interface system, the human-machine interface system including a touch screen display, the touch screen display communicating with a controller; The controller includes a vehicle operator recognition module and a human-machine interface module; and The human-machine interface module is configured to display the human-machine interface on a touch screen display, accumulate machine learning training datasets based on user interactions with the human-machine interface, use the machine learning training datasets to train the machine learning system, and automatically adapt the displayed human-machine interface based on the trained machine learning system.
2. The vehicle of claim 1, wherein the controller further comprises at least one data connection to at least one remote data source.
3. The vehicle of claim 2, wherein the remote data source comprises a set of processors configured to at least partially implement training the machine learning system using the machine learning training dataset, and to automatically adapt the displayed human-machine interface based on the trained machine learning system.
4. The vehicle of claim 2, wherein at least one remote data source includes at least one data storage of vehicle status.
5. The vehicle of claim 1, wherein accumulating the machine learning training dataset based on user interaction with the human-machine interface includes monitoring the interaction between the vehicle operator and the human-machine interface and extracting at least one machine learning feature from each interaction, and optionally, wherein the vehicle operator interaction includes voice interaction, touch interaction, eye-tracking interaction and gesture interaction.
6. The vehicle of claim 5, wherein each feature defines a single interaction and a set of conditions associated with said interaction, and wherein each feature has the same data format as each other feature.
7. The vehicle of claim 6, wherein the set of conditions includes at least one of a preceding interaction and a following interaction.
8. The vehicle of claim 6, wherein the set of conditions includes a plurality of features occurring concurrently with the interaction, the plurality of features occurring concurrently with the interaction including vehicle speed, seat position, driver position, weather conditions, ambient lighting, time of day, direction of travel, and traffic conditions.
9. The vehicle of claim 6, wherein the human-machine interface module is further configured to continuously update the displayed human-machine interface after adapting it by monitoring log interactions and conditions and updating the machine learning training dataset using features defining subsequent interactions; and The process of continuously updating the displayed human-machine interface after adapting it also includes identifying at least one target of the displayed human-machine interface and gaps in the recorded interactions with the adapted displayed human-machine interface.
10. The vehicle of claim 1, wherein the trained machine learning system automatically adapts the displayed human-machine interface by at least one of the following: changing the spacing between icons, changing the size of one or more icons, changing the brightness of one or more icons, changing the size of the touch area, and changing the options in the menu selection system by hiding at least one option.