Automated design system and method for human machine interface (HMI) configuration and operation

Abstract

A method of operating an automated design system for configuring a human machine interface (HMI) device includes receiving, by a large language model (LLM) module of the automated design system, a set of HMI system development metrics including system requirement data, system destination data, and / or source code data; and developing metrics using these HMI systems to generate an interactive sequence script containing a sequence of steps for executing HMI features on the HMI device. A system controller uses the set of HMI system development metrics to configure the HMI device, and then uses the interaction sequence script to generate a feedback log containing feedback data received from a test clinical contributor to test the configured HMI device. A multi-modal generative artificial intelligence (AI) module uses the feedback data to generate a set of proposed HMI changes. The system controller then reconfigures the HMI device using a subset of HMI changes selected from the proposed HMI changes.

Description

TECHNICAL FIELD

[0001] The present disclosure relates generally to human-machine interface devices. More specifically, aspects of the present disclosure relate to systems and methods for configuring and operating interactive telematics units and digital instrument panels of motor vehicles. BACKGROUND

[0002] Current production motor vehicles, such as modern automobiles, can initially be equipped with a resident network of electronic control units and interactive interface devices that provide enhanced driving and vehicle control features. As vehicle processing, communication, and sensing capabilities improve, manufacturers persist in supplying more automated driving capabilities in the face of a desire to produce fully autonomous "driverless" vehicles that are capable of navigating between heterogeneous vehicle types in both urban and rural scenarios. Original equipment manufacturers (OEMs) are approaching vehicles-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) "talking" cars with higher levels of driving automation that employ autonomous control systems to enable real-time vehicle routing with automated steering, lane changes, scenario planning, and the like. Automation path planning systems, for example, utilize vehicle state and dynamics sensors, geographic location information, map and road condition data, and path prediction algorithms to provide route export with lane center and lane change prediction.

[0003] To provide telecommunication and informatics functionality to occupants, many vehicle passenger compartments are now outfitted with a center stack telematics unit and interactive digital instrument cluster that operate as both a human-machine interface (HMI) and a computing device in the vehicle. The telematics unit, for example, can be wirelessly connected to a cellular network or satellite service for purposes such as real-time navigation assistance, customer support, vehicle tracking, system diagnostics, traffic data, and satellite radio and telephone service. In addition to telecommunication features, the telematics unit also functions as a two-way interface through which vehicle occupants interact with and control a variety of resident vehicle subsystems. During the initial design cycle and life of the vehicle, it can be desirable to develop or reprogram the telematics unit in the vehicle, for example, to support new features and functionality or address potential issues with existing features and functionality. SUMMARY

[0004] The following presents an automated design system with accompanying control logic for configuring and controlling HMI devices, methods for creating such a system, methods for operating such a system, and HMI devices in vehicles developed using such a system. As an example, and not a limitation, generative AI-based approaches automate manual and repetitive design tasks to improve quality, reduce human error, and increase efficiency throughout the HMI software development lifecycle. The Code Interpreter and Language Model (CILM) module processes the prototype software implementation and generates interaction sequences for a designated group of stakeholders, who may include user experience (UX) designers, leaders, and end-user participants. The CILM module can use a set of generative AI models to automate the creation of interaction sequences, system prototyping requirements, design objectives, etc., from the source code. The Large Language Model Insights (LLMI) module provides transcription, summarization, and generation of actionable processes from stakeholder feedback. The LLMI module can observe interactions and feedback in multiple modalities—verbal and non-verbal—and process this data using multimodal generative models. The Code Interpreter and Generative AI-Based Deployment (CIAID) module makes on-the-fly changes in the prototype software implementation. The CIAID module can use a code generation model to translate change requests into code changes for deployment with the primary HMI device.

[0005] This disclosure relates to AI-based HMI software development protocols and processor-executable control logic for configuring and operating HMI devices. In an example, a method is presented for operating an automated design system for configuring an HMI device, such as a telematics unit in a vehicle, a digital instrument panel (IP), a radio interface, a rear entertainment touchscreen panel, etc. In any order and in any combination with any of the options and features disclosed above and below, this representative method includes, for example, receiving a set of HMI system development assets / metrics including system requirement data, system purpose data, and / or source code data via an LLM module, a code interpreter of the automated design system from a developer / UX researcher; for example, generating an interactive sequence script containing a sequence of steps for performing new or redesigned HMI features on the HMI device using the received set of HMI system development assets / metrics via the LLM module; for example, via a resident or remote microcontroller, central processing unit, control module, programmable logic device, or network of processors / controllers / modules / devices (collectively, a "system controller"). The system uses received HMI system development assets / metrics to configure the HMI device; for example, via a system controller and resident storage database, it uses interactive sequence scripts to generate and store feedback logs containing feedback data received from groups of clinical contributors to test HMI features on the configured HMI device; for example, via a multimodal generative AI module of an automated design system (e.g., a bidirectional encoder representation (BERT) model and a long short-term memory (LSTM) deep neural network (DNN) model from a transformer) to generate a set of proposed HMI changes using the feedback data; and for example, via a system controller, it reconfigures the HMI device using a subset of HMI changes selected only from the set of proposed HMI changes.

[0006] Additional aspects of this disclosure relate to an automated design system having accompanying control logic for developing, reconfiguring, and deploying new or updated HMI devices, including wirelessly enabled interactive touchscreen displays for motor vehicles. As used herein, the terms "vehicle" and "motor vehicle" can be used interchangeably and synonymously to include any relevant vehicle platform, such as passenger vehicles, commercial vehicles, industrial vehicles, off-road and all-terrain vehicles (ATVs), motorcycles, farm equipment, aircraft, watercraft, spacecraft, etc. It is conceivable that the disclosed HMI development concepts can be applied equally to both automotive and non-automotive applications. In the example, an automated design system for configuring an HMI device is presented. Among other things, the automated design system includes a large language model module that: receives a set of HMI system development assets / metrics, including system requirement data, system purpose data, and source code data; and uses the received set of HMI system development assets / metrics to generate an interactive sequence script containing a sequence of steps for performing HMI features on the HMI device.

[0007] Continuing the discussion above, the automated design system also includes: a system controller that: configures the HMI device using the received set of HMI system development assets / metrics; and uses interactive sequence scripts to generate and store feedback logs containing feedback data received from a group of clinical contributors for testing HMI features on the configured HMI device; and a multimodal generative AI module that processes the feedback and generates software change requirements (“HMI changes”) to address the feedback. The system controller reconfigures the HMI device using a selected set of HMI changes that can be prioritized.

[0008] This disclosure also relates to a computer-readable medium (CRM) containing controller-executable instructions for developing and deploying new or updated features on an interactive HMI device. In an example, a non-transient CRM stores instructions executable by one or more system controllers and / or one or more system control modules of an automated design system. These CRM-stored instructions, when executed by one or more controllers and / or one or more control modules, cause the automated design system to perform operations including: receiving a set of HMI system development assets / metrics, including system requirement data, system purpose data, and / or source code data, via the LLM module; generating an interactive sequence script containing a sequence of steps for performing HMI features on the HMI device, using the received set of HMI system development metrics via the LLM module; configuring the HMI device using the received set of HMI system development metrics; generating a feedback log containing feedback data received from a group of clinical contributors to test the configured HMI device, using the interactive sequence script; generating a set of proposed HMI changes using the feedback data via the multimodal generative AI module; and reconfiguring the HMI device using a subset of HMI changes selected from the set of proposed HMI changes.

[0009] For any of the disclosed systems, methods, and CRMs, the contributor feedback data included in the feedback log may include multiple feedback statements. In this instance, generating the set of proposed HMI changes may include: for each feedback statement, determining an intention integer predicting the expected intention of the contributor who generated the feedback statement; and for each feedback statement, determining a sentiment integer predicting the expected sentiment of the contributor who generated the feedback statement. Generating the set of proposed HMI changes may also include weighting the score W for each feedback statement. n The calculation is as follows: W n =S n *U n 1+S n *U n 2...S n *U n i = ∑ j i＝1 (S n *U n i) Where S n N is the emotion integer of the feedback statement; and U n N is the intention integer of the feedback statement. Alternatively, selecting a subset of HMI changes may include filtering the proposed HMI changes based on a weighted score of the feedback statement and a utility function that predicts the return on investment (ROI) for each of the proposed HMI changes.

[0010] For any of the disclosed systems, methods, and CRMs, the system controller may transcribe user feedback audio and video files contained in the feedback data and summarize the transcribed audio and video files to identify and retain key information within the feedback data (e.g., discarding non-key data). Alternatively, generating an interaction sequence script may include: a BERT language model summarizing each HMI feature in the HMI system development metric set to include a feature objective, a feature process, and a feature result. In this example, the feature objective may define how to modify the HMI feature to enable its use by the user of the HMI device, the feature process may define how to reconfigure the HMI device to modify the HMI feature to achieve the feature objective, and the feature result defines the HMI feature reconfigured as a result when the feature process is implemented to achieve the feature objective. Generating an interaction sequence script may also include: a generative pre-trained transformer (GPT) model or a trained Baad transform language model (BARD) using the summary of HMI features in the HMI system development metric set to generate a sequence of steps for performing the HMI feature.

[0011] For any of the disclosed systems, methods, and CRMs, the automated design system may, for example, receive from HMI software developers and / or UX researchers a set of instructions to reorder the sequence of steps in an interactive sequence script used to perform HMI features on an HMI device. Once received, the system controller may reorder the sequence of steps for performing HMI features based on the instruction set. Alternatively, reconfiguring the HMI device may include: a code generator (LLM) modifying the source code of the main HMI device based on a subset of the HMI changes. While not inherently limited, the HMI device may be a digital instrument panel of a vehicle, an interactive telematics unit, an interactive touchscreen panel, or an HMI system throughout the vehicle. In this example, the system controller may deploy the modified source code to the digital IP / telematics unit / touchscreen panel, for example, before or after it is installed on the vehicle.

[0012] For any of the disclosed systems, methods, and CRMs, feedback data received from the group of clinical contributors may include audio, video, and / or biometric data from one or more end-user clinical participants. Feedback data may also include contributor feedback from a set of stakeholders (e.g., designers, program managers, engineers, etc.). As a further option, system requirements data may include a list of interactive features and corresponding characteristics by which users operate and control the HMI device. Additionally, system purpose data may include a list of design purposes by which users navigate and interact with the HMI device and use the interactive features. Source code data may include the HMI system architecture and accompanying software code used to provide the HMI device.

[0013] A method is provided for operating an automated design system for configuring a Human-Machine Interface (HMI) device, the method comprising: receiving, via a Large Language Model (LLM) module of the automated design system, a set of HMI system development metrics including system requirement data, system objective data, and / or source code data; generating, via the LLM module, an interactive sequence script containing a sequence of steps for performing HMI features on the HMI device using the received set of HMI system development metrics; configuring the HMI device via a system controller of the automated design system using the received set of HMI system development metrics; using the interactive sequence script to generate a feedback log containing feedback data received from a group of clinical contributors for testing the configured HMI device; generating a set of proposed HMI changes via a multimodal generative artificial intelligence (AI) module of the automated design system using the feedback data; and reconfiguring the HMI device via the system controller using a subset of HMI changes selected from the set of proposed HMI changes.

[0014] In the method, the feedback data in the feedback log includes multiple feedback statements, and generating the set of proposed HMI changes includes: deriving an intent integer for each of the feedback statements, the intent integer predicting the contributor intent of the contributor that produced the feedback statement; and deriving an emotion integer for each of the feedback statements, the emotion integer predicting the contributor emotion of the contributor that produced the feedback statement.

[0015] In the method, generating the set of proposed HMI changes further includes applying a weighted score W to each of the feedback statements. n The calculation is as follows: W n =S n *U n 1+S n *U n2...S n *U n i = ∑ j i＝1 S n *U n i Where S n The feedback statement is an integer of emotion n = 1 to j; and U n N is the integer N representing the intent of the feedback statement.

[0016] The method further includes selecting a subset of HMI changes by filtering the set of proposed HMI changes based on a weighted score of the feedback statement and a utility function that predicts the return on investment (ROI) for each of the proposed HMI changes.

[0017] The method further includes: transcribing the user feedback audio and video files contained in the feedback data; and summarizing the transcribed user feedback audio and video files to identify and retain key information within the feedback data.

[0018] In the method, generating the interactive sequence script includes: summarizing each of the multiple HMI features in the HMI system development metric set from the bidirectional encoder representation (BERT) language model of the transformer to include the feature purpose, feature process, and feature result.

[0019] In the method, the feature objective defines how to modify the HMI feature so that the user of the HMI device can use the HMI feature, the feature process defines how to reconfigure the HMI device to modify the HMI feature to achieve the feature objective, and the feature result defines the reconfigured HMI feature when the feature process is implemented to achieve the feature objective.

[0020] In the method, generating the interactive sequence script further includes: using a generative pre-trained transformer (GPT) model or a trained Baad transform language model (BARD) to generate a sequence of steps for performing the HMI features using a summary of HMI features in the HMI system development metric set.

[0021] The method further includes: receiving from a developer and / or researcher a set of instructions for reordering a sequence of steps for performing the HMI feature on the HMI device; and reordering the sequence of steps for performing the HMI feature based on the set of instructions via the system controller.

[0022] In the method, reconfiguring the HMI device includes: a code generator LLM modifying the source code based on a subset of HMI changes.

[0023] In the method, the HMI device includes a digital instrument panel and / or an interactive telematics unit for a vehicle, and the method further includes: the system controller deploying modified source code to the digital instrument panel and / or the interactive telematics unit.

[0024] In the method, the feedback data includes audio data, video data, and / or biometric data from one or more end-user contributors in a group of clinical contributors for testing.

[0025] In the method, the system requirement data includes a list of interactive features and corresponding characteristics by which a user operates the HMI device, the system purpose data includes a list of design purposes by which a user interacts with the HMI device and uses the interactive features, and the source code data includes the HMI system architecture for the HMI device.

[0026] An automated design system for configuring a Human-Machine Interface (HMI) device is provided, the automated design system comprising: a Large Language Model (LLM) module programmed to: receive a set of HMI system development metrics including system requirement data, system purpose data, and source code data; generate an interactive sequence script using the received set of HMI system development metrics, comprising a sequence of steps for performing HMI features on the HMI device; a system controller configured to: configure the HMI device using the received set of HMI system development metrics; and generate and store a feedback log containing feedback data received from a group of clinical contributors for testing the HMI features on the configured HMI device using the interactive sequence script; and a multimodal generative artificial intelligence (AI) module programmed to use the feedback data to generate a set of proposed HMI changes, wherein the system controller is further configured to reconfigure the HMI device using a subset of HMI changes selected from the set of proposed HMI changes.

[0027] A non-transient computer-readable medium (CRM) is provided to store instructions executable by one or more system controllers and / or one or more system control modules for an automated design system for configuring a human-machine interface (HMI) device. When executed, the instructions cause the automated design system to perform operations including: receiving, via a Large Language Model (LLM) module, a set of HMI system development metrics including system requirement data, system objective data, and / or source code data; generating, via the LLM module, an interactive sequence script containing a sequence of steps for performing HMI features on the HMI device using the received set of HMI system development metrics; configuring the HMI device using the received set of HMI system development metrics; using the interactive sequence script to generate a feedback log containing feedback data received from a group of clinical contributors to test the configured HMI device; generating a set of proposed HMI changes using the feedback data via a multimodal generative artificial intelligence (AI) module; and reconfiguring the HMI device using a subset of HMI changes selected from the set of proposed HMI changes.

[0028] In the non-transient CRM, the feedback data in the feedback log includes multiple feedback claims, and the generation of the proposed set of HMI changes includes: determining an intent integer for each of the feedback claims, the intent integer predicting the contributor intent of the contributor that generated the feedback claim; and determining an emotion integer for each of the feedback claims, the emotion integer predicting the contributor emotion of the contributor that generated the feedback claim.

[0029] In the non-transient CRM, generating the set of proposed HMI changes further includes applying a weighted score W for each of the feedback statements. n The calculation is as follows: W n =S n *U n 1+S n *U n 2...S n *U n i = ∑ j i＝1 S n *U n i Where S n The feedback statement is an integer of emotion n = 1 to j; and U n N is the integer N representing the intent of the feedback statement.

[0030] In the non-transient CRM, the instructions further enable the automated design system to select a subset of HMI changes by filtering the set of proposed HMI changes based on a weighted score of the feedback statement and a utility function that predicts the return on investment (ROI) for each of the proposed HMI changes.

[0031] In the non-transient CRM, generating the interactive sequence script includes summarizing each of the multiple HMI features in the HMI system development metric set from the bidirectional encoder representation (BERT) language model of the converter to include the feature purpose, feature process, and feature result.

[0032] In the non-transient CRM, the system requirement data includes a list of interactive features and corresponding characteristics by which the user operates the HMI device, the system purpose data includes a list of design purposes by which the user interacts with the HMI device and uses the interactive features, and the source code data includes the HMI system architecture for the HMI device.

[0033] The above description of the invention does not represent every embodiment or aspect of this disclosure. Rather, it provides only a summary of some of the novel concepts and features set forth herein. The above features and advantages, as well as other features and accompanying advantages of this disclosure, will become apparent from the following detailed description of the illustrated examples and representative modes used to implement this disclosure when taken in conjunction with the accompanying drawings and appended claims. Furthermore, this disclosure expressly includes any and all combinations and sub-combinations of the elements and features presented above and below. Attached Figure Description

[0034] Figure 1 This is a partial schematic side view of a representative motor vehicle having a network of controllers, HMI devices, sensing devices and communication devices in a vehicle, which can be used to practice aspects of this disclosure.

[0035] Figure 2 The diagram illustrates a flowchart of a representative control protocol and automated design system for configuring and deploying HMI devices according to aspects of this disclosure, wherein the control protocol may correspond to memory-stored instructions executable by a resident or remote microcontroller, central processing unit, control module, logic circuit, or other integrated circuit (IC) device or network (collectively, the “controller”).

[0036] Figure 3 The diagram illustrates a flowchart of a representative Large Language Model (LLM)-based control protocol for generating interactive sequence scripts for configuring and deploying HMI devices, according to aspects of this disclosure. This control protocol may correspond to memory-store instructions executable by a system controller.

[0037] Figure 4 The diagram illustrates a flowchart of a representative multimodal code generation control protocol for generating and testing weighted feedback for configuring and deploying HMI devices, according to aspects of this disclosure. This control protocol may correspond to memory storage instructions executable by a system controller.

[0038] This disclosure is subject to various modifications and alternatives, and some representative embodiments of this disclosure are shown by way of example in the accompanying drawings and will be described in detail herein. However, it should be understood that novel aspects of this disclosure are not limited to the specific forms illustrated in the drawings listed above. Rather, this disclosure covers all modifications, equivalents, combinations, arrangements, groupings, and alternatives that fall within the scope of this disclosure as covered, for example, by the appended claims. Detailed Implementation

[0039] This disclosure allows for embodiments in many different forms. While representative embodiments of this disclosure are provided as examples of the disclosed principles rather than as limitations on the broad aspects of this disclosure, these embodiments are shown in the accompanying drawings and will be described in detail herein. To this extent, elements and limitations described, for example, in the abstract, background, summary of the invention, description of the drawings, and detailed description sections but not expressly set forth in the claims should not be individually or collectively incorporated into the claims, by implication, inference, or otherwise. Furthermore, the use of terms such as “first,” “second,” “third,” etc., in the specification or claims is not in itself intended to establish series or numerical limitations; unless otherwise specifically stated, these designations may be used for convenience of referencing similar features in the specification and drawings and to demarcate between similar elements in the claims.

[0040] For the purposes of this disclosure, unless specifically waived: the singular includes the plural, and vice versa (e.g., the indefinite articles “a” and “one” should generally be understood to mean “one or more”); the words “and” and “or” should be both conjunctive and disjunctive; the words “any” and “all” should both mean “any and all”; and the words “including,” “contains,” “comprising,” “having,” etc., should all mean “including but not limited to.” Furthermore, in this document, similar words such as “about,” “almost,” “basically,” “generally,” “roughly,” etc., may be used to indicate examples such as “at,” “near,” or “almost at,” or “within 0-5% of,” or “within acceptable manufacturing tolerances,” or any logical combination thereof.

[0041] Now referring to the accompanying drawings, which are found throughout several views, similar reference numerals indicate similar features. Figure 1A representative motor vehicle is illustrated herein, generally designated at point 10 and depicted herein as a sedan-style electric vehicle for the purposes of discussion. The illustrated vehicle 10—also referred to herein simply as a "motor vehicle" or "vehicle"—is merely an exemplary application from which aspects of this disclosure can be practiced. Similarly, the implementation of the present concept for a central control panel telematics unit of a vehicle should be understood as a non-limiting implementation of the disclosed features. Thus, it should be understood that aspects and features of this disclosure can be applied to HMI devices in a wide variety of vehicles, incorporated into any logically related type of motor vehicle, and implemented for both motor vehicle and non-motor vehicle applications. Furthermore, only selected components of the motor vehicle and telematics system are shown and described in detail herein. However, the vehicles and systems discussed below may include numerous additional and alternative features for implementing the various methods and functions of this disclosure, as well as other available peripheral hardware.

[0042] Figure 1 The representative vehicle 10 is initially equipped with a vehicle telecommunications and information (“telematics processing”) unit 14, which communicates with a remotely located cloud computing hosting service 24 (e.g., via cellular networks, satellite services, wireless-enabled modems, etc.) Wireless communication. As a non-limiting example, Figure 1 Some of the other vehicle hardware components 16 shown in the overall diagram include an electronic video display device 18, a microphone 28, one or more audio speakers 30, and various user input control devices 32 (e.g., buttons, knobs, pedals, switches, touchpads, touchscreens, etc.). These hardware components 16 partially function as a human-machine interface (HMI), enabling the user to communicate with the telematics unit 14 and other components residing in and remote from the vehicle 10. The microphone 28, for example, provides the passenger with a means of inputting verbal commands; the vehicle 10 may be equipped with an embedded voice processing unit that utilizes audio filtering, editing, and analysis modules. Conversely, the speakers 30 provide audible output to the vehicle passenger and may be a separate speaker dedicated to the telematics unit 14 or part of an audio system 22. The audio system 22 is connected to a network connection interface 34 and an audio bus 20 to receive analog information via one or more speaker components, thereby presenting it as sound.

[0043] Coupling communicatively to the telematics unit 14 is a network interface 34, suitable examples of which include a twisted-pair / fiber Ethernet switch, a parallel / serial communication bus, a local area network (LAN) interface, a controller area network (CAN) interface, and so on. The network interface 34 enables the vehicle hardware 16 to send and receive signals with each other and with various systems located both on and off the vehicle body 12. This allows the vehicle 10 to perform a wide variety of vehicle functions, such as modulating powertrain output, activating friction or regenerative braking systems, controlling vehicle steering, and other automated functions. For example, the telematics unit 14 can exchange signals with the powertrain control module (PCM) 52, the advanced driver assistance system (ADAS) module 54, the electronic battery control module (EBCM) 56, the steering control module (SCM) 58, the brake system control module (BSCM) 60, and various other vehicle ECUs (such as the transmission control module (TCM), engine control module (ECM), sensor system interface module (SSIM), etc.).

[0044] Continue to refer to Figure 1 The telematics unit 14 is a hybrid of an onboard computing device and services provided individually and through communication with other networked devices. The telematics unit 14 may generally consist of one or more processors 40, each of which may be embodied as a discrete microprocessor, an application-specific integrated circuit (ASIC), or a dedicated control module. Centralized vehicle control may be provided by a central processing unit (CPU) 36 operatively coupled to a real-time clock (RTC) 42 and one or more electronic storage devices 38, each of which may take the form of a CD-ROM, disk, IC device, solid-state drive (SSD) memory, hard disk drive (HDD) memory, flash memory, semiconductor memory (e.g., various types of RAM or ROM), etc.

[0045] Long-range communication (LRC) capability with remote external devices can be provided via one or more of the following: cellular chipsets / components, navigation and location chipsets / components (e.g., GPS transceivers), or wireless modems, all of which are collectively represented at 44. Short-range communication (SRC) capability can be provided via short-range communication (SRC) devices 46 (e.g., The communication device described above can provide short-range wireless connectivity via a unit or near-field communication (NFC) transceiver, a dedicated short-range communication (DSRC) component 48, and / or dual antennas 50. The communication device can provide data exchange as part of periodic broadcasts in vehicle-to-vehicle (V2V) communication networks or vehicle-to-everything (V2X) communication systems (e.g., vehicle-to-infrastructure (V2I), vehicle-to-pedestrian (V2P), vehicle-to-device (V2D), vehicle-to-cloud (V2C), etc.).

[0046] CPU 36 receives sensor data from one or more sensing devices using technologies such as photoelectric detection, radar, laser, ultrasound, optics, infrared, or other suitable techniques, including short-range communication technologies (e.g., DSRC) or ultra-wideband (UWB) radio technology, for performing controller automation (AV / ADAS) driving operations or vehicle navigation services. According to the illustrated example, vehicle 10 may be equipped with one or more digital cameras 62, one or more distance sensors 64, one or more vehicle speed sensors 66, one or more vehicle dynamics sensors 68, and any necessary filtering, classification, fusion, and analysis hardware and software for processing the raw sensor data. The vehicle speed sensors (one or more) 66 may have properties such as single-axis or tri-axis accelerometers, angular rate sensors, inclinometers, steering wheel angle sensors, brake sensors, etc., for detecting longitudinal and lateral acceleration, yaw, roll and / or pitch rates, steering angle, and other dynamic parameters. The type, placement, number, and interoperability of the distributed array of sensors in the vehicle can be individually or collectively adapted to a given vehicle platform for achieving the desired level of automated vehicle operation.

[0047] To propel the motor vehicle 10, the electrified powertrain is operable to generate traction torque and deliver that traction torque to one or more of the vehicle's drive wheels 26. The powertrain in Figure 1The term refers to a rechargeable energy storage system (RESS) operatively connected to a traction motor (M) 78, which may have the characteristics of a chassis-mounted traction battery pack 70. The traction battery pack 70 generally consists of one or more battery modules 72, each containing a cluster of battery cells 74, such as pouch, can, or cylindrical lithium, zinc, or silicone cells. One or more motors (such as traction motor / generator (M) units 78) draw power from the battery pack 70 and optionally deliver power to it. A power inverter module (PIM) 80 electrically connects the battery pack 70 to the motor(s) 78 and modulates the current flow therebetween. The battery pack 70 may include an integrated package of electronics for implementing module-level management, cell sensing, etc., such as a wirelessly enabled cell monitoring unit (CMU) 76.

[0048] The following discusses an automated design system with accompanying control logic for configuring and controlling HMI devices using generative AI-based methods for automating manual and repetitive tasks, LLM-based methods for generating interaction sequences for stakeholders, and GPT or BARD-based methods for providing "on-the-spot" changes in prototype software implementations. As an example, a system based on a multimodal generative AI model uses multiple information sources to generate interaction sequences, including screen and system specifications, source code, prototype implementation requirements, user clinical objectives, etc. The multimodal generative AI system uses multiple methods to observe and record contributor feedback using various audio, visual, and biometric sources, including verbal and nonverbal feedback. Once filtered, preprocessed, and recorded, the system summarizes the contributor feedback data, extracts key insights from the summarized data, derives contributor intent and sentiment from the data, and logs the feedback data. The system then analyzes the feedback log to generate change requests and prioritize changes, such as based on impact frequency, rules and high ROI, ease of implementation, etc. The code generation AI model processes the proposed HMI changes and exports the source code changes to implement them. The exported source code is then deployed to HMI devices with accompanying software and firmware updates to improve device functionality.

[0049] Next reference Figures 2-4 The flowchart, according to aspects of this disclosure, generally describes at points 200, 300, and 400, methods for configuring and deploying HMI devices (such as, Figure 1 An improved control protocol and automated design system for the vehicle telematics unit 14. Figures 2-4 Some or all of the operations illustrated in the diagram and described in further detail below can be represented, for example, in primary or secondary or remote memory (e.g., Figure 1The algorithms corresponding to non-transient processor-executable instructions are stored in one or more vehicle-residing memory devices 38 and / or remote cloud computing service databases 24. These instructions may be provided by, for example, electronic controllers, processing units, dedicated control modules, logic circuits, or other modules or devices or networks of controllers / modules / devices (e.g., Figure 1 The CPU 36 and / or cloud host service 24BO server-type computer station) of the vehicle executes to perform any or all of the functions described above and below in connection with the disclosed concept. It should be understood that the order of execution of the illustrated operation boxes can be changed, additional operation boxes can be added, and some of the operations described herein can be modified, combined, or eliminated.

[0050] Method 200 can be used Figure 2 Beginning at terminal block 201, the routine contains memory-stored processor-executable instructions for initializing a system control protocol for HMI software development using generative AI techniques. This routine can be real-time, near real-time, continuous, systematic, sporadic, and / or at predefined time intervals (e.g., at...). Figure 1 The terminal box 201 is initialized every 10 or 100 milliseconds during the use of the telematics unit 14. Alternatively, the terminal box 201 may be initialized in response to user command prompts (e.g., input control via telematics 14), vehicle controller prompts (e.g., from CPU 36), or broadcast prompts received from a centralized back office (BO) vehicle service system (e.g., from cloud host service 24). As a non-limiting example, HMI developers or UX designers may invoke method 200 from a BO server-class computer station during the initial design cycle and lifespan of the main HMI device. Figure 2 When some or all of the control operations presented in the method are completed, method 200 may proceed to the end terminal box 223 and temporarily terminate, or alternatively may loop back to terminal box 201 and run in a continuous loop.

[0051] Proceeding from terminal box 201 to system asset data input box 203, method 200 may receive as input a set of HMI system development assets / metrics containing details for developing new or updated features on the HMI device. According to the illustrated example, a group of interested stakeholders (e.g., HMI developers, UX designers, leadership, etc.) provides a set of system requirements 202, a set of system objectives 204, and source code 206 for the subject HMI device. System requirements 202 data may include a list of new or updated interactive features with corresponding characteristic features by which end users operate the subject HMI device (e.g., the total number of user-selectable icons; the naming of the function of each icon; the location, design, and size of each icon; the background / foreground color of the interface; etc.). In contrast, system objectives 204 data may include a list of design objectives by which users interact with the HMI device and use the new / updated interactive features (e.g., where users find specific features within the HMI system architecture; how users select and / or control each feature; how the HMI communicates the functionality of the features, etc.). HMI system source code 206 data may include an HMI system architecture with accompanying software code for providing interactive HMI device features.

[0052] Upon receiving the required HMI system development metrics, method 200 can execute the Large Language Model subroutine box 205 and feed the received data into a generative AI-based LLM module. The LLM module can be trained via unsupervised deep neural network (DNN) learning techniques from the relevant HMI development dataset to generate an interaction sequence script using the HMI system development metrics as multimodal input and natural language prompts, as indicated in the interaction script document output box 207. The interaction sequence script can contain a guided usability testing script that provides an overview and background information for interacting with the subject HMI device, general instructions for operating the HMI device, and a sequence of steps for performing one or more new / updated HMI features on the subject HMI device. The interaction sequence script can also provide a predefined set of feedback prompts for contributors to input when attempting to operate a specified interactive feature on the HMI device. The following... Figure 3 Additional information related to the generation of interactive sequence scripts is provided in the discussion.

[0053] While exporting the sequence scripts for interacting with the HMI device, method 200 can execute the HMI update subroutine box 209 and deploy the compiled source code to the test HMI device (e.g., a target platform in a simulator, test bench, or vehicle) based on HMI system development assets / metrics. For example, a resident system controller of an automated design system can use the set of HMI system development metrics received at box 203 to configure an in-cabin telematics unit or a test bench telematics simulator. Groups of test clinical contributors—end-user clinical participants, stakeholders, leaders, etc.—use the interaction sequence scripts generated at box 207 to test the HMI device configured at box 209. At the interaction and feedback data input box 211, these test clinical contributors can generate feedback data detailing each contributor's experience when interacting with the HMI device to use one or more new / updated HMI features. Continuing to the feedback log database box 213, method 200 can generate and store a feedback log containing the contributor feedback data generated at box 211. The feedback data may include audio data, video data, biometric data, time data, and observation data for one or more contributors in the testing clinical group.

[0054] After generating and storing contributor feedback, method 200 continues to process and evaluate user feedback to extract actionable insights for reconfiguring and controlling the principal HMI device. For example, method 200 may execute a multimodal model subroutine box 215, whereby a multimodal generative AI module derives a set of proposed HMI changes based on the feedback data. The multimodal module can process a variety of different data sources and formats (e.g., audio, video, text, sensor data, etc.) to form insights with the proposed change requirements. Developers / UX researchers can then examine the proposed HMI changes and determine which changes are "core" and therefore should be implemented. For example, an accept / reject subroutine box 217 can provide executable code whereby a verified developer and / or researcher inputs a set of instructions to perform the following operations: (1) accept the proposed changes; (2) reject the proposed changes; (3) modify the proposed changes; (4) shelve the proposed changes for future evaluation; and / or (5) reorder the sequence of steps for implementing HMI features on the HMI device. The decision to accept / reject / prioritize a proposed change can be based on a utility function that indicates the return on investment (ROI) of making that change. It can depend on feedback log analysis, from which the system infers how many contributors expect the specific change, how strong that preference is, etc. Upon receiving verified instructions from developers / researchers, method 200 can either reject and discard or alternatively accept and implement any or all of their proposed modifications (e.g., reordering the sequence of steps for executing the new / updated HMI features based on the instruction set).

[0055] Continue to refer to Figure 2 Method 200 can proceed to source code generation subroutine box 219 and employ a code interpreter and generative AI-based deployment (CIAID) module to automate changes to the prototype software to implement one or more of the proposed HMI changes. The CIAID module can use a code generation model (e.g., a Megatron-Turing Natural Language Generation (MT-NLG) model or other monolithic converter language models) to translate change requirements into code changes for deployment with the host HMI device. The HMI source code can be updated at source code documentation output box 221, simultaneously deploying the HMI source code to the host HMI device. In doing so, the automated design system reconfigures the HMI device using a selected subset of HMI changes chosen from the initial set of proposed HMI changes. For example, the system controller can perform a final code check and then submit and push the code changes to the telematics unit, digital instrument panel (IP), and / or interactive touchscreen panel in the vehicle; the HMI system can be restarted to allow the device driver and any relevant HMI files to be updated. The following... Figure 4The discussion provides additional information related to processing user feedback to extract actionable insights and pushing updated source code for HMI development. At this point, method 200 may temporarily terminate at the end terminal box 223, or may loop back to the start terminal box 201.

[0056] Figure 3 An LLM-based control method 300 for generating interactive sequence scripts to facilitate the configuration and control of HMI devices is presented. Method 300 can begin at a system metric file input box 301 and receive a set of system requirement documents and feature files (e.g., system requirements 202 and system objectives 204 in text file format). The feature files can be text files storing features to be tested, scenarios, and feature descriptions, provided with the extension ".feature" (e.g., Gherkin format). Upon receiving the requirement document and associated feature files, method 300 can then execute a BERT language model subroutine box 303 and feed the received documents and files into a retrained Bidirectional Encoder Representation (BERT) language module from the transformer. The BERT module can be: an open-source machine learning framework for Natural Language Processing (NLP) designed to understand ambiguous language in text files by using surrounding text to build context and meaning. The BERT module can be configured as: a deep learning model where each output element is connected to each input element, with weights dynamically computed between them based on their connections.

[0057] Method 300 can proceed from subroutine block 303 to feature summary file output block 305, where the LLM model formats the information contained in the requirement document and feature file by: (1) purpose; (2) process; and (3) result. For example, the BERT module can summarize each new or updated HMI feature in the set of feature files to include the feature purpose, feature process, and feature result. The “purpose” of an HMI feature can define how the feature is modified so that users of the HMI device can use the HMI feature (e.g., modify the descriptor and increase the font size). In contrast, the “process” of an HMI feature can define how the HMI device is reconfigured to modify the HMI feature to achieve the corresponding feature purpose (e.g., redraw the HMI screen layout so that a request for a larger font size with a modified descriptor is accommodated). The “result” of a feature can define the HMI feature reconfigured as a result when the feature process is implemented to achieve the corresponding feature purpose. At source code file input block 307, the automated design system can receive existing HMI system source code for the subject HMI device (e.g., Figure 2 The source code (206) is a text file.

[0058] After formatting the requirements document and feature files, method 300 can execute HMI update subroutine box 309 and deploy the formatted HMI system development metrics to the test HMI device. According to the illustrated example, a generative pre-trained transformor (GPT) model or a trained Baad transform language model (BARD) LLM model, after making one or more changes to the HMI source code, utilizes the deployment and implementation process of the feature changes to execute user interface (UI) automation tools (e.g., the Appium mobile automation testing tool or the Robotics Framework automation framework). The large language model takes the feature summary and system source code as input and generates a sequence of steps for interacting with each HMI feature and executing each HMI feature on the HMI device. Figure 2 In the context of subroutine block 209, Figure 3 Method 300 can simultaneously execute the HMI update subroutine block 311 and deploy HMI system development metrics to the test HMI device.

[0059] During the execution of subroutines 309 and 311, the LLM model can simulate the operation and control of the HMI system by interacting with the UI API of the HMI device according to the LLM / HMI interaction script. In response, the HMI system can output a result script detailing the interaction as feedback to the LLM model. The large language model takes the feature summary and system source code along with the feedback from the HMI system as input and uses UI automation tools to interact with the HMI application to generate an updated sequence of steps for interacting with and executing each HMI feature. Method 300 can output the updated step sequence to stakeholders at the interaction sequence file output box 313. Developers, UX researchers, or verified users can then analyze the updated step sequence to correct / accept / reject / reorder the steps generated by the LLM for each feature, as indicated at the sequence correction subroutine box 315. A revised "final" interaction sequence script with selected features to be tested based on predefined purposes can be created and output at the interaction script document output box 317, for example, with Figure 2 The document output box 207 is synchronized.

[0060] Figure 4A multimodal code generation control method 400 is presented for generating and testing weighted contributor intent and emotional feedback to facilitate the configuration and control of HMI devices. Method 400 may begin at a biofeedback sensor data box 401 and a contributor feedback data input box 403 to receive, respectively, a set of biometric data from one or more biometric sensors and a set of audio and / or video data from one or more sound sensors and / or optical sensors as input. Once received, method 400 may then execute a contributor intent DNN model subroutine box 405, where a retrained Long Short-Term Memory (LSTM) DNN model can use biometric data files, audio / video data files, clinical supervision observation data files, etc., to derive a numerical representation of the contributor's intent when testing new / updated features on the configured HMI device. Proceeding to the intent integer text output box 407, for example, an automated design system may derive an intent integer for each feedback statement provided by a clinical contributor. This intent integer can predict the intent of the contributor who generated the feedback statement when testing the configured HMI device. Intent indicators can be integer values ​​representing a user's intent, [-1: negative intent, 0: neutral, 1: positive intent]. Alternatively, intent indicators can be decimal values ​​ranging from 0 (no expectation) to 1 (maximum expectation) to show how much an individual wants the change he or she describes.

[0061] While automating the prediction of contributor intent, method 400 can execute the text ranking subroutine box 409 and generate a text transcription of the audio / video data received at the contributor feedback data input box 403. For example, a graph-based TextRank (text summarization) natural language processing (NLP) algorithm can be used to extract, transcribe, and summarize contributor feedback transcriptions. After transcribing the user feedback audio and video files contained in the HMI system feedback data, the automated design system can summarize the transcribed feedback files to identify and retain key information within the feedback data. Simultaneously, the design system can automatically de-prioritize or discard non-key information. A summary of the feedback transcription can be output at the transcription summary document output box 411.

[0062] Upon receiving the summarized feedback transcription, method 400 may execute language transformer model subroutine box 413 and feed the summarized transcription into the BERT language model. The pre-trained BERT transformer model extracts the summary statement from the received transcription and derives a numerical representation of the contributor's emotion and tension as new / updated features are tested on the configured HMI device. Proceeding to emotion integer text output box 415, for example, the automated design system may derive an emotion integer for each feedback statement provided by a clinical contributor. This emotion integer can predict the emotion of the contributor who generated the feedback statement when testing the configured HMI device. The emotion indicator may be an integer value representing the user's intent, [-1: negative emotion, 0: neutral, 1: positive emotion].

[0063] After deriving scalar values ​​for predicting user intent and emotion, method 400 may execute a weighted feedback subroutine box 417 to generate a weighted score for each feedback statement by combining integers of emotion and intent from clinical contributors for each statement (the proposed HMI change). As an example, and not a limitation, the weighted score W may be calculated for each feedback statement. n The calculation is as follows: W n =S n *U n 1+S n *U n 2...S n *U n i =∑ j i＝1 (S n *U n i) Where S n It refers to the sentiment integers n = 1 to j for the feedback statement; and U n i is an integer n representing the intent of the feedback statement, such as that derived from the audio / video analysis of the contributor who expressed the statement. W n Therefore, it can be represented as a weighted sum of the emotions and tensions of each statement, using the expressions of different users analyzed from various feedback modalities.

[0064] Continue to refer to Figure 4Method 400 can proceed to feedback filtering process box 419 and select a subset of HMI changes to be implemented by filtering the feedback statements (proposed HMI changes) based on a weighted score calculated at subroutine box 417. As mentioned above in the discussion of accept / reject subroutine box 217, the decision to accept / reject / modify / reorder / prioritize the proposed changes can be based on a utility function that predicts the return on investment (ROI) for making the proposed HMI change. Code generation subroutine box 421 can implement a code generation LLM model (e.g., MT-NLG model) to systematically evaluate the original version of the source code and the filtered change requests to generate a revised version of the source code to implement the subset of changes. At test and deployment process box 423, the automated design system can test and debug the revised source code adapted to utilize the subset of HMI changes; if satisfied, the system can release a “new build” of the reconfigured HMI device.

[0065] In some embodiments, aspects of this disclosure may be implemented by computer-executable instructions (such as program modules), generally referred to as a software application or application program executed by any of the controllers or variations thereof described herein. In non-limiting examples, the software may include routines, programs, objects, components, and data structures that perform specific tasks or implement specific data types. The software may form interfaces to allow a computer to react based on a source of input. The software may also cooperate with other code segments to initiate various tasks in response to data received from a source in conjunction with received data. The software may be stored on any of a variety of memory media, such as CD-ROM, disk, and semiconductor memory (e.g., various types of RAM or ROM).

[0066] Furthermore, aspects of this disclosure can be practiced using a variety of computer systems and computer network configurations, including multiprocessor systems, microprocessor-based or programmable consumer electronics, microcomputers, mainframe computers, and so on. Additionally, aspects of this disclosure can be practiced in distributed computing environments where tasks are executed by resident and remote processing devices linked via communication networks. In distributed computing environments, program modules can reside on both local and remote computer storage media, including memory storage devices. Therefore, aspects of this disclosure can be implemented by combining various hardware, software, or combinations thereof in computer systems or other processing systems.

[0067] Any of the methods described herein may include machine-readable instructions executable by (a) a processor, (b) a controller, and / or (c) any other suitable processing device. Any algorithm, software, control logic, protocol, or method disclosed herein may be embodied as software stored on a tangible medium such as, for example, flash memory, solid-state drive (SSD) memory, hard disk drive (HDD) memory, CD-ROM, digital versatile disc (DVD), or other memory devices. The entire algorithm, control logic, protocol, or method and / or portions thereof may alternatively be executed by a device other than a controller, and / or embodied in firmware or dedicated hardware (e.g., implemented by application-specific integrated circuits (ASICs), programmable logic devices (PLDs), field-programmable logic devices (FPLDs), discrete logic, etc.). Furthermore, while specific algorithms may be described with reference to the flowcharts and / or workflow diagrams depicted herein, many other methods for implementing the example machine-readable instructions may be used alternatively.

[0068] Aspects of the invention have been described in detail with reference to the illustrated embodiments; however, those skilled in the art will recognize that many modifications can be made thereto without departing from the scope of this disclosure. This disclosure is not limited to the precise construction and composition disclosed herein; any and all modifications, alterations, and variations apparent from the foregoing description are within the scope of this disclosure as defined by the appended claims. Furthermore, this concept explicitly includes any and all combinations and sub-combinations of the foregoing elements and features.

Claims

1. A method for operating an automated design system for configuring human-machine interface (HMI) devices, the method comprising: The automated design system receives a set of HMI system development metrics, including system requirement data, system objective data, and / or source code data, via its Large Language Model (LLM) module. The LLM module uses the received set of HMI system development metrics to generate an interactive sequence script containing a sequence of steps for performing HMI features on the HMI device. The system controller of the automated design system uses the received set of HMI system development metrics to configure the HMI device; The interactive sequence script is used to generate a feedback log containing feedback data received from a group of clinical contributors to test the configured HMI device. The set of proposed HMI changes is generated using the feedback data via the multimodal generative artificial intelligence (AI) module of the automated design system. as well as The HMI device is reconfigured via the system controller using a subset of HMI changes selected from the proposed set of HMI changes.

2. The method of claim 1, wherein the feedback data in the feedback log includes a plurality of feedback statements, and wherein generating the set of proposed HMI changes includes: Derive an intent integer for each of the feedback statements, the intent integers predicting the contributor intent of the contributors who generated the feedback statements; as well as Derive a sentiment integer for each of the feedback statements, the sentiment integers predicting the contributor sentiment of the contributors who produced the feedback statements.

3. The method of claim 1, wherein generating the proposed set of HMI changes further comprises weighting the scores W for each of the feedback statements. n The calculation is as follows: W n =S n *U n 1+S n *U n 2...S n *U n i=∑ j i＝1 S n *U n i Where S n The feedback statement is an integer of emotion n = 1 to j; and U n N is the integer N representing the intent of the feedback statement.

4. The method of claim 3, further comprising: A subset of HMI changes is selected by filtering the set of proposed HMI changes based on a weighted score of the feedback statement and a utility function that predicts the return on investment (ROI) for each of the proposed HMI changes.

5. The method of claim 3, further comprising: Transcribe the user feedback audio and video files contained in the feedback data; as well as Transcribed user feedback audio and video files are summarized to identify and retain key information within the feedback data.

6. The method of claim 1, wherein generating the interaction sequence script comprises: The Bidirectional Encoder Representation (BERT) language model from the converter summarizes each of the multiple HMI features in the set of HMI system development metrics to include the feature purpose, feature process, and feature result.

7. The method of claim 6, wherein the feature objective defines how to modify the HMI feature so that a user of the HMI device can use the HMI feature, the feature process defines how to reconfigure the HMI device to modify the HMI feature to achieve the feature objective, and the feature result defines the reconfigured HMI feature when the feature process is implemented to achieve the feature objective.

8. The method of claim 6, wherein generating the interaction sequence script further comprises: Generative pre-trained transformer (GPT) models or trained Baad transform language models (BARD) use a summary of HMI features in the HMI system development metric set to generate a sequence of steps for executing the HMI features.

9. The method of claim 1, further comprising: Receive a set of instructions from the developer and / or researcher to reorder the sequence of steps for performing the HMI features on the HMI device; as well as The sequence of steps for executing the HMI feature is reordered based on the instruction set via the system controller.

10. The method of claim 1, wherein reconfiguring the HMI device comprises: The code generator LLM modifies the source code based on a subset of changes made by the HMI.

Images