SYSTEM AND METHOD FOR CREATING AN AD-HOC ACTIVE USER INTERFACE AND USER EXPERIENCE

Abstract

A system for generating an ad-hoc, actionable user interface (UI) and user experience (UX) in real time comprises one or more data sources, a visual display device for presenting actionable visual screens, and an input device for receiving external triggers. The system also includes an electronic controller programmed with UI and UX algorithms. The UI algorithm receives a trigger event from the input device and data from the data source(s) that is contextually related to the received trigger event. The UI algorithm also determines an expected user request in response to the received trigger event and contextual data. The UI algorithm further generates one or more actionable visual screens with one or more user prompts corresponding to the determined expected user request.The UX algorithm selects an actionable visual screen from the generated screen(s) and displays it on the visual display device to create the ad-hoc actionable UI and UX.

Description

INTRODUCTION

[0001] The present disclosure relates to a system and a method for generating an ad-hoc user interface and user experience.

[0002] In general, a user interface (UI) is the means by which a computer system user and the computer system interact. Specifically, a user interface is a point of human-computer interaction and communication within a device. Such an interface typically involves the use of input devices and software, such as display screens, keyboards, a computer mouse, and the appearance of a desktop.

[0003] A user interface also defines how a user interacts with a computer application or website, using visual and audio elements such as fonts, icons, buttons, animations, and sounds. The goal of such human-computer interaction is to enable the practical operation and control of the machine from the human end, while the machine simultaneously provides feedback to support the operator's decision-making process. An effective user interface follows design principles that allow a user to navigate the interface and easily use it for its intended purpose. DESCRIPTION

[0004] A system for generating an ad-hoc, actionable user interface (UI) and user experience (UX) in real time comprises at least one data source, a visual display device configured to present actionable visual screens, and an input device configured to receive external triggers. The system also includes an electronic controller communicating with the data source(s), the visual display device, and the input device, and programmed with a UI algorithm and a UX algorithm. The UI algorithm is configured to receive a trigger event from the input device and data from the data source(s) that is contextually related to the received trigger event.The UI algorithm is also trained to determine an expected user request in response to the received trigger event and contextual data received from the data source(s). The UI algorithm is further trained to generate at least one actionable visual screen, each containing one or more user prompts corresponding to the determined expected user request. The UX algorithm is trained to select an actionable visual screen from the at least one generated actionable visual screen and display it on the visual display device, thereby generating the ad-hoc actionable UI and UX.

[0005] Each of the visual display devices, input devices, and electronic controls can be a component of a vehicle infotainment system.

[0006] The data source(s) can include at least one from a vehicle sensor, an information technology (IT) cloud server and the World Wide Web (WWW).

[0007] The UI algorithm can be trained to determine, i.e., predict, the expected user request via an artificial intelligence (AI) agent.

[0008] The UX algorithm can use the Markov Decision Process (MDP) to determine the expected user request.

[0009] The expected user request can be determined by optimizing a reward function defined using the trigger event and contextual data.

[0010] The UI algorithm can additionally be trained to construct, in real time, software code that is trained to select a design and layout of the actionable visual screen(s).

[0011] The constructed software code may include access to a UI formulator trained to generate the actionable visual screen(s).

[0012] The UI formulator can include access to a generative adversarial network (GAN). The GAN can include a design origin function trained to generate a set of alternative actionable visual screens. The GAN can also include a discriminator function trained to filter the generated set of alternative actionable visual screens using criteria defined by the specific expected user request.

[0013] The GAN can include selective access to a database of designs and layouts of actionable visual screens and a library of components for the alternative actionable UIs.

[0014] A method for creating an ad-hoc actionable user interface (UI) and a real-time user experience (UX) is also revealed.

[0015] The above-mentioned features and advantages, as well as other features and advantages of the present disclosure, will be readily apparent from the following detailed description of the embodiment(s) and the best way(s) for carrying out the described disclosure in conjunction with the attached drawings and attached claims. BRIEF DESCRIPTION OF THE DRAWINGS Fig. Figure 1 is a schematic representation of a vehicle comprising a visual display device and a system that uses an electronic control to generate an ad-hoc actionable user interface (UI) and user experience (UX) in real time as disclosed. Fig. Figure 2 is a schematic view of a representative actionable visual screen located on the in Fig. 1 is shown as a visual display device. Fig. Figure 3 is a schematic representation of a system layout for generating the ad-hoc actionable UI and UX, which is in Fig. 1 are shown, according to the revelation. Fig. Figure 4 is a schematic representation of the Markov Decision Process (MDP) as applied to the generation of the ad-hoc UI and UX used by a UX algorithm programmed into the electronic control system according to the disclosure. Fig. Figure 5 is a schematic representation of a representative generative tree derived from the one in Fig. The UX algorithm shown in section 4 is generated using MDP as disclosed. Fig. Figure 6 is a schematic representation of the information flow between individual components in the system for generating the ad-hoc actionable UI and UX, which is in Fig. Figure 1 shows the results generated by a UI algorithm programmed into the electronic control system according to the disclosure. Fig. Figure 7 is a flowchart of a process that is designed to create an ad-hoc actionable UI and UX that is in Fig. 1- Fig. 6 are shown, to be generated in real time according to the revelation. DETAILED DESCRIPTION

[0016] The embodiments of the present disclosure, as described herein, are intended to serve as examples. Other embodiments may take different and alternative forms. Furthermore, the drawings are generally schematic and not necessarily to scale. Some features may be exaggerated or minimized to show details of certain components. Therefore, the specific structural and functional details disclosed herein are not to be interpreted as limiting, but merely as a representative basis for teaching a person skilled in the art to use the present disclosure in various ways.

[0017] Certain terminology may be used in the following description for reference purposes only and is therefore not intended to be restrictive. For example, terms such as "above" and "below" refer to directions in the drawings being referenced. Terms such as "front," "back," "front," "back," "left," "right," "back," "sideways," "up," "down," "above," and "below," etc., describe the orientation and / or location of sections of the components or elements within a consistent but arbitrary frame of reference, as clarified by reference to the text and the associated drawings describing the components or elements under discussion.

[0018] Furthermore, terms such as "first," "second," "third," and so on may be used to describe separate components. Such terminology may include the words expressly mentioned above, derivatives thereof, and words of similar meaning, and is used descriptively for the figures and does not constitute any limitation of the scope of disclosure as defined by the appended claims. Moreover, the teachings herein may be described in terms of functional and / or logical block components and / or various processing steps. It should be recognized that such block components may include a number of hardware, software, and / or firmware components configured to perform the specified functions.

[0019] Referring to the drawings, where the same reference numerals in the different views refer to the same components, states Fig. Figure 1 schematically depicts a vehicle 10. The vehicle 10 is generally characterized by a vehicle body 12, which is surrounded by an external environment 14. The vehicle body 12 defines a vehicle interior or vehicle cabin 16, which is designed to accommodate a driver and one or more passengers, for example in a generally seated position, and a vehicle infotainment system 18. With further reference to Fig. System 20, operable from the vehicle cabin 16, is designed to facilitate the real-time generation of an ad-hoc, actionable user interface (UI) and user experience (UX) 22 (discussed in detail below). Although System 20 can be implemented in a variety of environments and settings, the description of the object system will henceforth be primarily in relation to the motor vehicle 10. With respect to the vehicle 10's environment, the driver and the passenger(s) can be the intended users of System 20.

[0020] System 20 includes data sources such as vehicle sensors 24-1, an information technology (IT) cloud server 24-2, and the World Wide Web (WWW) 24-3, communicating wirelessly with the vehicle 10 via a cellular or wireless fidelity (Wi-Fi) connection, and a global positioning satellite (GPS) 25 for localization and positioning. System 20 also includes a visual display device 26, shown to be arranged within the cabin 16, configured to display information using changeable or selectable and actionable visual screens 28. With regard to the visual display screens 28, the term "actionable" refers to user-interactive screens configured to elicit further actions from the system user and generate a resulting system response.The vehicle's in-vehicle visual display device 26 can be part of the vehicle's infotainment system 18, with connections to a navigation system and GPS antenna, and communicating with vehicle cameras and other sensors. Alternatively, the visual display device 26 can be a mobile device, such as a mobile phone or a laptop running a mobile software application. The system 20 also includes an input device 30, such as a user-operated keyboard, selector, mouse, or vehicle bus connector. The input device 30 is generally configured to receive external triggers, such as user requests, inputs or selections, signals corresponding to vehicle operating parameters, fault codes, etc.

[0021] System 20 further includes an electronic control unit 32 communicating with the data source(s), e.g., 24-1, 24-2, 24-3, the visual display device 26, and the input device 30. In the context of the vehicle 10, each of the visual display device 26, the input device 30, and the electronic control unit 32 can be a component of the vehicle infotainment system 18. As part of System 20, the vehicle infotainment system 18 additionally includes a GPS antenna and a receiver (for communication with the GPS 25) and is configured to access various applications and maps, either online or on board the vehicle, i.e., programmed into the electronic control unit 32. The electronic control unit 32 can be a central processing unit (CPU) configured to receive data signals from various vehicle sensors and to control the operation of vehicle systems.The electronic control 32 includes a memory that is tangible and non-volatile. The control memory can be a writable medium involved in providing computer-readable data or process instructions. Such a medium can take many forms, including, but not limited to, non-volatile and volatile media.

[0022] Non-volatile media used by the electronic control unit 32 may include, for example, optical or magnetic disks and other persistent storage media. Volatile media for any memory of the control unit may include, for example, dynamic random-access memory (DRAM), which may constitute main memory. Such instructions may be transmitted through one or more transmission media, including coaxial cable, copper wire, and fiber optic cable, including the wires comprising a system bus coupled to the vehicle systems. The memory of the electronic control unit 32 may also include a flexible disk, a hard disk, magnetic tape, another magnetic medium, a CD-ROM, a DVD, another optical medium, etc.The electronic control 32 can be equipped with a high-speed primary clock, required analog-to-digital (A / D) circuits and / or digital-to-analog (D / A) circuits, input / output (I / O) circuits and devices, as well as suitable signal conditioning and / or buffer circuits.

[0023] Algorithms required or accessible by the electronic control unit 32 can be programmed in the control unit, stored in memory, and executed automatically to provide the required functionality. Specifically, the electronic control unit 32 is programmed with a UI algorithm 34A and a UX algorithm 34B to operate the system 20 and is tasked with generating an ad-hoc actionable user interface (UI) and user experience (UX) 22 in real time. In the context of the vehicle 10, the ad-hoc actionable UI and UX 22 generally take the form of a user-interactive display screen or flow, i.e., a chain of screens for the display device 26. In this context, the term "ad-hoc" defines a user interface or actionable visual screen that was not designed or constructed in advance.

[0024] The UI algorithm 34A is configured to receive a signal or code from the input device 30 indicating a trigger event 36, which may be, for example, a vehicle user input or selection or a vehicle system warning. The UI algorithm 34A is also configured to request and receive data 38 from the data source(s), e.g., 24-1, 24-2, 24-3, which is contextually related to the received trigger event 36. The UI algorithm 34A is additionally configured to anticipate a subsequent user request 40 (in Fig. 1, Fig. 3 and Fig. (shown in Figure 6) or to determine or predict an interaction of the system 20 via the input device 30 in response to the received trigger event 36 and the contextual data 38. With respect to the user, the contextual data 38 can be considered reasonably relevant to support decision-making or to form a solution to a likely problem defined by the trigger event 36.

[0025] The UI algorithm 34A is further trained to use the specified expected user request 40 to generate an actionable visual screen or a flow of linked actionable visual screens 28. As in Fig. As shown in Figure 2, each of the generated actionable visual screen(s) 28 includes one or more user prompts 28A, e.g., selectable virtual “buttons,” that correspond to the specified expected user request 40 and are intended to link successive (first, second, third, etc.) visual screens 28 in a generated chain 42 of screens. The UI algorithm 34A can access the expected user request 40 via an artificial intelligence (AI) agent 44 using machine learning, such as a generative pre-trained transformer (GPT), and be trained to determine it. The UX algorithm 34B is designed to select an actionable visual screen 28 from the generated actionable visual screen(s) and display it on the visual display device 26 in order to generate the ad-hoc actionable UI and UX 22.The depicted actionable visual screen 28 can define a starting point for a flow of linked actionable visual screens in response to the user's selection via the prompt(s) 28A. The UX algorithm 34B can use the Markov Decision Process (MDP) to determine or predict a likely flow of visual screens 28 based on the expected user request 40.

[0026] The Markov decision process, or stochastic control problem, is a model for sequential decision-making when outcomes are uncertain. An MDP builds upon the idea of ​​a Markov chain but adds the element of decision-making. In an MDP, an agent makes decisions that influence the transitions between states. Each decision (or action) made in a given state results in a probability distribution over the next possible states, similar to a Markov chain. However, unlike a simple Markov chain, in an MDP the agent can actively choose actions to optimize a particular goal, typically maximizing some kind of cumulative reward. Fig. Figure 4 presents the UX algorithm 34B using MDP 45 to optimize the expected user request 40 via a reward function R. t to determine. In Fig. Section 4 represents the current status of and information regarding the interface design, while S t+1 represents the status of the interface design at a subsequent timeframe or the next point in time. At represents feasible actions available to the UX algorithm 34B, which is executed by the UX algorithm, based on the current state S. t depend on each other. A t-gen represents generative actions that the system 20 (using an artificial intelligence algorithm) generates based on the specified expected user request 40. A generative tree 46 of a probable chain or flow 42 of visual screens 28, generated by the UX algorithm 34B (using the one in Fig. The MDP shown in section 45) is generated on-the-fly based on the expected user request 40, and is in Fig. 5 shown.

[0027] As in Fig. As shown in Figure 3, the UI algorithm 34A can further be trained to construct, in real time, a software code 48 trained to select a design and layout of the actionable visual screen(s) 28. Thus constructed, the software code 48 is intended to include access to a UI formulator 50 trained to generate the actionable visual screen(s) 28. For example, the UI formulator 50 can include access to a generative adversarial network (GAN) 52. In general, a generative adversarial network is a class of machine learning frameworks that approximate generative AI. In a GAN, two neural networks compete with each other in the form of a zero-sum game, where the gain of one agent is the loss of another. Given a training set, this technique learns to generate new data with the same statistics as the training set.The core idea of ​​a GAN is based on "indirect" training by a discriminator—another neural network used to determine how "realistic" the input appears—which is itself dynamically updated. As a result, the generator is not trained to minimize the difference from a specific image or target outcome, but rather to deceive the discriminator. This approach allows the GAN model to learn unsupervised.

[0028] With further reference to Fig. 3. The GAN 52 used can thus include a design origin function 52A, which is configured to generate a set 28-1 of actionable visual screen(s) 28. The GAN 52 can additionally include a discriminator function 52B, which is configured to filter the generated set 28-1 of alternative actionable visual screen(s) 28 by assessing how closely each of the visual screen(s) 28 meets criteria defined or specified by the particular expected user request 40. The discriminator function 52B can additionally be configured to limit the generated content to meet predefined constraints of the actionable visual screen, such as space, color, font, text length, grouping, etc., in the generated set 28-1 of alternative actionable visual screen(s) 28.The GAN 52 can include selective access to a database 54 of designs and layouts of interactive display screens for selection in the set 28-1 of alternative actionable visual screens. The GAN 52 can also include selective access to a library 56 of screen components that can be used to construct individual interactive visual display screens. Thus constructed, individual visual display screens 28 can be combined to formulate the set 28-1 of alternative actionable visual screens and to generate the chain 42 or flow of screens for the ad-hoc actionable UI and UX 22, as shown in . Fig. 3 and Fig. 6 shown.

[0029] Fig. 7 presents a method 100 for generating an ad-hoc actionable user interface (UI) and user experience (UX) 22 in real time via the system 20, as above in relation to Fig. 1- Fig. The method 100 is described in section 6. It can be implemented in the vehicle 10 using the vehicle's infotainment system 18 or in other settings using interactive visual screen(s) 28 for user interaction. The method 100 begins in frame 102 when the electronic control unit 32 receives the trigger event 36 communicated by the input device 30. Following frame 102, the method transitions to frame 104, where the method involves the electronic control unit 32 receiving data 38 from the data source(s), such as vehicle sensors 24-1, the IT cloud server 24-2, and the World Wide Web (WWW) 24-3, which is contextually related to the received trigger event 36.

[0030] Following framework 104, the procedure transitions to framework 106. Within framework 106, the procedure involves determining the expected user request 40 via the UI algorithm 34A in response to the received trigger event 36 and contextual data 38 received from the data source(s). As with regard to Fig. As described in section 6, the expected user request 40 can be determined by the UI algorithm 34A using the AI ​​agent 44. For example, the UI algorithm 34A can use a pre-trained AI framework or model, such as GPT. According to framework 106, the procedure proceeds to framework 108.

[0031] Within the scope of 108, the method involves generating one or more actionable visual screens 28 using the UI algorithm 34A, each visual screen having one or more user prompts 28A corresponding to the specified expected user request 40. Within the scope of 108, the method may additionally involve constructing software code 48 in real time using the UI algorithm 34A, which is responsible for selecting the design and layout of the ad-hoc actionable UI and UX 22. Within the scope of 108, the method may additionally involve accessing the UI formulator 50 via the constructed software code 48, as described in relation to Fig. 3, to generate the ad-hoc actionable UI and UX 22. The UI formulator 50 can in turn access the generative adversarial network (GAN) 52. As described in relation to Fig. 3 and Fig. As described in section 6, the GAN 52 can include the design origin function 52A for generating the set 28-1 of alternative actionable visual screen(s) 28 and the discriminator function 52B for filtering the generated set of alternative actionable screen(s) using criteria defined by the specified expected user request 40.

[0032] Within frame 108, the method can additionally include selective access to the database 54 of designs and layouts of interactive UIs and the library 56 of components for set 28-1 of alternative interactive visual screen(s) 28 via the GAN 52. From frame 108, the method transitions to frame 110. Within frame 110, the method includes selecting and displaying an interactive visual screen 28 from the generated interactive visual screen(s) on the visual display device 26 via the UX algorithm 34B, thereby generating the ad-hoc interactive UI and UX 22. Within frame 110, the method can, as in Fig. As described in section 4, the UX algorithm 34B determines the expected user request using Markov decision processing (MDP). Additionally, the UX algorithm 34B can generate a generative tree 46 of a probable flow of visual screens 28 based on the expected user request 40 using MDP on-the-fly, as described in Fig. 5 shown and described in relation to these.

[0033] Following frame 110, the method can return to frame 102 to receive another trigger event 36 via the input device 30, such as from the system or the vehicle user, to initiate a flow of user-interactive screens. Additionally, the method 100 can enable the implementation of changes to the ad-hoc actionable UI and UX 22 via screen regeneration on the visual display device 26 or via other feedback by returning to frame 104. Alternatively, the method 100 can terminate in frame 112 once the ad-hoc actionable UI and UX 22 have been generated, the flow of user-interactive screens has been displayed, and the user request has ended.

[0034] Overall, the real-time, on-the-fly generation of ad-hoc, actionable UIs is intended to improve the user experience and interface flows with personalized, up-to-date content. System 20 and Procedure 100 enable a visual display device to generate actionable user interfaces in real time for unplanned situations and to use data that may not have been available when the display device or host vehicle was manufactured. System 20 and Procedure 100 are also trained to receive data and content updates from the cloud and generative knowledge agents to enable the generation of actionable user interfaces with information and screen layouts specifically tailored to unforeseen situations and user requests.

[0035] The detailed description and the drawings or figures support and describe the disclosure, but the scope of the disclosure is defined exclusively by the claims. While some of the best ways and other embodiments for carrying out the claimed disclosure have been described in detail, there are various alternative designs and embodiments for implementing the disclosure, which are defined in the appended claims. Furthermore, the embodiments shown in the drawings or the features of various embodiments mentioned in this description are not necessarily to be understood as independent embodiments.Rather, it is possible that each of the features described in one of the examples of an embodiment can be combined with one or a multitude of other desired features from other embodiments, leading to other embodiments that are not described in words or by reference to the drawings. Accordingly, such other embodiments fall within the scope of the appended claims.

Claims

[1] System for generating an ad-hoc actionable user interface (UI) and user experience (UX) in real time, comprising: at least one data source; a visual display device designed to display actionable visual screens; an input device that is designed to receive external triggers; an electronic control in communication with the at least one data source, the visual display device and the input device and programmed with a UI algorithm and a UX algorithm, wherein the UI algorithm is: to receive a trigger event from the input device; to receive data from at least one data source that is contextually related to the received trigger event; to determine an expected user request in response to the received trigger event and contextual data received from the at least one data source; and to generate at least one actionable visual screen element, each containing one or more user prompts that correspond to the specified expected user request; and wherein the UX algorithm is configured to select an actionable visual screen from the at least one generated actionable visual screen and to display it on the visual display device in order to generate the ad-hoc actionable UI and UX. [2] System according to claim 1, wherein each of the visual display device, the input device and the electronic control is a component of a vehicle infotainment system. [3] System according to claim 2, wherein the at least one data source includes at least one from a vehicle sensor, an information technology (IT) cloud server and the World Wide Web (WWW). [4] System according to claim 1, wherein the UI algorithm is configured to determine the expected user request via an artificial intelligence (AI) agent. [5] System according to claim 4, wherein the UX algorithm uses the Markov Decision Process (MDP) to determine the expected user request. [6] System according to claim 5, wherein the expected user request is determined by optimizing a reward function defined using the trigger event and contextual data. [7] System according to claim 1, wherein the UI algorithm is further configured to construct in real time a software code which is configured to select a design and layout of the at least one actionable visual screen. [8] System according to claim 7, wherein the constructed software code includes access to a UI formulator configured to generate the at least one actionable visual screen. [9] System according to claim 8, wherein the UI formulator includes access to a generative adversarial network (GAN) comprising: a design origin function trained to generate a set of alternative actionable visual screens; and a discriminator function trained to filter the generated set of alternative actionable visual screens using criteria defined by the specific expected user request. [10] System according to claim 9, wherein the GAN includes selective access to the following: a database of designs and layouts for interactive visual screens; and a library of components for the set of alternative actionable visual screens.

Images