Device and method for operating a user interface of a vehicle

Abstract

The invention relates to a device (100) for operating a user interface of a vehicle (102), the device comprising an output unit (106) of the vehicle (102), the output unit being designed to output at least one element of the user interface of the vehicle (102) to a user (104). The device (100) also comprises a detection unit (108) which is designed to detect at least one item of information corresponding to a user input from the user (104) and / or to environmental information relating to the surroundings of the vehicle (102), and to generate input data corresponding to the detected information. The device (100) also comprises a control unit (110) which is designed to control the output unit (106) of the vehicle (102) and to process the input data using at least one machine learning method. The control unit (110) is also designed to generate, based on the input data and using the machine learning method, at least one element of the user interface of the vehicle (102), which element can be output by the output unit (106), and to control the output unit (106) for outputting the generated element.

Description

[0001] Device and method for operating a user interface of a vehicle

[0002] The invention relates to a device for operating a user interface of a vehicle. The invention further relates to a method for operating a user interface of a vehicle.

[0003] The user interface of a modern vehicle comprises a multitude of output units, in particular visual output units for outputting visual information. Examples of visual output units are an instrument cluster, a central information display (CID), a projector, or a head-up display. However, lighting elements arranged inside the vehicle interior or on the outside of the vehicle can also represent visual output units. Vehicles also have auditory output units, for example, speakers of a vehicle's media playback system. Elements output by the output units, for example elements of a graphical user interface or voice output, typically follow a uniform design. The design is predefined at the factory, and a user can choose from at most a few designs. However, many users have the need to personalize the design of the user interface.However, storage space and bandwidth are limited, so it is not easy to provide a large number of user interface elements in a variety of designs.

[0004] DE 10 2021 206 537 A1 discloses a method in which graphical user interfaces for various devices and platforms are generated using a neural network. In this method, a second user interface for a second platform is generated based on functional and visual features of a first user interface of a first platform. The second user interface can be generated, in particular, taking into account limitations of the second platform.

[0005] US Patent No. 1,045,2902 B1 discloses a method in which image components of a patent drawing are generated using a generative adversarial network. One example is a patent drawing of a graphical user interface.

[0006] The object of the invention is to provide a device and a method for operating a user interface of a vehicle, which can have a variety of designs and use little memory and / or bandwidth.

[0007] This object is achieved by a device having the features of claim 1 and by a method having the features of the independent method claim. Advantageous further developments are specified in the dependent claims.

[0008] The proposed device for operating a user interface of a vehicle comprises an output unit of the vehicle, which is designed to output at least one element of the user interface of the vehicle to a user. The device also comprises a detection unit, which is designed to detect at least one piece of information corresponding to a user input of the user and / or environmental information related to the surroundings of the vehicle, and to generate input data corresponding to the detected information. The device further comprises a control unit, which is designed to control the output unit of the vehicle and to process the input data using at least one machine learning method.The control unit is further configured to generate, based on the input data and using the machine learning method, at least one element of the vehicle's user interface that can be output by the output unit, and to control the output unit to output the generated element. The output unit can, for example, comprise a display surface facing the vehicle driver and / or another occupant, or a vehicle loudspeaker. The detection unit comprises, in particular, an input unit for detecting the user input and / or an environmental sensor system of the vehicle for detecting the environmental information. Further examples of the output unit and the detection unit are mentioned in the following description in connection with embodiments. At least part of the control unit can be formed by a processing unit remote from the vehicle.For example, the machine learning process can be executed on the remote processing unit, while the control of the output unit takes place locally in the vehicle.

[0009] The device first captures the information on the basis of which the user interface element is to be generated. This information can be user input, for example, a selection from predefined text modules or a prompt freely entered by the user. The information can also be environmental information, for example, the time of day, the time of year, or the geographical region in which the vehicle is currently located. Furthermore, the captured information can also be a combination of the user input and the environmental information. The captured information is used to generate the input data that serves as input to the machine learning process. The input data can, for example, be modified, or the input can be restricted to a predefined framework, for example to create a brand-appropriate aesthetic framework for generating the element.The user interface element is then generated using the machine learning method. The machine learning method can be a generic model or a fine-tuned model based on a small training dataset. A generic model has the advantage that a very large number of very different designs can be realized. A fine-tuned model has the advantage that, for example, it can be ensured that the generated elements are designed in accordance with the design principles of a brand identity. Machine learning methods are inherently noisy, so the output varies even with the same input. For the device, this means that each generated element is unique, making the user interface hyper-personalizable. This allows the device to satisfy the user's need for personalization of the user interface.However, because the user interface elements are generated and not deployed, the device uses less storage space and / or bandwidth for this personalization than known personalizations.

[0010] In one embodiment, the output unit is designed to output visual information to the user, and the generated element is a visual element. The control unit is in particular designed to generate an image as the visual element. In such an embodiment, the output unit can in particular comprise a display unit of the vehicle, for example a graphic instrument cluster, a central information display (CID), a head-up display, in particular a panoramic head-up display, or other display surfaces facing the occupants, for example projection surfaces belonging to a projector. However, the output unit can also comprise a controllable lighting element which is designed, for example, to generate light of an adjustable color and / or an adjustable sequence of different colored light.In such an embodiment, the generated visual element is the light color or the sequence of different colored light. In particular, by generating exterior lighting, a personalized welcome scenario can be realized, for example. Providing various visual elements locally requires a particularly large amount of storage space. Accordingly, transmitting visual elements from a remote storage location requires a significant amount of bandwidth. Thus, generating visual elements can save a significant amount of storage space or bandwidth.In a further embodiment, the control unit is configured to generate a first image having a first resolution using the machine learning method and, based on the first image, to generate a second image having a second resolution as the visual element using a further machine learning method, wherein the second resolution is higher than the first resolution. For example, a background image for a display surface of the vehicle can initially be generated at a low resolution based on a prompt entered by the user and then upscaled to obtain a high-resolution background image. Generating images is more computationally intensive than increasing the resolution. Thus, in this embodiment, the computing power used to generate the visual element can be optimally utilized.

[0011] In a further embodiment, the control unit is designed to generate at least one visual element, which is part of a graphical user interface of the vehicle, based on the input data and using the machine learning method. The visual element can be, for example, an operating element such as a button, a functional element such as a warning notice, or a decorative element such as a background image. The visual element can be static or dynamic, for example animated. The user interacts with the vehicle primarily via the graphical user interface. The elements of the graphical user interface are therefore particularly formative for the impression the user interface leaves on the user. Accordingly, the option of customizing the graphical user interface offers considerable added value.On the other hand, a typical graphical user interface comprises a large number of visual elements. By generating these visual elements, great added value can be created for the user with little expenditure on storage space or bandwidth. In a further embodiment, the control unit is designed to generate at least one further element of the user interface based on the generated visual element. The further element can in particular be generated based on a template in order to obtain, for example, a uniform design of the graphical user interface in accordance with the design principles of a brand identity despite individualization. In one example, the control unit first generates a background image as the visual element. Using a clustering algorithm, for example median cut, the control unit then extracts dominant colors from the generated background image.The control unit then generates the remaining elements of the graphical user interface based on the extracted color. For example, the control unit colors the remaining elements according to the extracted color. Since in this embodiment, only one visual element is initially generated using the machine learning method, this embodiment is particularly resource-efficient.

[0012] In a further embodiment, the control unit is configured to generate, based on the input data and using the machine learning method, a plurality of visual elements that are part of a uniform skin of the graphical user interface. A skin, or theme, refers to the uniform design of the elements of the graphical user interface. In other words, in this embodiment, for example, the layout, typography, colors, patterns, as well as icons and buttons, are generated uniformly and in a coordinated manner according to predefined specifications. The visual elements generated in this way adhere to a common scheme and create a consistent impression for the user.

[0013] In a further embodiment, the detection unit is designed to provide the input data in the form of text in natural language. The control unit is designed to process the input data using a text-to-image model as the machine learning method in order to generate the visual element. The input data can, for example, be a prompt that the user has freely entered or composed based on predefined text modules. Alternatively or additionally, the text in natural language can also be generated by a further machine learning method based on the environmental information. For example, the further machine learning method can be used to classify the immediate surroundings. In one example, the control unit determines, based on the environmental information and using the further machine learning method, that the vehicle is driving through an alpine landscape.The control unit then generates the prompt "alpine landscape" as input for the text-to-image model to generate the visual element. A number of generic text-to-image models exist in the prior art that can be easily adapted to the requirements of the device. This embodiment is therefore particularly simple to implement.

[0014] In a further embodiment, the output unit is configured to output auditory information to the user, and the element of the user interface is an auditory element. Audible elements, such as short pieces of music (jingles) or voice outputs, are also part of the look and feel of the user interface. These audible elements can also be generated using the machine learning method to enable greater customization.

[0015] In a further embodiment, the detection unit is configured to detect, as information, navigation information relating to the vehicle and / or weather information relating to the vehicle's surroundings. For example, the detection unit can receive the position of the vehicle from a global satellite navigation system. The navigation information is, for example, the geographical region in which the vehicle is currently located. The weather information can, for example, be the current weather or the time of year detected or received from a corresponding service. Based on the navigation information and / or the weather information, the element of the user interface can be adaptively adjusted to the vehicle's surroundings without having to provide many storage- or bandwidth-intensive elements.In one example, the control unit determines, based on weather information, that it is snowing in the area surrounding the vehicle. The control unit then generates visual elements with a snow theme using the machine learning technique.

[0016] In a further embodiment, the detection unit comprises an image detection unit of the vehicle, which is designed to capture at least one image with a depiction of an area outside the vehicle as the information and to generate image data corresponding to the image, and the input data comprises the image data. The image detection unit is in particular an external camera of the vehicle and is designed to capture a two-dimensional image of an area outside the vehicle. The image detection unit captures in particular light in the optical spectrum and / or in the infrared spectrum. As an alternative to an optical camera, the environment detection unit can also comprise a RADAR or LIDAR system, which generates a three-dimensional image of the area outside the vehicle, in particular in the form of a point cloud.A series of information can be extracted from the image data, on the basis of which the element of the user interface can be generated. In one example, the control unit determines the predominant color in the surroundings of the vehicle based on the image data and generates a visual element of the graphical user interface based on the predominant color. In another example, the control unit determines, based on the image data, that the vehicle is currently driving through an alpine landscape and generates visual elements of the graphical user interface with an alpine theme that corresponds to the landscape currently being driven through. In a further embodiment, the detection unit comprises a mobile terminal that is configured to detect the information, in particular the user input.The end device can, for example, be a smartphone or a tablet computer running an application that communicates with the control unit. User input can be captured particularly easily via the mobile device, particularly when the user is not inside the vehicle. To create a particularly consistent look and feel, the control unit can also be configured to generate elements of a user interface on the mobile device that correspond to the elements generated in the vehicle's user interface. For example, the control unit can generate visual elements of the application's graphical user interface that correspond to the visual elements of the vehicle's graphical user interface.

[0017] In another embodiment, the machine learning method is a generative model, in particular a generative adversarial network. Generative models, in particular diffusion models and generative adversarial networks, exist in the prior art and can be easily adapted to the requirements of the device. The device is thus easy to implement. In particular, the generative model can be fine-tuned based on a small training dataset to create the elements of the user interface according to predefined design principles.

[0018] In a further embodiment, the control unit is configured to control the output unit to output at least one predefined question to the user. The acquisition unit is configured to acquire the user's response in the form of a user input as the information and generate the input data of the response accordingly. The predefined question can, for example, ask the user about their favorite artist, favorite vacation destination, and / or favorite color. This guides the user through the creation of the element and facilitates the use of the device.

[0019] In a further embodiment, the detection unit is configured to detect user feedback related to the generated element and to generate feedback data corresponding to the feedback. The control unit is configured to modify the machine learning method based at least on the feedback data. The feedback may include, for example, whether the user is satisfied with the generated element. Based on this feedback, the machine learning method can then be modified until an element satisfactory to the user is generated. This increases the user-friendliness of the device.

[0020] The invention further relates to a method for operating a user interface of a vehicle. In the method, at least one piece of information is acquired that corresponds to a user input from the user and / or environmental information related to the vehicle's surroundings. Input data corresponding to the acquired information is generated. Based on the input data and using a machine learning method, at least one element of the vehicle's user interface is generated. The generated element is output by an output unit of the vehicle.

[0021] The method has the same advantages as the claimed device.

[0022] In particular, the method can be further developed with the features of the dependent claims directed to the device. Furthermore, the device described above can be further developed with the features described in this document in connection with the method.

[0023] Embodiments of the invention are explained in more detail below with reference to the figures. Figure 1 shows a schematic representation of a device for operating a user interface of a vehicle; and

[0024] Figure 2 shows a flowchart of the method for operating the vehicle's user interface.

[0025] Figure 1 shows a schematic representation of a device 100 for operating a user interface of a vehicle 102.

[0026] The device 100 serves to personalize the user interface of the vehicle 102 for a user 104. To this end, the device 100 generates at least one element of the user interface, which can be output by an output unit 106, based on a user input and / or environmental information related to the surroundings of the vehicle 102.

[0027] The element generated by device 100 can, for example, be an operating element or a decorative element of a graphical user interface that is output to user 104 via a screen and / or a head-up display of vehicle 102 as output unit 106. Audible elements, such as a short piece of music or a voice output, can also be generated by device 100 and output, for example, via a loudspeaker of vehicle 102 as output unit 106. Furthermore, the element can also be abstract, such as a light color or a sequence of light colors that can be output by lighting elements of vehicle 102 as output unit 106.

[0028] To capture information on the basis of which the user interface element is generated, the device 100 comprises a capture unit 108. The capture unit 108 processes the captured information and generates input data therefrom, which the capture unit 108 transmits to a control unit 110 of the device 100. The input data can be generated, in particular, in the form of a text in natural language, which is particularly easy to further process. However, the input data can also be generated in the form of any token that can be further processed by the control unit 110.

[0029] In the exemplary embodiment shown in Figure 1, the detection unit 108 comprises an input unit 112 configured to detect a user input from the user 104. The input unit 112 may be a haptic input unit, for example, an ego commander, a keyboard, or a touchscreen of the vehicle 102. However, the input unit 112 may also comprise a microphone and be configured to receive the user input in spoken form from the user 104. Furthermore, the input unit 112 may also be configured to receive the user input in the form of a gesture from the user 104.

[0030] The user input can, for example, be a prompt composed by the user 104 from predefined text modules or freely formulated, which describes aesthetic qualities of the element to be created.

[0031] In particular, the device 100 can also be designed to motivate the user 104 to enter the prompt by means of a corresponding output.

[0032] For example, device 100 can ask user 104 about their favorite vacation destination, favorite artist, and / or favorite color. This output can be visual, for example, through text displayed on a screen and / or a head-up display of vehicle 102. However, the output can also be audible, for example, in the form of a voice output generated by device 100. In the exemplary embodiment shown, detection unit 108 also includes an environmental sensor system of vehicle 102 configured to detect environmental information. The environmental sensor system is shown purely by way of example as an external camera 114 of vehicle 102. Using external camera 114, for example, a predominant color tone in the surroundings of vehicle 102 can be determined as the environmental information.The predominant color tone can be determined, for example, by the detection unit 108 processing an image captured by the exterior camera 114 using a clustering algorithm. Furthermore, the detection unit 108 can be configured to determine a time of day, a season, and / or the weather in the surroundings of the vehicle 102 as the environmental information based on the image.

[0033] In the illustrated embodiment, the detection unit 108 further comprises a navigation system 116 configured to determine or receive the position of the vehicle 102. The detection unit 108 may, for example, be configured to determine the geographical region in which the vehicle 102 is currently located as the environmental information based on the position of the vehicle 102.

[0034] The control unit 110 of the device 100 is configured to process the input data using a machine learning method to generate the user interface element. The machine learning method is, in particular, a generative model, for example, a generative adversarial network. If a visual element is to be generated, the control unit 110 can, in particular, use a text-to-image model to process the input data in the form of text in natural language. The generation of the user interface element is described in more detail below with reference to Figure 2. In the exemplary embodiment shown in Figure 1, the control unit 110 is part of the vehicle 102 itself. In other embodiments, at least part of the control unit 110 can be formed by a processing unit remote from the vehicle 102. For example, the machine learning method can be executed on the remote processing unit.This allows the machine learning process to be updated more easily by the provider and does not consume any memory space of the vehicle 102.

[0035] The exemplary device 100 shown in Figure 1 further comprises a mobile terminal 118 of the user 104, which is shown purely as an example as a smartphone. The mobile terminal 118 is connected to the device 100, for example, via a wireless data connection such as Bluetooth®, WLAN, or near-field communication (NFC). An application can run on the mobile terminal 118 that enables the user 104 to make user inputs to the device 100 via the mobile terminal 118. Furthermore, the control unit 110 can be configured to generate at least one element of a user interface of the mobile terminal 118 corresponding to the generated element of the user interface of the vehicle 102 and to transmit it to the mobile terminal 118. As a result, the user interfaces of the vehicle 102 and the mobile terminal 118 have a uniform look and feel.

[0036] Figure 2 shows a flowchart of a method for operating the user interface of the vehicle 102.

[0037] The method is started in step S200. The method can be started, in particular, based on a corresponding user input from user 104. In step S202, the control unit 110 controls the acquisition unit 108 to acquire the information on the basis of which the user interface element is to be generated. In a first example, the device 100 requests the user 104, via a corresponding output, to formulate a prompt from a set of predefined text modules. In a second example, the device 100 asks the user 104, via a corresponding output, about the user 104's favorite vacation destination and favorite artist in order to generate the input data from the user 104's response.In a third example, the detection unit 108 captures an image of the surroundings of the vehicle 102 and, based on the image and using an image classification method, determines that the vehicle 102 is currently located in an alpine environment. Based on this environmental information, the detection unit 108 generates the input data, for example, as natural language text: "alpine landscape."

[0038] In step S204, the control unit 110 generates the element of the user interface of the vehicle 102 based on the input data and using the machine learning method. In the first example, the control unit 110 generates operating elements and decorative elements of a graphical user interface of the vehicle 102 using a text-to-image model based on the prompt formulated by the user 104. The control unit 110 further generates a short piece of music based on the prompt, which is output to the user 104 as part of a welcome scenario. In the second example, the control unit 110 generates a background image for the graphical user interface of the vehicle 102 using a generative adversarial network. This background image corresponds to an image of a landscape of the user's 104 favorite vacation destination in the style of the user's 104 favorite artist.In the third example, the control unit 110 generates controls and decorative elements of the graphical user interface of the vehicle 102 with an alpine theme.

[0039] In the optional step S206, the control unit 110 generates further elements of the user interface based on the already generated element. In the second example, the control unit 110 first extracts predominant colors from the generated background image using a clustering algorithm. The control unit 110 then colors the controls and decorative elements of the graphical user interface of the vehicle 102 in the extracted colors. The control unit 110 further generates instructions for lighting elements of the vehicle 102 to generate light in at least one of the predominant colors of the background image in order to illuminate the vehicle interior in these colors.

[0040] In step S208, the control unit 110 controls the output unit 106 to output the generated element(s). In the examples, the control unit 110 controls, for example, a screen of the vehicle 102 to display the graphical user interface with the generated controls and decorative elements. In the first example, the control unit 110 further controls a speaker of the vehicle 102 as the output unit 106 to play the generated piece of music when the user 104 enters the vehicle 102. In the second example, the control unit 110 further controls the lighting elements to illuminate the vehicle interior in the predominant colors of the background image.

[0041] The method is then terminated in step S210.

[0042] In the exemplary embodiment described with reference to Figures 1 and 2, at least the output unit 106, the detection unit 108, and the control unit 110 form the device 100 for operating the user interface of the vehicle 102. Further elements and features shown in Figures 1 and 2 and mentioned in the preceding description may be part of the device 100. Likewise, method steps described with reference to the device 100 may be part of the claimed method. List of Reference Symbols

[0043] 100 device

[0044] 102 vehicles

[0045] 104 users

[0046] 106 Output unit

[0047] 108 registration unit

[0048] 110 Control unit

[0049] 112 Input unit

[0050] 114 Image acquisition unit

[0051] 116 Navigation system

Claims

Claims 1. A device (100) for operating a user interface of a vehicle (102), comprising an output unit (106) of the vehicle (102) configured to output at least one element of the user interface of the vehicle (102) to a user (104); a detection unit (108) configured to detect at least one piece of information corresponding to a user input of the user (104) and / or environmental information related to the surroundings of the vehicle (102), and to generate input data corresponding to the detected information;and with a control unit (110) which is designed to control the output unit (106) of the vehicle (102) and to process the input data using at least one machine learning method, wherein the control unit (110) is further designed to generate, on the basis of the input data and using the machine learning method, at least one element of the user interface of the vehicle (102) which can be output by the output unit (106), and to control the output unit (106) to output the generated element.; 2. The device (100) of claim 1, wherein the output unit (106) is configured to output visual information to the user (104), and the generated element is a visual element.

3. The device (100) of claim 2, wherein the control unit (110) is configured to generate a first image having a first resolution using the machine learning method and to generate a second image having a second resolution as the visual element based on the first image using a further machine learning method, the second resolution being higher than the first resolution.

4. The device (100) according to claim 2 or 3, wherein the control unit (110) is configured to generate at least one visual element that is part of a graphical user interface of the vehicle (102) based on the input data and using the machine learning method.

5. Device (100) according to claim 4, wherein the control unit (110) is configured to generate at least one further element of the user interface based on the generated visual element.

6. The device (100) according to any one of claims 2 to 5, wherein the control unit (110) is configured to generate, based on the input data and using the machine learning method, a plurality of visual elements that are part of a uniform skin of the graphical user interface.

7. The device (100) according to any one of claims 2 to 6, wherein the acquisition unit (108) is configured to provide the input data in the form of natural language text, and the control unit (110) is configured to process the input data using a text-to-image model as the machine learning method to generate the visual element.

8. Device (100) according to one of the preceding claims, wherein the output unit (106) is designed to output auditory information to the user (104), and the element of the user interface is an auditory element.

9. Device (100) according to one of the preceding claims, wherein the detection unit (108) is designed to detect as the information navigation information related to the vehicle (102) and / or weather information related to the surroundings of the vehicle (102).

10. Device (100) according to one of the preceding claims, wherein the detection unit (108) comprises an image detection unit (114) of the vehicle (102) which is designed to detect as the information at least one image with a depiction of an area outside the vehicle (102) and to generate image data corresponding to the image, and the input data comprise the image data.

11. Device (100) according to one of the preceding claims, wherein the detection unit (108) comprises a mobile terminal (118) which is designed to detect the information, in particular the user input of the user (104).

12. Device (100) according to one of the preceding claims, wherein the machine learning method is a generative model, in particular a generative adversarial network.

13. Device (100) according to one of the preceding claims, wherein the control unit (110) is designed to control the output unit (106) to output at least one predefined question to the user (104), and the detection unit (108) is designed to detect the answer of the user (104) in the form of a user input as the information and to generate the input data of the response accordingly.

14. Device (100) according to one of the preceding claims, wherein the detection unit (108) is designed to detect feedback from the user (104) related to the generated element and to generate feedback data corresponding to the feedback, and the control unit (110) is designed to modify the machine learning method at least on the basis of the feedback data.

15. A method for operating a user interface of a vehicle (102), in which at least one piece of information is detected which corresponds to a user input of the user (104) and / or environmental information relating to the surroundings of the vehicle (102); input data corresponding to the detected information is generated; at least one element of the user interface of the vehicle (102) is generated on the basis of the input data and using a machine learning method; and the generated element is output by an output unit (106) of the vehicle (102).

Images