Method and vehicle for adapting images displayed on a display unit of a vehicle to real environmental conditions

Abstract

The invention relates to a method for adapting images displayed on a display unit of a vehicle to real environmental conditions, in which method a real image (7) is processed by means of a generative artificial intelligence (GAN) to form a modified image (19). In a method in which the images are interactively adapted to the environment of the vehicle, location data regarding the vehicle (1) and conditions prevailing at this location (13) are determined, the real image (7) comprising an image (7) which is in a context with the determined location data, and the real image being loaded into a computing unit (3), where the real image (7) which is in the context with the determined location (13) is modified by the generative artificial intelligence of the computing unit (3) on the basis of the conditions detected at the location (13) of the vehicle (1) and the modified image (19) is output on the display unit (5).

Description

[0001] Mercedes-Benz Group AG

[0002] Method and vehicle for adapting images shown on a display unit of a vehicle to real environmental conditions

[0003] The invention relates to a method for adapting images shown on a display unit of a vehicle to real environmental conditions, in which a real image is processed into a modified image using generative artificial intelligence, as well as to a vehicle for carrying out the method.

[0004] DE 102020 102 549 A1 discloses domain stylization using a neural network model, in which a photorealistic image is processed by the neural network model to generate predicted recognition data for the photorealistic image. Training data sets consisting of real images and predicted recognition data, and a synthetic training data set from the neural network model are used for style transfer to generate stylized synthetic images.

[0005] The object of the invention is to provide a method and a device for adapting images shown on a display unit of a vehicle to real environmental conditions, which interactively adapt to the environment of the vehicle.

[0006] The invention is based on the features of the independent claims. Advantageous developments and refinements are the subject of the dependent claims. Further features, possible applications, and advantages of the invention will become apparent from the following description and the explanation of exemplary embodiments of the invention illustrated in the figures.

[0007] The problem is solved by the subject matter of patent claim 1 or 12.

[0008] In the method explained above for adapting images shown on a display unit of a vehicle to real environmental conditions, in which a real image is processed using generative artificial intelligence to create a modified image, location data of the vehicle and the conditions prevailing at that location are determined. The real image comprises an image in context with the determined location data and is loaded into a computing unit, where the real image in context with the determined location is modified by the generative artificial intelligence of the computing unit based on the conditions detected at the location of the vehicle, and the modified image is output on the display unit. In the context of the application, generative artificial intelligence is understood to mean a preferably trained model operating according to the principle of generative artificial intelligence.This has the advantage that the underlying generative artificial intelligence makes the adjustments independently, without the driver having to proactively adjust anything. This eliminates menu navigation and the driver is not distracted. A context is defined as a variety of information obtained from the internet or via sensors. This includes points of interest, sights, navigation destinations, moods, music, trade fairs, events, and the like. Adapting to the context ensures an immersive experience in the vehicle and makes the driving experience more personal. Vehicle occupants also have the option of accessing additional, generic information via widgets (components on the graphic display), such as a description of a point of interest or information on current events.

[0009] Advantageously, a generative adversarial network (GAN) is used as the artificial intelligence. Such a generative adversarial network comprises two neural networks. One of the networks is configured as a generator, while the other neural network acts as a discriminator. The generator creates an image using the input data mentioned, such as the location and conditions recorded at the vehicle's location. The discriminator compares the image created by the generator with several current images and determines whether the created image statistically corresponds to one of the comparison images or not. The result is sent as feedback to the generator, whose ability to adapt the generated image to the current image is improved at each training stage, thereby improving the quality of the modified image.GAN networks are available as ready-made modules and can be cost-effectively integrated into the computing unit for implementing the method. In another alternative or additional embodiment, a diffusion model is provided as generative artificial intelligence.

[0010] In one embodiment, the artificial intelligence is implemented on the vehicle's processing unit or on another external processing unit. Depending on the required computing power, the choice of which processing unit to install the artificial intelligence on can be made. Since the vehicle itself only has limited computing power available for many different functions, installing it on a vehicle backend is recommended for very high computing demands to reduce the vehicle's load.

[0011] In one variant, a stored image adapted using a generative artificial intelligence generator, such as a GAN network, is used as the real image contextualized with the determined location data and loaded into the computing unit. By repeatedly processing the stored images in different learning sections of the generator, the informative value of the images is increased. By loading an image already adapted using generative intelligence, adaptation based on the conditions recorded at the location is possible very quickly, i.e., ideally in real time.

[0012] In one embodiment, the loaded stored image includes static and time-varying information. This ensures that the image is always adapted to the current conditions of the vehicle.

[0013] It is advantageous if the stored image is loaded from the external processing unit into the vehicle. This reduces storage capacity in the vehicle while still allowing the modified image to be updated in the vehicle at any time.

[0014] In a further embodiment, the image loaded by the generative artificial intelligence generator and the conditions at the vehicle's location are used as input variables for a generative artificial intelligence discriminator, for example, a GAN network. This means that the comparison images stored in the discriminator always correspond to the current situation in the vehicle's surroundings. In another variant, the computing unit modifies a background of the image in context with the determined location data. Thus, current background information is adaptively changed, and as the vehicle passes by, the background of the display unit is adjusted depending on the cities, landscapes, landmarks, weather, etc. currently being driven through.

[0015] In a further embodiment, the conditions determined at the location of the vehicle include, in addition to images of the surroundings, the surrounding and / or environmental conditions and / or driving conditions of the vehicle, which are determined using sensors. Sensors already present in the vehicle are used to record these conditions, thereby reducing the effort required to perform these measurements. Advantageously, settings for displaying the modified image on the display unit are specified via a user interface of the computing unit. This allows the vehicle occupant to influence the representation of the image on the display unit and select key points in the display for which they would like more detailed information.

[0016] A further aspect of the invention relates to a vehicle for carrying out the method according to at least one feature described in this patent application, comprising

[0017] Means for determining a location of the vehicle,

[0018] Means for determining the conditions prevailing at the location, Means for loading a real image in context with the location of the vehicle,

[0019] Means for modifying the real image in context with the location using generative artificial intelligence based on the conditions prevailing at the location of the vehicle and

[0020] Means for displaying the image modified by the generative artificial intelligence.

[0021] Further advantages, features, and details will become apparent from the following description, in which at least one exemplary embodiment is described in detail—possibly with reference to the drawings. Described and / or illustrated features may form the subject matter of the invention alone or in any meaningful combination, possibly independently of the claims, and may, in particular, also be the subject of one or more separate applications. Identical, similar, and / or functionally equivalent parts are provided with the same reference numerals.

[0022] They show:

[0023] Fig. 1 shows an embodiment of the vehicle according to the invention for carrying out the method according to the invention,

[0024] Fig. 2 shows an embodiment of a display unit with a modified image.

[0025] Fig. 1 shows an exemplary embodiment of the vehicle according to the invention for carrying out the method according to the invention. The vehicle 1 comprises a computing unit 3 into which an artificial generative intelligence model such as a generative adversarial network (GAN) is integrated. This computing unit 3 collects various information, data, and images and outputs a passively adaptable generative visualization in the form of images 19 to a display unit 5. Based on context information and scene information about the location of the vehicle 1, the time of year, weather conditions, traffic density, terrain, etc., a user interface and / or the background image of a surface adapts accordingly.

[0026] The computing unit 3 receives the information necessary to create a modified image 19 on the display unit 5 from various sources. A key component is a positional context image 7 of a city or landscape. Using a location determination unit 9, for example a GPS unit or GALILEO, the current position of the vehicle 1, on which the context image 7 is based, is determined. This data is fed to a position processing unit 11, which compares the position of the context image 7 with potentially interesting points, such as points of interest 13. These potentially interesting points 13 are taken from a digital map 15 and / or a database 17 loaded with corresponding images.

[0027] Furthermore, depending on the direction in which the vehicle 1 is traveling or a direction specified by the route planned in the navigation system, directional information 21 is taken into account when determining the potentially interesting points 13. For example, if a scene has several different landscape features at different locations, these landscape features are weighted depending on the direction in which the vehicle 1 is traveling or intends to travel.

[0028] The position processing unit 11 transmits this position or direction data 23 to a unit 25 for processing the location-dependent context data. For processing, further context information is retrieved from a local or online database 27, which contains static data or seasonal or temporal information. The static information includes general information about a city, a building, a landscape, a scenic route, or a tourist attraction. This information can be a short article or an aggregated summary, either through available APIs (application programming interfaces) that interface with a mobile app and forward information from it, knowledge databases, or sophisticated large-scale language models.

[0029] Seasonal or temporal information refers to information about events, shows, updated opening hours, etc., as well as seasonal and time data that are only valid for a specific period of time. The data provided by unit 25 for processing the location-dependent context data is processed in computing unit 3. Computing unit 3, working with the generating artificial intelligence or a model of the generating artificial intelligence, adaptively and contextually creates a modified version of image 7. To reduce the computational effort, preprocessed generative images can also be extracted from the database 17 loaded with images.

[0030] In addition, further context information and data can be extracted from meta and sensor sources and processed in a corresponding data processing unit 29. Sensory data is provided by various sensor units 31 of the vehicle 1. This includes speed, direction / orientation, inclination or gradient information 31.1, a camera system 31.3 for better scene understanding, a light sensor 31.5 for providing a lighting environment for the camera system 31.3, and weather data such as rain, temperature, and humidity provided by a sensor group 31.7. Furthermore, date, time, and seasonal information 33 as well as information 35 provided from the internet, such as weather forecasts and trend-dependent or location-dependent social media activities, can be added.The data processing unit 29 processes all meta and sensor information into a context-related data packet 37, which is fed to the computing unit 3 for further processing.

[0031] As an additional optional control device, the driver can set preferred settings for the modified image 19 in a user interface 39. The settings to be edited include the selection or removal of context-related input data, the refresh rate, for example, of the current image background or the location and direction depending on changes in local points of interest, or widget settings, such as removing superfluous information and selecting the content a vehicle occupant wishes to read. The user interface 39 can be controlled by the vehicle occupant manually or via voice or gestures.

[0032] The computing unit 3 processes all the data described above and creates the adaptive generative background appearance of the modified image 19, which is displayed on the display unit 5.

[0033] However, all of the described processes can also run on a backend system external to the vehicle (not shown further). The backend system should be configured to retrieve all meta, time / date, sensor, and location information of the vehicle 1 or a context-related data packet. Furthermore, the backend system can retrieve a scene outside the vehicle 1 and understand the context within the vehicle 1 in order to create a generative interface to the vehicle 1. Furthermore, the backend system can create location-dependent images of the desired points of interest, including landscapes, scenic routes, cities, and their skylines, and evaluate location-, time-, and season-dependent information.A model integrated into the backend system using generative artificial intelligence generates background images for the display unit 5 of the vehicle 1 from this context, with the generative background images being loaded into a frontend system of the vehicle 1. The generated data can be combined into information blocks, for example based on text information about points of interest, cities, events, or landscapes. Fig. 2 shows an embodiment of a display unit 5 with a modified image 19 that was generated in the vehicle 1 or the backend system. The display unit 5 is designed as a display and consists of three segments: a digital combination instrument 5.1 for the driver, a central display 5.3 between the driver and front passenger, and a front passenger display 5.5.This display unit 5 extends between the two A-pillars of vehicle 1 and is also referred to as a pillar-to-pillar display. The virtual distances between segments 5.1, 5.3, and 5.5 are indicated by vertical dashed lines SEP.

[0034] On display unit 5, a city skyline is displayed as a modified image 19. The modified image 19 shows the sky above the city and represents a slightly cloudy spring day. Changes in the weather or other environmental data result in a changing appearance of the background, while the skyline remains unchanged. However, the computing unit 3 uses generative artificial intelligence to add fog, rain, and snow. Changing lighting conditions are also taken into account.

[0035] Additionally, context-related widgets 41 are displayed on the display unit 5, allowing the vehicle occupant to read additional information about the city or landscape. Furthermore, these widgets 41, 43 can contain information about current events. In this case, additional widgets 43 can be added and placed at desired points on the modified image 19 depending on the information. The computing unit 3 records the vehicle occupant's interaction with the widgets 41, 43 to minimize the occurrence of redundant or already displayed data.

[0036] The image information underlying the modified image 19 changes with the position of the vehicle 1. For example, if the driver drives past a city on a highway or enters another city, another area or another landscape, the generative appearance and the corresponding widgets 41, 43 change accordingly.

[0037] The speed of vehicle 1 is also taken into account when generating the modified image 19. Driving slowly within a city allows for a more differentiated view of the city and its surroundings. More detailed modified images 19 can be displayed. The widget 41, 43 can then display static events or other temporal activities. To prevent distractions, the widget in the driver's segment 5.1 can be deactivated.

[0038] The described solution is implemented either fully or partially as software in vehicle 1. In the partial implementation, certain computationally intensive elements of the software are stored and executed outside the vehicle, for example, within a cloud or a vehicle backend.

Claims

Mercedes-Benz Group AG Patent claims 1. Method for adapting images shown on a display unit of a vehicle to real environmental conditions, in which method a real image (7) is processed using generative artificial intelligence to form a modified image (19), characterized in that location data of the vehicle (1) and conditions prevailing at this location (13) are determined, wherein the real image (7) comprises an image in context with the determined location data and is loaded into a computing unit (3), where the real image (7) in context with the determined location (13) is modified by the generative artificial intelligence of the computing unit (3) on the basis of the conditions detected at the location (13) of the vehicle (1), and the modified image (19) is output on the display unit (5).

2. Method according to claim 1, characterized in that a Generative Adversarial Network (GAN) is used as artificial intelligence.

3. Method according to claim 1 or 2, characterized in that the generative artificial intelligence is implemented on the computing unit (3) of the vehicle (1) or a further computing unit external to the vehicle.

4. Method according to claim 3, characterized in that the creation of the modified image (19) takes place as a function of the expected required computing power in the computing unit (3) of the vehicle (1) or the further computing unit external to the vehicle.

5. Method according to at least one of the preceding claims, characterized in that a stored image adapted by means of a generator of generative artificial intelligence is used as the real image (7) which is in context with the determined location data and which is loaded into the computing unit (3).

6. Method according to claim 5, characterized in that the loaded stored image comprises static and time-varying information.

7. Method according to claim 5 or 6, characterized in that the stored image is loaded into the vehicle (1) from the vehicle-external computing unit.

8. The method according to claim 5, 6 or 7, characterized in that the image loaded by the generative artificial intelligence generator and the conditions at the location (13) of the vehicle (1) are used as input variables of a generative artificial intelligence discriminator.

9. Method according to at least one of the preceding claims, characterized in that the computing unit (3) modifies a background of the image (19) in context with the determined location data.

10. Method according to at least one of the preceding claims, characterized in that the conditions determined at the location (13) of the vehicle (1) include, in addition to surrounding images, surrounding and / or environmental conditions and / or driving conditions of the vehicle (1), which are determined by means of sensors (31).

11. Method according to at least one of the preceding claims, characterized in that Settings for displaying the modified image (19) on the display unit (5) are specified via a user interface (39) of the computing unit (3).

12. Vehicle for carrying out the method according to at least one of the preceding claims, comprising Means (9) for determining a location (13) of the vehicle (1), means (15, 17, 21) for determining the conditions prevailing at the location (13), Means (31.3) for loading a real image (7) in context with the location (13) of the vehicle (1), Means (3) for modifying the real image (7) in context with the location (13) using a generative artificial intelligence on the basis of the conditions present at the location (13) of the vehicle (1), and means (5) for displaying the image (19) modified by the generative artificial intelligence.

Images