Method for generating display content for a display device

Abstract

In a method for generating display content (9) for a display device (6) in a vehicle, initial content (7, 8) is provided as the basis for display content (9) to be created for the display device (6). At least one vehicle parameter is detected. Display content (9) for the display device (6) is then created proceeding from the initial content (7, 8), wherein the display content is created by means of a generative machine learning model taking into account the at least one vehicle parameter in such a way that the generated display content (9) represents the initial content (7, 8) and is adapted to the at least one vehicle parameter.

Description

[0001] METHOD FOR GENERATING DISPLAY CONTENT FOR A DISPLAY DEVICE

[0002] The present invention relates to a method for generating display content for a display device, particularly in a vehicle. In particular, a display content should be flexibly generated for a multitude of different display devices based on an input content.

[0003] Most vehicles come standard with at least one display device to show various content, such as vehicle information or functions like navigation, but increasingly also for passenger entertainment. This could be, for example, a central display on the dashboard visible to the driver and front passenger, one or more displays for rear passengers, or even a display specifically for the front passenger.

[0004] Such displays are often designed differently in various vehicle models or series, meaning they can have different sizes, geometries, and resolutions. For example, in addition to purely rectangular displays, displays with other form factors are increasingly common, such as round displays or even those with other non-rectangular geometries. The displays within a single vehicle are also typically different. To display content, especially backgrounds, across the entire screen, this content must be created specifically for each display type. For instance, content created for a rectangular display of a certain size cannot simply be displayed on a smaller, larger, or even round display, as edges would be cut off, black borders would appear, or the content would be distorted.

[0005] Thus, a limited selection of display content is currently created, all of which can be based on the same source content but are adapted to the different display types. This is both time-consuming and costly. Furthermore, such content is typically created only once, for example, before a vehicle is delivered, and is therefore permanently assigned to a specific vehicle and display type and cannot be changed. Flexible adaptation to current or even external application content, synchronization with new or personal content, consideration of driver preferences or driving situations, and the like are not possible with such a static collection of display content. While display content could potentially be updated via an online software update, this is complex and expensive and does not occur in real time, ultimately resulting in yet another static collection of display content.

[0006] The present invention is based on the objective of generating display content for a display device, in particular in a vehicle, which can be flexibly adapted to the display device.

[0007] The solution to this problem is achieved according to the teaching of the independent claims. Various embodiments and further developments of the invention are the subject of the dependent claims.

[0008] A first aspect of the invention relates to a method, particularly a computer-implemented one, for generating display content for a display device in a vehicle. In this method, an output content is provided as the basis for generating display content for the display device, and at least one vehicle parameter is acquired. Starting from the output content, a display content for the display device is generated, the generation being carried out using a generative machine learning model, taking into account the at least one vehicle parameter, such that the generated display content represents the output content and is adapted to the at least one vehicle parameter. The generated display content can then be displayed on the display device.

[0009] The aforementioned method, as described in the first aspect, is therefore based in particular on the fact that at least one vehicle parameter is taken into account when creating display content in order to adapt an input content accordingly. A generative machine learning model is used for this purpose. Thus, starting from any input content, such as current multimedia content or even external content, a suitable display content for the display device can be flexibly generated without the need for pre-defined display content.

[0010] In general, the first aspect of the process is based on the fundamental idea of ​​providing automatic generation, for example, of background images. By using generative artificial intelligence, flexible and automatic generation can occur without prior intervention from the user or a developer. As will be described in more detail below, the first aspect of the process, by considering one or more vehicle parameters, makes it possible to adapt display content to current conditions at any time, incorporate changing circumstances, and thus create an improved user experience.

[0011] The term "vehicle" used here refers in particular to a passenger car, including all types of motor vehicles, hybrid and battery-powered electric vehicles, as well as vehicles such as sedans, vans, buses, trucks, delivery vans and the like.

[0012] The term "display device," as used herein, refers in particular to a device for displaying images or videos, i.e., content that a user can visually perceive. This may include a display, a screen, and the like. In particular, a display device may be located in a vehicle. Displays and the like of other terminal devices, such as mobile devices like smartphones, may also be referred to as "display devices."

[0013] The term "form factor," as used herein, refers specifically to the physical size, shape, and general dimensions of a display device. This can include various aspects such as the aspect ratio and diagonal of the display. The "form factor" is also referred to herein as the "geometry" of the display device. Furthermore, a display device can be characterized by its resolution or number of pixels.

[0014] The term "display content," as used herein, refers in particular to visually representable content that can be displayed on a display device so that it can be visually perceived by a user. This can be an image, a graphic, an animation, and the like. The display content can be static or moving, and in particular can also be a video or a video sequence.

[0015] The term "source content," as used herein, refers specifically to data that may not be directly displayed or intended to be displayed as "display content" (even though this may be possible in principle). Rather, the source content serves as a basis for generating actual display content, particularly through generative artificial intelligence and taking into account further parameters, which can then be displayed on a display device.

[0016] The term "vehicle parameters" as used here refers specifically to parameters that pertain to a vehicle as such, as well as its operation, including the user in the vehicle. These can be physical parameters of the vehicle, such as dimensions, layout, etc., of the vehicle or parts thereof, including one or more display devices. Vehicle parameters can also include (possibly measurable by means of appropriate sensors) driving parameters, such as vehicle speed, driving situation, ADAS level (degree of autonomous vehicle control), etc. User parameters can be captured as vehicle parameters, for example, via a driver monitoring system, which may include at least one camera in the vehicle's interior.This allows characteristics of a user, especially the driver, to be recorded, such as mood or state of mind, in particular attention status such as alertness or fatigue, direction of gaze, identity and the like.

[0017] The term "generative artificial intelligence" or "generative machine learning mode" used here refers specifically to artificial intelligence (i.e., a machine learning method) capable of generating texts, images, videos, or other data using generative models. This involves taking into account input data, such as output content (image, video) and at least one vehicle parameter. Generative AI models learn the patterns and structure of input training data and then generate new data with similar characteristics. In this particular case, generative artificial intelligence is used to generate display content for display devices.

[0018] Any terms used herein, such as "comprises," "includes," "features," "has," "with," or any other variant thereof, are intended to cover non-exclusive inclusion. For example, a method or apparatus that includes or features a list of elements is not necessarily limited to those elements but may include other elements not expressly listed or inherent in such method or apparatus.

[0019] Furthermore, unless explicitly stated otherwise, "or" refers to an inclusive or and not an exclusive "or". For example, a condition A or B is satisfied by one of the following: A is true (or present) and B is false (or not present), A is false (or not present) and B is true (or present), and both A and B are true (or present).

[0020] The terms "ein" or "eine," as used here, are defined as "one or more." The terms "ein anderer" and "ein Weitere," as well as any other variant thereof, are to be understood as "at least one more."

[0021] The term "plural", as used here, is to be understood in the sense of "two or more".

[0022] The terms "configured" or "set up" to perform a specific function (and their respective variations) are understood within the meaning of the invention to mean that the corresponding device already exists in a configuration or setting in which it can perform the function, or at least that it is adjustable—i.e., configurable—so that it can perform the function after appropriate adjustment. Configuration can be achieved, for example, by adjusting parameters of a process sequence or by using switches or similar devices to activate or deactivate functionalities or settings. In particular, the device can have several predetermined configurations or operating modes, so that configuration can be carried out by selecting one of these configurations or operating modes.

[0023] Preferred embodiments of the method are described below, which, unless expressly excluded or technically impossible, can be combined with each other and with the other described aspects of the invention as desired.

[0024] In some embodiments, at least one form factor of the display device is captured as a vehicle parameter, defining a size and / or shape of the display device. The output content is a display element with a format unsuitable for the display device's form factor. The display element is then created in such a way that its format matches the display device's form factor. In this way, any output content can be adapted to the display device, particularly so that the display content can be shown across the entire display area, meaning that neither edges need to be cut off, nor black (unused) borders added, nor does the display content need to be distorted. It is understood that, in addition to the form factor, the resolution or number of pixels of the display device can also be advantageously taken into account.

[0025] In some embodiments, capturing the source content includes analyzing it to identify at least one content parameter representing at least one essential feature of the source content. When creating the display content, this content parameter is taken into account in such a way that the display content includes the essential feature. This can advantageously also be done using a machine learning model. In particular, this ensures that the display content not only looks appealing but also retains relevant content. The content parameter can include at least one of the following: the type of source content, at least one essential image component, at least one essential text component, or metadata of the source content (especially if the source content is multimedia content).For example, relevant text can be placed at the edge of a rectangular source content, which can then be appropriately integrated into a circular display. Additional parameters, such as the genre of a music track, current volume, etc., can also be incorporated into the design of the display content to further enhance the user experience.

[0026] In some embodiments, at least one driving parameter is captured as a vehicle parameter, representing at least one characteristic of the vehicle's current journey, in particular a journey during which the display content is to be created (and displayed). The driving parameter can include at least one of the following: driving situation (e.g., whether the vehicle is moving or stationary), navigation information (e.g., route information, destination information), vehicle speed, driving mode, journey duration, current time (clock time or, in particular, time of day), weather information (weather forecast or current weather). External cameras or other external or internal vehicle sensors can be used to capture driving parameters, especially environmental conditions, which can be incorporated into the creation of the display content. The display content can be further refined by taking driving parameters into account. For example,It should be intended that the display content only contains moving components when the vehicle is stationary or a corresponding driving autonomy mode is active, in order not to distract the driver.

[0027] In some embodiments, at least one user parameter is recorded as a vehicle parameter, representing at least one current characteristic of a current user. This user parameter can include at least one of the following: the user's identity, the user's mood or state of mind, the user's gaze direction, or other parameters obtained from a driver monitoring system, particularly via a camera or other (interior) sensors, as well as the user's seating position or, more generally, the seat occupancy status. The display content can thus be adapted to personal needs, which can improve both the user experience and driving safety, for example, by taking into account the driver's gaze direction and level of attention.

[0028] In some designs, at least one type and / or location of the display device in the vehicle is recorded as a vehicle parameter. This allows, for example, a distinction to be made as to whether the display device is visible to the driver or whether it serves only as entertainment for rear passengers. The display content can then be adjusted accordingly, particularly with regard to driving safety.

[0029] In some embodiments, the display content is background content for the display device, and a foreground content to be displayed is also analyzed, which is to be displayed together with the display content as background content, taking the foreground content into account when creating the display content. This can, for example, prevent relevant parts of the background from being obscured by the foreground or parts of the foreground and background from being poorly recognizable due to an unfavorable overlap.

[0030] A second aspect of the invention relates to a data processing system comprising at least one processor configured to perform the method according to the first aspect of the invention.

[0031] A third aspect of the invention relates to a computer program with instructions which, when executed on a system according to the second aspect, cause the system to execute the method according to the first aspect.

[0032] The computer program can be stored, in particular, on a non-volatile data carrier. Preferably, this is a data carrier in the form of an optical data carrier or a flash memory module. This can be advantageous if the computer program itself is to be handled independently of a processor platform on which the one or more programs are to be executed. In another implementation, the computer program can exist as a file on a data processing unit, in particular on a server, and be downloadable via a data connection, for example, the Internet or a dedicated data connection, such as a proprietary or local network. Furthermore, the computer program can comprise a plurality of interacting individual program modules.

[0033] The system described in the second aspect may accordingly have a program memory in which the computer program is stored. Alternatively, the system may also be configured to access an external computer program, for example on one or more servers or other data processing units, via a communication link, in particular to exchange data with it that is used during the execution of the procedure or computer program or represents outputs of the computer program.

[0034] The features and advantages explained in relation to the first aspect of the invention also apply accordingly to the other aspects of the invention.

[0035] Further advantages, features and possible applications of the present invention will become apparent from the following detailed description in conjunction with the drawings.

[0036] This shows:

[0037] Fig. 1 shows a flowchart of a process according to one embodiment; and

[0038] Fig. 2 shows a view of a vehicle interior;

[0039] Fig. 3 shows different examples of display form factors; and

[0040] Fig. 4 shows an example of creating display content for a round display.

[0041] The same reference numerals are used throughout the figures for the same or corresponding elements of the invention.

[0042] Figure 1 shows a flowchart of an embodiment of a method 100 for generating display content. Figure 2 shows a view of a vehicle interior 1 with three different display devices 2, 3, 4. Firstly, a central display device 2 is provided in the dashboard, which typically serves as the vehicle's main display and can show various content, such as vehicle data and settings, navigation data, and multimedia data. A display device 3 is also provided on the passenger side, primarily for the passenger's entertainment. Finally, a round display device 4 is provided in the instrument panel, which can display individual vehicle information as well as current multimedia content. Additional display devices may also be provided in the rear for the passengers' entertainment (not shown).A camera 5 serves as part of a driver monitoring system for monitoring the driver.

[0043] Figure 2 clearly illustrates the different form factors of the display devices 2, 3, 4 within a single vehicle. Figure 3 shows examples of various display sizes and form factors 10 that can occur within the vehicle fleet of a single vehicle manufacturer. This makes it clear that significant resources would have to be invested in adapting display content to each individual display before a vehicle is launched on the market.

[0044] The basic idea of ​​the present invention is to generate display content, for example in the form of background images, based on an input content for a display device. This content automatically adapts to the form factor of the display and is based on the multimedia content currently being played on a central display, a rear-seat display, a display for individual passengers such as the front passenger, or other entertainment devices in the vehicle. Furthermore, contextual information based on data from the interior camera, vehicle sensors, route planning, and live vehicle information is taken into account when creating these background images. Based on the driver's personalized information, these images can be further tailored to the driver's preferences.

[0045] By adapting the display content to the specific display device in the vehicle, black bars at the edges, for example, can be avoided. Since in-vehicle entertainment displays are relatively small (e.g., compared to home entertainment systems), the available space should be used optimally. Entertainment content should not be cropped, but rather adapted to the form factor of the car's display. Many users always want fresh and up-to-date content, not just a collection of standard images available at the point of sale. Furthermore, an immersive multimedia experience is often desired. The experience can be further enhanced for a user by highlighting their personality, expectations, and current mood (e.g., detected by the in-cabin camera), tailored to the current occasion and the context of the journey.In other words, the goal is to display the right content to the right person in the right situation.

[0046] Furthermore, the driver should not be distracted while driving. It is desirable for the display content to adapt according to the ADAS level. Some users also expect that features and display content from the vehicle can be seamlessly used on their own mobile devices, such as a smartphone, smartwatch, tablet, or others. Finally, there may be legal requirements, for example, that some entertainment content may not be restricted due to Digital Rights Management (DRM).

[0047] To meet all the aforementioned needs and requirements, display content is created from an initial input using method 100. This involves considering a number of vehicle parameters (step S2), including the geometry or form factor of the display device, as well as content parameters, driving parameters, and user parameters. While these are captured in separate steps as shown in Fig. 1, the order and combination are arbitrary. All vehicle parameters can also be captured and processed simultaneously.

[0048] In step S1, initial content is provided. This could be, for example, current multimedia content (audio, video, game). In step S2a, content parameters are derived from this. These include, for example: the type of multimedia content, cover art (as an image), genre, current volume, frequency or speed of the song, etc. If these (content) parameters are not readily available, an AI algorithm can be used to predict them.

[0049] In step S2b, driving parameters are recorded, such as driving status, time (day or night), speed, route information, driving situation / criticality (e.g., autonomous mode), etc. It can also be determined where in the vehicle the display device is located (driver, passenger, front or rear). External cameras or vehicle sensors can be used to record further driving parameters, such as the vehicle's environmental conditions.

[0050] In step S2c, user parameters or personal parameters are recorded, such as the mood of the driver / passenger via the interior camera, the driver's gaze, the reason for the journey, personal preferences of the passengers, etc.

[0051] In step S3, these vehicle parameters are fed into a generative AI algorithm that generates an image (or, in another embodiment, a moving graphic or a complete video) precisely adapted to the situation, the display device, and the content. The generated display content can then be shown on the display device (step S4).

[0052] Fig. 4 shows an example in which a display content 9 was generated for the display device 6 for a played song. The initial content 7 does not fill the area of ​​the display device 6, resulting in unused or black areas 7'. If parts are added to fill the area, an initial content 8 could be generated, but this would have to be cut off at the edges 8' because it has a different form factor (here rectangular instead of round). Only the display content 9 generated by the method using the generative machine learning model, taking into account the parameters mentioned above, fits the display device 6 perfectly. It is also conceivable that the contour graphics could be moving elements that change their speed and color depending on the intensity of the music.

[0053] Method 100 also allows external content, such as third-party apps, which are typically in a format and design defined by the third-party provider, to be adapted to the display device. For example, third-party apps are often square and difficult to adapt to different display shapes in vehicles. Method 100 now makes it possible to flexibly adapt such external content to various display devices with different form factors, so that it appears, for example, in full screen without any content being cropped or distorted. Free spaces around the original app design could then be filled, for example, simply by continuing patterns or automatically with app content using generative artificial intelligence.

[0054] In particular, the 100-step process addresses the needs outlined above. Display content is generated that is optimized for the specific display size, ensuring that the full and unique screen area is used to its full potential. Content is not unintentionally cropped, as the display format is taken into account in the generated content. Thanks to the generative AI algorithm, display content is always fresh and up-to-date, creating the image "quasi-automatically" based on current media content. There is no need to create predefined images. The user experience is enhanced by adjusting the color scheme of the generated display content based on the source content, as well as other parameters such as time of day, driving condition, and music parameters like volume and genre.By using the data collected by the interior camera, sensor information, routing information and personal preferences to increase contextual relevance for the user at that time, the personal experience can also be improved.

[0055] Taking the display's location within the vehicle into account ensures driving safety. For example, as long as the image / video is visible to the driver, it will be adjusted to prevent distractions (e.g., by reducing detail, blurring content, or omitting movement). Considering the ADAS status also allows the display content to be adapted to the situation. For instance, if Level 3+ autonomous driving is active, moving content may be permitted; otherwise, it may not.

[0056] Finally, it's also conceivable that the system could consider not only the devices integrated into the vehicle, but also the personal devices of the respective user, such as their smartphone. For this to work, the necessary information must be synchronized between the vehicle and the personal device. For example, metadata about the content currently playing in the vehicle could be wirelessly transmitted to the smartphone. Accordingly, a vehicle app on the mobile device could adjust its background based on this metadata and depending on the size and form factor of the device (e.g., square for a smartphone, round for a smartwatch, augmented reality for AR glasses, etc.).

[0057] While at least one exemplary embodiment has been described above, it should be noted that a large number of variations exist. It should also be noted that the described exemplary embodiments are merely non-limiting examples, and it is not intended to restrict the scope, applicability, or configuration of the devices and methods described herein. Rather, the preceding description will provide the person skilled in the art with guidance for implementing at least one exemplary embodiment. It is understood that various modifications to the function and arrangement of the elements described in an exemplary embodiment can be made without derogating from the subject matter defined in the appended claims and their legal equivalents.

[0058] REFERENCE MARK LIST

[0059] 100 methods for generating display content

[0060] 1 Vehicle interior 2 Display device (central display)

[0061] 3 Display device (passenger display)

[0062] 4 Display device (instrument display)

[0063] 5 cameras

[0064] 6 Display device 7 Output content

[0065] 7' unused area

[0066] 8 Initial content

[0067] 8' protruding edge

[0068] 9 generated display content 10 form factors

Claims

REQUIREMENTS 1. Method (100) for generating a display content (9) for a display device (6) in a vehicle, the method comprising: - Providing an output content (7, 8) as a basis for creating a display content (9) for the display device (6); - Recording at least one vehicle parameter; and - Creating a display content (9) for the display device (6) starting from the output content (7, 8), wherein the creation is carried out using a generative machine learning model taking into account the at least one vehicle parameter such that the generated display content (9) represents the output content (7, 8) and is adapted to the at least one vehicle parameter.

2. Method according to claim 1, wherein at least one form factor (10) of the display device (6) is detected as a vehicle parameter, which defines a size and / or shape of the display device (6), and wherein the output content (7, 8) is a display content which has a format which is unsuitable for the form factor of the display device (6), wherein the display content (9) is created such that the format of the created display content (9) fits the form factor of the display device (6).

3. Method according to claim 1 or 2, wherein the acquisition of the output content (7, 8) comprises analyzing the output content (7, 8) in order to acquire at least one content parameter which represents at least one essential feature of the output content (7, 8), wherein when creating the display content (9) the content parameter is taken into account such that the display content (9) contains the essential feature.

4. Method according to claim 3, wherein the content parameter comprises at least one of the following: type of source content, at least one essential image component, at least one essential text component, metadata of the source content.

5. Method according to one of the preceding claims, wherein at least one driving parameter is recorded as a vehicle parameter, which represents at least one feature of a current journey of the vehicle.

6. Method according to claim 5, wherein the driving parameter comprises at least one of the following: driving situation, navigation information, vehicle speed, driving mode, driving duration, current time, weather information.

7. Method according to one of the preceding claims, wherein at least one user parameter is recorded as a vehicle parameter, which represents at least one current characteristic of a current user.

8. Method according to claim 7, wherein the user parameter comprises at least one of the following: user identity, user mood, user gaze direction, user seating position, seat occupancy status.

9. Method according to one of the preceding claims, wherein at least one type and / or location of the display device in the vehicle is recorded as a vehicle parameter.

10. A method according to one of the preceding claims, wherein the display content is a background content for the display device, the method further comprising: - Analyzing foreground content to be displayed, which is to be shown together with the display content as background content, taking the foreground content into account when creating the display content.

11. Data processing system comprising at least one processor configured to perform the method according to any of the preceding claims.

12. Computer program with instructions which, when executed on a system according to claim 11, cause the system to execute the method according to any one of claims 1 to 10.

Images