Method for generating video content adapted to a vehicle environment and user-specific features in a motor vehicle, as well as a machine learning model and vehicle system

Abstract

The invention relates to a method for generating video content (7) adapted to the vehicle environment and user-specific characteristics in a motor vehicle, as well as a machine learning model and a vehicle system. First, environmental data of the vehicle's surroundings are acquired using at least one sensor of the motor vehicle, wherein the environmental data includes at least GNSS (Global Navigation Satellite System) data and / or image data. In addition, at least one user-specific preference of at least one user (5) is acquired. The environmental data and the at least one user-specific preference are combined, thereby generating context data. A story is then generated from the context data using a machine language model. Finally, a visual representation of the story is generated using a machine learning model, whereby the video content (7) adapted to the vehicle environment and user-specific characteristics is generated.In this process, if environmental data is unavailable, the machine learning model generates adapted video content (7) based on the most recently acquired contextual data, which updates automatically as soon as environmental data becomes available again. The adapted video content (7) is then displayed on at least one display device in the vehicle interior (8).

Description

[0001] The invention relates to a method for generating video content adapted to a vehicle environment and user-specific in a motor vehicle, as well as a machine learning model and a vehicle system.

[0002] Motor vehicles often have an integrated (advanced) entertainment or vehicle system that provides at least one user with access to content, particularly films and / or music, and / or games. A machine learning model or artificial intelligence (AI) can be used to recommend content based on the individual preferences of that user. Furthermore, AI-generated videos or video content are becoming increasingly important and can now produce highly detailed, realistic scenes.

[0003] US patent 2021 174 590 A1 discloses a system for providing interactive content.

[0004] US patent 11 483 486 A1 discloses systems and methods for creating user-specific virtual videos.

[0005] US patent 2022 224 963 A1 discloses methods and devices for personalized output of video-based content in vehicles.

[0006] DE 10 2023 100 278 A1 discloses a method for providing graphic content.

[0007] DE 10 2023 003 005 A1 discloses a method for individually designing the display content of a display unit.

[0008] German patent DE 10 2018 213 654 A1 discloses a method for operating a mobile, portable output device in a motor vehicle. The aforementioned prior art may allow for further developments, particularly those that dynamically adapt video content to specific users.

[0009] The present invention is based on the objective of dynamically and / or in real time generating video content in a motor vehicle that is adapted to a vehicle environment and user-specific.

[0010] The problem is solved by the subject matter of the independent patent claims. Advantageous further developments of the invention are described by the dependent patent claims, the following description, and the figure.

[0011] The invention relates to a method for generating or producing video content adapted or generated to a vehicle environment and user-specific in a motor vehicle, comprising the following steps, which are carried out, for example, by means of a control device: a) Acquisition of environmental data of the vehicle environment using at least one sensor of the motor vehicle, wherein the environmental data includes at least GNSS data (Global Navigation Satellite System) and / or image data, b) Capturing at least one user-specific preference of at least one user, c) Combining or fusing the environment data and at least one user-specific preference, e.g. by means of a fusion logic of the control device, thereby generating context data, whereby a story is generated from the context data by means of a machine language model, d) Generating a visual representation of the story using a machine learning model, generating video content adapted to the vehicle environment and user-specific requirements, e) Outputting or reproducing the adapted video content on at least one display device in the vehicle interior.

[0012] In other words, the invention relates to a method by which video content is created or generated that is tailored to the environment of the motor vehicle and to at least one preference of at least one user.

[0013] The vehicle's sensor system can include, for example, a camera and / or a radar and / or lidar system and / or an ultrasonic sensor and / or a GNSS module and / or an accelerometer and / or a temperature sensor. Image data can be captured using the camera, for instance, while the GNSS data is provided by the GNSS module.

[0014] The at least one user-specific preference can include, for example, (predefined) content interests of the at least one user, such as sports and / or music and / or news and / or culture. In particular, the at least one user-specific preference can include at least one (predefined) style, such as a humorous and / or informative and / or emotional presentation of the video content. Additionally or alternatively, the at least one user-specific preference can include at least one (predefined) visual design, in particular at least one (predefined) stylistic influence, such as a retro and / or futuristic and / or documentary and / or cinematic style. In particular, the at least one user-specific preference can include at least one (predefined) genre, such as documentary and / or comedy and / or action and / or drama.

[0015] The story can be dynamically generated using the machine learning model by serving previously generated contextual data as input parameters and / or feeding it into the machine learning model, particularly into a machine language model of the machine learning model. The machine language model can include, for example, Chat-GPT®, Generative Pre-trained Transformer, and / or MS Copilot®. Specifically, the story can be generated based on internal probability distributions for natural language.

[0016] Step d) can be implemented using one or more text-to-video and / or graphics engines and / or media generation systems of the machine learning model. Such systems for generating moving image sequences from natural language input are known. One example is Pika Labs®. Further systems are disclosed, for example, at https: / / www.heise.de / news / Sechs-KI-Videogeneratoren-im-Test-10024926.html [accessed on March 26, 2025].

[0017] The generated video content can be output in step e) on at least one display device, e.g. designed as a display or screen in the vehicle interior.

[0018] Furthermore, the invention provides that, in the event of unavailability of environmental data (or sensor and / or network information), particularly when passing through a tunnel, the machine learning model generates adapted video content based on the most recently acquired contextual data and automatically updates itself as soon as environmental data becomes available again. The most recently acquired contextual data can therefore be cached and / or retrieved. Additionally, the environmental data and / or at least one user-specific preference can be stored (locally) in a buffer. As soon as new environmental data is received, the old environmental data can be overwritten. In the event of a connection loss, the most recently stored (i.e., still available) environmental data (from the buffer) can be used.

[0019] This can, for example, reduce data consumption and / or maintain the generated video content in the event of an unstable connection and / or ensure it is presented in a visually consistent manner.

[0020] The invention enables the creation of a personalized user experience. The ability to generate a story and / or video content in real time can increase the interactivity and / or engagement of at least one user. In other words, generated video content can be individually tailored to the preferences of at least one occupant and the current surroundings or vehicle environment, which can create, for example, a unique and / or immersive experience. Specifically, the system or control device can adapt to different scenarios and / or preferences (at least one user-specific preference), which can, for example, increase the variety and / or appeal of the (generated or to-be-generated) video content.

[0021] In contrast, the current state of the art has the disadvantage that dynamic adaptation to the current vehicle environment and consideration of user-specific preferences of at least one user are not provided for.

[0022] The invention also includes further developments that result in additional advantages.

[0023] Further training stipulates that the environmental data must include weather data and / or at least one traffic condition and / or at least one local point of interest. The weather data can be obtained, for example, via an interface to an online weather service and / or at least one sensor. This sensor could be, for instance, a temperature, rain, and / or light sensor. The traffic condition can be captured, for example, using GNSS data in conjunction with real-time traffic information, such as via an online traffic service and / or V2X (Vehicle-to-Everything) communication. The local point of interest can be determined, for example, using the vehicle's geolocation, such as by comparing it with a digital map of the surrounding area and / or a POI (Points of Interest) database, which contains, for example, georeferenced information about culturally and / or touristically relevant locations.The generated video content can therefore be based on the same environmental data that describes the current situation of at least one user. This allows that user to experience the generated video content as a story or film tailored to them, instead of passively consuming a generic film.

[0024] A further training program proposes that at least one user-specific preference be captured, at least partially, from existing user data relating to media content previously consumed by the respective user. This media content can include, for example, books read and / or consumed, and / or videos and / or audio content played by the user. This eliminates the need for manual input from the user, as preferences, or at least one user-specific preference, are implicitly determined through past (media) consumption behavior. In particular, the at least one user-specific preference can be derived from a streaming service and / or a media library. Since the at least one user-specific preference is derived from (already) actually consumed content, the generated video content, in particular, can reflect the actual interests of the user.

[0025] Further training envisages capturing at least one user-specific preference, at least partially, using a machine learning model. This involves recording (and / or analyzing) the user behavior of at least one user inside the vehicle and mapping it to preference categories trained within the machine learning model. In other words, the behavior of at least one user in the vehicle can be recorded. Specifically, it can be recorded where the user looks, what they click on, which media they select, and how they react to the media and / or the generated video content. This user behavior can include, in particular, the determination of gaze patterns and / or movement data of the at least one user.The machine learning model can be trained to interpret predefined gaze patterns and / or movement data to determine whether they indicate interest or disinterest on the part of at least one user in relation to the respective media content and / or generated video content. Behavioral data can be extracted from this data and fed into the machine learning model. The machine learning model can be pre-trained with (predefined) preference categories (in particular, at least one user-specific preference). User behavior, especially the behavioral data, can then be assigned to these preference categories. Thus, the machine learning model can deduce from the behavior of at least one user whether that user prefers or rejects a particular preference category.

[0026] A training program stipulates that at least one user-specific preference is captured, at least partially, through manual input from at least one user. In other words, at least one user can manually enter their user-specific preference. Specifically, they can initiate this manual input in writing, via a voice command, and / or using gesture control. Thus, at least one user can communicate or enter their user-specific preference directly. This eliminates the need for automated evaluation by a machine learning model.

[0027] Further training stipulates that the display device be designed as virtual reality glasses and / or augmented reality glasses and / or a head-mounted display (a wearable display device worn on the head) and / or as a screen. The display device can therefore be designed flexibly and / or in a variety of ways.

[0028] A training course stipulates that the adapted video content must include at least one element of the vehicle's surroundings. In other words, the generated video content incorporates a component from the real-world environment in which the vehicle is currently located. This at least one element could, for example, include a building, traffic, infrastructure, animals, or a landscape feature, particularly a lake or a forest. If, for instance, the user drives past a castle, the castle can be identified as an element and reproduced in the adapted video content. The adapted video content thus relates to the (current) surroundings or driving environment of the vehicle.

[0029] Further training stipulates that at least one element is modeled and rendered in the virtual environment created with virtual reality glasses and / or overlaid as at least one animated and / or interactive element in the vehicle interior using augmented reality glasses. In other words, at least one element can be virtually reconstructed. For example, it can be intended that at least one user wearing AR glasses sees at least one element from the vehicle environment through the AR glasses (in real life). Thus, the generated video content can include at least one element from reality and / or virtually reconstruct it. This allows the generated video content to be personalized.

[0030] Further training stipulates that at least one element is displayed as video data of the vehicle's surroundings. In other words, the generated video content can include real video recordings or camera images from the vehicle's environment, captured, for example, with a vehicle camera. This ensures that the generated video content matches the current driving situation. Furthermore, no complex 3D rendering is necessary, as real images or video data from the vehicle's surroundings are used, particularly raw and / or unfiltered footage.

[0031] One training course involves creating customized video content using real-time 3D graphics rendering. In other words, 3D graphics technology is used to calculate and / or display the customized video content in real time. Examples of such 3D graphics technology include Unreal Engine® and / or NVIDIA Omniverse®. This allows the customized video content to be generated dynamically and / or individually.

[0032] One advanced training approach envisions the story being generated and / or updated in real time and / or continuously during a journey. This is achieved by using an autoregressive model architecture (particularly with a feedback mechanism) in an iterative loop. The model provides a current output as context (input) for subsequent outputs. Previous results can thus be used as a basis for the next steps, allowing the story and / or the generated video content to be built or created gradually and / or coherently. In other words, input vectors based on previous outputs can be generated, and a video stream is created from the resulting outputs. Specifically, optical flow and / or latent temporal consistency checks can be applied.This allows for visual consistency, especially between successive frames and / or image sequences.

[0033] One advanced training approach involves storing the customized video content in the vehicle's internal memory and / or in a cloud environment. This customized video content is incorporated as part of at least one user-specific preference by weighting it with a user rating (positive or negative) and feeding it into the machine learning model as an example output of a good or bad result (generated video content). The weighting can be implemented, for example, using sentiment analysis or mood detection. It can therefore be implemented that the customized video content is weighted with a user rating (positive or negative) and fed into the machine learning model as an annotated reference for updating its model parameters. This allows the machine learning model to learn which customized video content is well-received by at least one user (e.g.,...).(e.g., showing positive user ratings) and which do not (e.g., showing negative user ratings). New video content to be generated can then be automatically better tailored based on these user ratings, in particular by the machine learning model drawing conclusions from the rated video content about preferred or rejected preference categories that characterize the generated video content.

[0034] For use cases or application situations that may arise during the procedure and are not explicitly described here, it may be provided that, according to the procedure, an error message and / or a request for user feedback is issued and / or a default setting and / or a predetermined initial state is set.

[0035] The invention also includes the machine learning model for the motor vehicle. The machine learning model can comprise a data processing device or a processor circuit configured to execute an embodiment or further development of the method according to the invention. For this purpose, the processor circuit can comprise at least one microprocessor and / or at least one microcontroller and / or at least one FPGA (Field Programmable Gate Array) and / or at least one DSP (Digital Signal Processor). In particular, a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or an NPU (Neural Processing Unit) can be used as the microprocessor. Furthermore, the processor circuit can comprise program code configured to execute the embodiment of the method according to the invention when carried out by the processor circuit.The program code can be stored in a data memory of the processor device. The processor device can be based, for example, on at least one circuit board and / or on at least one SoC (System on Chip).

[0036] The invention also includes the vehicle system. The vehicle system can comprise the machine learning model. It can be provided that the machine learning model is installed in a motor vehicle or distributed between the motor vehicle and a stationary backend of the motor vehicle.

[0037] The motor vehicle according to the invention is preferably designed as a motor vehicle, in particular as a passenger car or truck, or as a passenger bus or motorcycle.

[0038] As a further solution, the invention also includes a computer-readable storage medium comprising program code which, when executed by a computer or a computer network, causes it to execute an embodiment of the method according to the invention. The storage medium can be provided at least partially as a non-volatile data storage medium (e.g., as flash memory and / or as an SSD - solid state drive) and / or at least partially as a volatile data storage medium (e.g., as RAM - random access memory). The storage medium can be located within the computer or computer network. However, the storage medium can also be operated, for example, as an app store server and / or cloud server on the internet. The computer or computer network can provide a processor circuit with, for example, at least one microprocessor.The program code can be provided as binary code, assembly code, source code in a programming language (e.g., C), or a program script (e.g., Python). Alternatively, the computer-readable storage medium can be implemented as a signal containing computer-readable data, such as a time-varying voltage signal or a radio signal.

[0039] The invention also includes combinations of the features of the described embodiments. The invention therefore also includes realizations that each exhibit a combination of the features of several of the described embodiments, provided that the embodiments have not been described as mutually exclusive.

[0040] The following are exemplary embodiments of the invention described. This is illustrated by: Fig. a schematic representation according to an embodiment for generating video content adapted to a vehicle environment and user-specific in a motor vehicle.

[0041] The exemplary embodiments described below are preferred embodiments of the invention. In these exemplary embodiments, the described components each represent individual features of the invention, which can be considered independently of one another and each further develops the invention independently. Therefore, the disclosure is intended to include combinations of features of the embodiments other than those shown. Furthermore, the described embodiments can also be supplemented by further features of the invention already described.

[0042] In the figure, identical reference symbols denote functionally equivalent elements.

[0043] The figure shows a schematic representation according to an embodiment for generating video content 7 adapted to a vehicle environment 4 and user-specific in a motor vehicle.

[0044] The figure shows a boy (or at least a user 5) inside a motor vehicle. He holds a display device 11, designed as a tablet or tablet computer, which displays generated video content 7. Next to him, on the inside of the door, a hologram or augmented reality projection of the generated video content 7 is displayed. The hologram or augmented reality projection can be controlled, for example, from the tablet (display device 11). In other words, the generated video content 7 can be displayed as a hologram or AR projection.

[0045] According to the concept, environmental data from the vehicle's surroundings 4 can first be acquired using at least one sensor on the vehicle, with the environmental data including at least GNSS (Global Navigation Satellite System) data and / or image data. Then, at least one user-specific preference of at least one user 5 can be acquired. The environmental data and the at least one user-specific preference can be combined, thereby generating context data. A story can then be generated from the context data using a machine language model. Finally, a visual representation of the story can be generated using a machine learning model, producing video content 7 adapted to the vehicle's surroundings 4 and the user's preferences. The adapted video content 7 can then be displayed on at least one display device 11 in the vehicle's interior 8.

[0046] The idea envisions a vehicle entertainment system or vehicle system that uses AI (Artificial Intelligence) or a machine learning model to create live-generated videos or generated video content 7 based on the preferences, or at least a user-specific preference, of the occupants or at least one user 5 and the (current) vehicle environment 4. For example, as shown in the figure, a child who loves dinosaurs or dragons, for instance, could experience or be shown a personalized dinosaur story or story as generated video content 7 while driving through the desert. In particular, the story and / or the customized video content 7 can be generated in real time. The generated video content 7 can be adapted to the vehicle environment 4 and displayed on the screens or at least a display device 11 in the vehicle or as a virtual reality (VR) projection in glasses.

[0047] The idea is that environmental data and / or sensor and user data can be continuously incorporated to adapt the story and / or video content in a context-sensitive way.

[0048] The vehicle can be equipped with sensors, or at least one sensor, that detect the surroundings or vehicle environment (e.g., by means of a navigation system and / or cameras). The resulting environmental data can be analyzed and / or recorded in real time. The machine learning model can, for example, include an artificial neural network and / or AI algorithms that learn the preferences, or at least one user-specific preference, of the occupants or at least one user (e.g., through previous input or through machine learning and / or by considering books read and / or consumed in the vehicle and / or previously played video content and / or audio content (music tracks)). The at least one user-specific preference or user data can be combined with the environmental data.Based on the combined data—user data and environmental data—the machine learning model can generate a suitable story and create and / or play back corresponding video content. The story can be generated using Large Language Models (LLMs) or a language model (such as Chat-GPT®, Generative Pre-trained Transformer, and / or MS Copilot®). This can be achieved using text-to-video technologies and / or graphics engines. The generated video content can be displayed on at least one display device, on the screens in the vehicle or motor vehicle, or as a VR experience or projection in VR glasses.

[0049] This innovation can revolutionize vehicle entertainment and / or make journeys more exciting and / or entertaining for all users. In particular, it may be intended that the vehicle is designed to drive (fully) autonomously. Specifically, it may be intended that the driver is also recognized as at least one user and receives and / or is shown video content adapted to the vehicle's environment and at least one of their user-specific preferences during autonomous driving.

[0050] Overall, the examples show how AI-generated videos can be provided spontaneously and / or live using a vehicle entertainment system or vehicle system. Reference symbol list 4 Vehicle environment 5 users 7 Video content 8 Vehicle interior 11 Display device

Claims

Method for generating video content (7) adapted to a vehicle environment (4) and user-specific features in a motor vehicle, comprising the steps of: - a) capturing environmental data of the vehicle environment (4) using at least one sensor of the motor vehicle, wherein the environmental data includes at least GNSS (Global Navigation Satellite System) data and / or image data, - b) capturing at least one user-specific preference of at least one user (5), - c) combining the environmental data and the at least one user-specific preference, thereby generating context data, wherein a story is generated from the context data using a machine language model, - d) generating a visual representation of the story using a machine learning model, wherein the video content (7) adapted to the vehicle environment (4) and user-specific features is generated.wherein, in the event of unavailability of environmental data, the machine learning model generates adapted video content (7) based on the most recently acquired context data and automatically updates itself as soon as environmental data becomes available again; e) Output of the adapted video content (7) on at least one display device (11) in the vehicle interior (8) of the motor vehicle. Method according to claim 1, wherein the environmental data includes weather data and / or at least one traffic condition and / or at least one local landmark. Method according to one of the preceding claims, wherein the at least one user-specific preference is captured at least partially from existing user data of media content previously consumed by the respective user (5). Method according to one of the preceding claims, wherein the at least one user-specific preference is captured at least partially by means of the machine learning model by capturing user behavior of the at least one user (5) in the vehicle interior (8) and mapping it by means of the machine learning model with respect to preference categories trained into the machine learning model. Method according to one of the preceding claims, wherein the at least one user-specific preference is captured at least partially by means of a manual input by the at least one user (5). Method according to one of the preceding claims, wherein the display device (11) is designed as virtual reality glasses and / or augmented reality glasses and / or head-mounted display and / or as a screen. Method according to one of the preceding claims, wherein the adapted video content (7) comprises at least one element of the vehicle environment (4). Method according to claim 7, wherein the at least one element is modeled and rendered in the virtual environment generated with the virtual reality glasses according to claim 6 and / or superimposed as at least one animated and / or interactive element in the vehicle interior (8) using the augmented reality glasses according to claim 6. Method according to claim 7 or 8, wherein the at least one element is displayed as video data of the vehicle environment (4). Method according to one of the preceding claims, wherein the adapted video content (7) is created by means of real-time 3D graphics rendering. Method according to one of the preceding claims, wherein the story is generated and / or updated in real time and / or continuously during a ride by the machine learning model using an autoregressive model architecture in an iterative loop to specify an output as context for a subsequent output. Method according to one of the preceding claims, wherein the adapted video content (7) is stored in an internal vehicle memory and / or in a cloud environment, wherein the adapted video content (7) is taken into account as part of the at least one user-specific preference by weighting the adapted video content (7) with a user rating and feeding it into the machine learning model as an example output for a good or bad result. Machine learning model, wherein the machine learning model comprises a control device having a processor unit with program instructions which, when executed by the processor unit, cause it to perform a method according to one of the preceding method claims. Vehicle system comprising a machine learning model according to claim 13, wherein the machine learning model is installed in a motor vehicle or divided between the motor vehicle and a stationary backend of the motor vehicle.

Images