Display system for a vehicle for displaying object image data of an object in a driving scene, display network, method and computer program

Abstract

The invention relates to a display system 2 for a vehicle 5 for displaying object image data of an object 3 in a driving scene 4, comprising a sensor arrangement 6 for recording environmental data of the driving scene 4, an object provisioning arrangement 7 for providing object image data to the object 3 in the driving scene 4, an adaptation arrangement 8 for correctly displaying the object image data in the driving scene 4, and a display arrangement 9 for displaying the object image data in the driving scene 4, wherein the object provisioning arrangement 7 includes an AI object image generator device 18 for generating the object image data.

Description

[0001] The invention relates to a display system for a vehicle for displaying object image data of an object in a driving scene having the features of the preamble of claim 1. The invention also relates to a display network having a plurality of such display systems, a method and a computer program.

[0002] While older vehicles primarily display vehicle information such as speed, engine speed, temperatures, etc., modern vehicles have displays that, for example, show a selected route on a navigation system or, via rear-view cameras, display the driving area behind the vehicle with superimposed driving lines. These displays overlay the real driving areas with artificial driving lines to assist the driver in certain driving situations.

[0003] EP 3 244 591 A1 discloses a method for the virtual representation of objects in a vehicle. A server (e.g., a cloud server) receives a request from a first autonomous vehicle to provide content. In response to the request, an image analysis is performed on an image obtained from the request to determine three-dimensional (3D) positional information of the image. A list of content items is identified based on current vehicle information of the first autonomous vehicle with respect to a user profile of a user driving the first autonomous vehicle. A first content item selected from the list of content items is inserted into the image based on the 3D positional information of the image, thereby generating an augmented image.The augmented image is transmitted to the first autonomous vehicle, where the augmented image is to be displayed on a display device within the autonomous vehicle in a type of virtual reality.

[0004] It is an object of the present invention to provide a display system with improved assistance for a driver.

[0005] This object is achieved by a display system having the features of claim 1, by a display network having the features of claim 12, by a method having the features of claim 13, and by a computer program having the features of claim 15. Preferred or advantageous embodiments of the invention emerge from the subclaims, the following description, and the attached clocks.

[0006] The subject matter of the invention is a display system that is suitable and / or designed for a vehicle. The vehicle is designed, in particular, as a road vehicle, such as a passenger car, a truck, a bus, etc.

[0007] The display system is designed to display image data of an object in a driving scene. A driving scene is defined as a visible area around the vehicle, which is used, for example, for current or planned navigation. For example, the driving scene is formed by a forward viewing area or a rearward viewing area. Alternatively, the driving scene can also be formed by a lateral viewing area.

[0008] The object can be configured as a stationary object or as a mobile object. The object image data represents, in particular, an image of the object, whereby the object image data can be used to display the object, for example, on a screen.

[0009] The display system includes a sensor array for recording environmental data in the driving scene. The sensor array may include, for example, LIDAR, long-, medium-, and / or short-range radar, cameras, etc.

[0010] In particular, the sensor arrangement can comprise image sensors and / or two-dimensional sensors. The sensor arrangement is preferably assigned to the host vehicle. In other embodiments, the sensor arrangement can also be assigned to another vehicle, so that the host vehicle can utilize environmental data of its own driving scene from other vehicles.

[0011] The display system has an object provision arrangement for providing the object image data for the object in the driving scene.

[0012] Furthermore, the display system comprises an adaptation arrangement for the correct positional display of the object image data in the driving scene. Alternatively or additionally, the object image data and / or the object in the driving scene are displayed with the correct perspective. In particular, the object image data and / or the objects are in a 3-dimensional relationship with the driving scene.

[0013] The display system further comprises a display arrangement for displaying the object image data in the driving scene. Thus, a user of the display system can identify the object in the driving scene in the correct position and / or perspective using the superimposed object image data.

[0014] Within the scope of the invention, it is proposed that the object provision arrangement comprise an AI object image generator device for generating the object image data. AI stands for artificial intelligence. The AI ​​object image generator device is implemented in particular as an AI image generator that generates the object image data. The AI ​​object image generator device is designed in particular as a Generative Artificial Intelligence (GAI) and / or as a neural network. In particular, the AI ​​object image generator device can be designed as a convolved network and / or as a transformer network and / or as a diffusion model, which have the property of generating the object image data for the object, preferably from classification data of the object, which object image data can then be displayed via the display arrangement. The object image data are thus designed as artificially generated image data.For the design as a GAI, the AI ​​object image generator device is implemented in particular as a GAN (Generative Adversarial Network).

[0015] The object provision arrangement and / or the adaptation arrangement are designed in particular as digital data processing devices.

[0016] The invention has the advantage that the user of the display system is not shown pre-defined icons or the same old images of the objects. Instead, the object image data for the object in the driving scene is generated by AI, adapted to the situation, and can thus be integrated into the driving scene in a particularly realistic way. This makes it easier for the user to recognize and process the objects represented by the object image data.

[0017] In a preferred embodiment of the invention, the object image data is artificially generated by AI and / or the object is embodied as an AI art object. The object is represented in the object image data, in particular, as a real object. In particular, the object is not represented as an icon or the like; rather, the user recognizes it as real or quasi-real, so that the user can perceive the object very easily without performing any analytical activity. This allows the user to better process the object's information and react to it more quickly.

[0018] In a preferred development of the invention, the object provision arrangement comprises an object classification device. The object classification device is configured to classify the object and generate classification data. The AI ​​object image generator device is configured to generate the object image data based on the classification data. The object classification device is configured, in particular, as a further AI, specifically as a neural network. In this embodiment, it can be implemented as a convolved neural network and / or as a transformer network. In particular, the AI ​​of the object classification device is trained.

[0019] In one possible embodiment, the object provision arrangement comprises a detection device for detecting the object in the environmental data. For example, the detection device can operate based on digital image processing. The detection device is configured to generate the detection data, and the object classification device is configured to classify the object based on the detection data.

[0020] Alternatively, it can be provided that the classification device is designed to detect and classify the object on the basis of the environmental data.

[0021] It is preferred that the object classification device comprises the additional AI and a training module, wherein the training module is configured to train the AI ​​based on the environmental data and / or the detection data from the vehicle and, optionally, additionally from other vehicles. Thus, the acquired environmental data and / or detection data from other vehicles can also be used to improve and / or accelerate the training of the additional AI.

[0022] In a preferred application, the object is designed as an object that is partially or completely hidden in the driving scene - hereinafter also referred to collectively as a hidden object. The object can be hidden, for example, by other objects in the driving scene. For example, the object can be arranged behind another vehicle as a further object. It is also possible for the hidden object to be arranged behind a person in the driving scene. It is also possible for the object to be partially or completely hidden by weather influences, such as rain, fog, or splashing water. The hiding can in particular be designed as 100% hiding in the hidden area of ​​the partially or completely hidden object, by arranging another solid object in the driving scene, in particular as an interfering contour, in front of the hidden object.

[0023] Based on the object image data, the object in the driving scene is shown in its entirety. In particular, the obscured object is shown with its entire, complete, perspectively correct visible surface.

[0024] In particular, the obscured area of ​​the occluded object is shown by the object image data. The object in the driving scene can also be composed of the unobscured area of ​​the object and a region of the object image data that shows the object in the obscured area, with the two regions complementing each other to form the object.

[0025] In particular, the object is shown so completely that a viewer of the display arrangement can recognize the object and react to the object appropriately to the situation.

[0026] A further consideration is that objects are often partially or completely obscured in driving scenes. If these are completely obscured, they can be detected, for example, through a temporal analysis of the surrounding data or via surrounding data from other sensor arrangements. If the objects are partially obscured, they can be detected and recognized, for example, through object detection.

[0027] Here, digital image processing offers significant advantages over the human perception of a driving scene. When humans perceive a driving scene, partially obscured objects are overlooked, or completely obscured objects are virtually forgotten or not recognized at all, for example, if they are located behind a fog screen, which can be penetrated by the sensor array, especially a radar, but remains hidden from human perception. Because the object can be displayed on the display arrangement in the driving scene despite its partial or complete obscuration, a user of the display system can clearly recognize it and react appropriately to the situation.

[0028] The object image data can be generated by excluding non-occluded areas of the object. In particular, the object image data is then created without using graphical input data relating to the concrete and / or current object. It can be provided that the adaptation arrangement either represents the object image data completely as the object, so that the entire object is designed as a virtual object, or that the object image data fills the concealed areas of the object, so that the object is designed as a virtual-real object. Alternatively, the object image data can be generated by also using non-occluded areas of the object, so that the AI ​​object image generator device artificially expands the non-occluded areas of the object to form the entire object. In this case, the generated object image data are particularly close to the reality of the current and / or concrete object.

[0029] In one possible embodiment of the invention, the display arrangement is designed to overlay the object image data with the real driving scene in the vehicle's surroundings. For example, the user of the display system can view the real driving scene with the natural eye, with the object image data being overlaid with the particularly obscured object in the driving scene in the correct position and / or perspective.

[0030] In an alternative embodiment of the invention, the adaptation arrangement is designed to determine driving scene image data for the driving scene based on the environmental data. The driving scene image data shows the driving scene. The object image data is superimposed into the driving scene image data. In this case, an augmented reality can be formed, wherein the driving scene is represented as a real scene (for example, as images from a camera) and the object, in particular hidden, is superimposed and / or overlaid as a virtual object. Alternatively, a virtual reality can be generated, wherein both the driving scene and the object, in particular hidden, are virtualized and displayed together.

[0031] The display arrangement is designed to display the driving scene image data together with the object image data. In this embodiment of the invention, the driving scene is thus presented to the user of the display system together with the concealed object.

[0032] The obscured object can, for example, be transparent or designed like an x-ray image in the driving scene to inform the user that it is partially obscured. It is also possible for the obscured object to flash to indicate its occlusion. Furthermore, visible (unobscured) areas of the object can be displayed as they appear in reality, while obscured areas can be grayed out, shaded, highlighted, or otherwise graphically identified.

[0033] In principle, the display arrangement can be designed as a HUD (Head Up Display), HMD (Head-Mounted Display), CID (Center Information Display) or display in front of the driver or passenger.

[0034] Particularly preferably, the display arrangement comprises an HMD (Head Mounted Device), in particular AR glasses (Augmented Reality glasses), for displaying the object image data, optionally with or without the driving scene image data. This configuration has the advantage that it is easier for the display system to display the obscured object, in particular, in the correct position and / or perspective, since known technologies make it easy to determine the viewing direction in which the HMD is currently directed, and the object image data can be displayed in relation to this viewing direction.

[0035] In a possible further development of the invention, the adaptation device is configured to blend the hidden object into the driving scene when a viewing direction of the HMD is directed toward the hidden object. Alternatively or additionally, the adaptation device is configured to mask the object from the driving scene when a viewing direction of the HMD is directed away from the hidden object. Thus, the hidden object is only blended in when the viewing direction of the HMD is directed toward the hidden object. This has the advantage that the user—when looking in a different direction—is not distracted by the hidden object.

[0036] Another object of the invention relates to a display network with a plurality of display systems as described above or according to one of the preceding claims. It is provided that the display network has a central data device, wherein classification data of possible (particularly hidden) objects are stored in the data device as data sets. It is provided that the object classification device can access the data sets. In this way, it is possible for the object classification device to access a larger selection of data sets of classification data and also to utilize experience from other vehicles designed as the data sets, so that the classification data can be delivered more accurately.

[0037] Another subject of the invention relates to a method for operating the display system and / or the display network according to one of the preceding claims. In the method, object image data for at least one object are generated, in particular artificially, and the object image data are superimposed into the driving scene in the correct position.

[0038] The invention further relates to a computer program for execution, implementation, and / or application on a computer and / or on the display system and / or the display network as described above. The computer program is designed and / or configured to execute, implement, and / or apply the method according to the invention.

[0039] A further optional subject matter of the invention is a machine-readable storage medium, wherein the computer program is stored on the storage medium.

[0040] Further features, advantages, and effects of the invention will become apparent from the following description of a preferred embodiment of the invention and the accompanying figure. This figure shows: The Fig. 1 shows a schematic block diagram of a display network 1 with a plurality of display systems 2 as an embodiment of the invention. The display system 2 is used to display object image data of an object 3 in a driving scene 4, as shown, for example, as a hidden object in the following figures. Each display system 2 is assigned to a vehicle 5, as shown in the following figures. Optionally, the display systems 2 can exchange environmental data.

[0041] The display system 2 has a sensor arrangement 6, wherein the sensor arrangement 6 can comprise one or more sensors, in particular image sensors, wherein the sensor arrangement 6 monitors the driving scene 4 and determines environmental data from the driving scene 4. The environmental data can be embodied, for example, as camera images of the driving scene 4.

[0042] The display system 2 has an object provision arrangement 7, which provides the object image data for the object 3 in the driving scene 4. The object image data are at least partially or completely formed as AI-generated image data, which may be based on real models, but in the concrete embodiment are at least partially or completely formed as purely imaginary entities.

[0043] Furthermore, the display system 2 has an adaptation arrangement 8, which is designed to display, in particular to overlay, the object image data in the driving scene 4 in the correct position and / or perspective. The display system 2 has a display arrangement 9 for displaying the object image data in the driving scene 4.

[0044] The object provision arrangement 7 has an AI object image generator 18 for generating the object image data. The AI ​​object image generator 18 is embodied as a Generative Artificial Intelligence (GAI) and is, for example, designed as or comprises a GAN or a Diffusion Model. These or other AI generators, which can be used as the AI ​​object image generator 18, make it possible to generate the object image data from input data, such as environmental data or classification data. In particular, the AI ​​object image generator 18 is embodied as a text-to-image AI system. Such AI generators are known under the designations Midjourney, DALL-E, or Stable Diffusion.

[0045] The object provision arrangement 7 comprises an object classification device 19, which enables the classification of the object 3 and the generation of classification data. The classification data describe the object 3 and form the basis for the reconstruction of the hidden object 3 in the AI ​​object image generator device 18. In particular, the classification data form the input data for the reconstruction of the object 3.

[0046] The object provision arrangement 7 also has a detection device 20, wherein the detection device 20 is configured to detect the object 3 in the environmental data and to generate detection data. The object classification device 19 is configured to classify the object 3 optionally based on the environmental data and / or on the detection data.

[0047] The display network 1 can comprise a plurality of such display systems 2. The display network 1 can comprise a central data device 20 that holds data sets of classification data for various objects 3. These data sets can be made available to the object classification device 19, so that previously detected objects 3, including objects 3 detected by other display systems 2, can be made available to all object classification devices 19.

[0048] The following figures show application cases, wherein the object 3 is embodied as a partially or completely obscured object. The obscured object 3 is partially or completely obscured in the driving scene 4, for example, by another object or an interfering contour. In contrast, the object image data in the driving scene 4 shows the obscured object 3, in particular the contour, completely. Thus, the obscured object 3 is displayed at least virtually in its entirety to a user of the display arrangement 9, even though it is partially or completely obscured from the perspective of the driving scene 4 shown.

[0049] The Fig. 2 a, b, and c show an exemplary application, where the vehicle 5 is performing a reversing maneuver. The sensor arrangement 6 includes rearward-facing sensors, such as a camera, that record the driving scene 4.

[0050] In the driving scene 4, an interfering contour 10 in the form of a person is arranged, wherein the person is either present by chance or intends to assist with the reversing maneuver. A traffic cone 11 is arranged in the line of sight of the vehicle 5 and / or the sensor arrangement 6, wherein the traffic cone 11 is configured as an example of a concealed object 3 that is partially concealed by the interfering contour 10.

[0051] The concealed object 3 has a non-concealed area 12 and a concealed area 13. The display system 2 is configured to provide object image data for the concealed object 3 based on the environmental data via the object provision arrangement 7. The display arrangement 9 is configured to display the object image data in the driving scene 4, so that the concealed object 3, although located behind the interference contour 10 according to purely human perception, is shown completely in the driving scene 4, as is represented by the translucent contour of the concealed object 3. In this exemplary embodiment, the display arrangement 9 is configured as a central display 14.

[0052] The Fig. 3 a, b, c show an exemplary second application, wherein the vehicle 5 is in a highway situation. The sensor arrangement 6 has forward-facing sensors, such as a camera, which record the driving scene 4. In addition, the sensor arrangements 6 of further vehicles 15 also record environmental data of the same driving scene 4 and transmit them to the display system 2 of the vehicle 5. In the driving scene 4, one of the further vehicles 15 is designed as an interference contour 10, wherein the further vehicle 15 shields a concealed vehicle 16 as a concealed object 3 as an interference contour 10. In this exemplary embodiment, the concealed object 3 is completely concealed. The display system 2 of the vehicle 5 is designed to provide object image data for the concealed object 3 via the object provision arrangement 7 based on the environmental data from the further vehicles 15 and / or its own environmental data.The display arrangement 9 is designed as a head-up display, wherein the object image data of the real driving scene 4 are superimposed, so that the hidden object 3 is arranged behind the interference contour 10 according to purely human perception and is thus hidden, but is shown completely in the driving scene 4.

[0053] The Fig. 4 a, b, c show an exemplary third application, wherein the vehicle 5 is positioned in front of a fog wall 17 as a disturbing contour 10. The sensor arrangement 6 has forward-facing sensors, such as a camera or a radar, which record the driving scene 4. A concealed vehicle 16 is shown, which is driving behind the fog wall with respect to the line of sight of the sensor arrangement 6 of the vehicle 5. The concealed vehicle 16 can optionally be partially concealed, as shown in the Fig.4b, or even be completely obscured from human perception and only detectable by radar or the like. The display system 2 is configured to provide object image data for the obscured object 3 via the object provision arrangement 7 based on the environmental data of the sensor arrangement 6. The display arrangement 9 is configured to display the object image data in the driving scene 4, so that the obscured object 3 is shown completely in the driving scene 4. In this exemplary embodiment, the display arrangement 9 is again configured as a head-up display. List of reference symbols 1 display network 2 Display system 3 hidden object 4 Driving scene 5 vehicle 6 Sensor arrangement 7 Object provision order 8 Adaptation arrangement 9 Display arrangement 10 Interference contour 11 traffic cones 12 non-covered area 13 hidden area 14 Center display 15 additional vehicles 16 concealed vehicle 17 Fog Screen 18 AI object image generator device 19 Object classification device 20 Detection device 21 central data facility QUOTES CONTAINED IN THE DESCRIPTION

[0000] This list of documents submitted by the applicant was generated automatically and is included solely for the convenience of the reader. This list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions. Cited patent literature

[0000] EP 3 244 591 A1

[0003]

Claims

[1] Display system (2) for a vehicle (5) for displaying object image data of an object (3) in a driving scene (4) with a sensor arrangement (6) for recording environmental data of the driving scene (4), with an object provision arrangement (7) for providing object image data, the object (3) in the driving scene (4), with an adaptation arrangement (8) for the correct insertion of the object image data in the driving scene (4), with a display arrangement (9) for displaying the object image data in the driving scene (4), characterized by , that the object provision arrangement (7) has an AI object image generator device (18) for generating the object image data. [2] Display system (2) according to claim 1, characterized by that the object (3) is designed as an at least partially or completely AI art object and / or the object image data is designed as AI-generated object image data. [3] Display system (2) according to one of the preceding claims, characterized by in that the object provision arrangement (7) has an object classification device (19), wherein the object classification device (19) is designed to classify the object (3) and to generate classification data, wherein the AI ​​object image generator device (18) is designed to generate the object image data on the basis of the classification data. [4] Display system (2) according to claim 3, characterized by in that the object provision arrangement (7) has a detection device (20) for detecting the object in the environmental data, wherein the detection device (20) is designed to generate detection data, wherein the object classification device (19) is designed to classify the object (3) on the basis of the detection data. [5] Display system (2) according to claim 4, characterized bythat the object classification device (19) is designed to classify the object on the basis of the environmental data. [6] Display system (2) according to one of claims 3 to 5, characterized by that the object classification device (19) has a trainable AI and a training module, wherein the training module is designed to train the AI ​​on the basis of the environmental data and / or the detection data from the vehicle (5) or from other vehicles (15). [7] Display system (2) according to one of the preceding claims, characterized by that the object (3) is designed as an object that is partially or completely hidden in the driving scene (4) and the object (3) is shown completely on the basis of the object image data. [8] Display system (2) according to one of the preceding claims, characterized by that the display arrangement (9) is designed to superimpose the object image data of the object (3) with the real driving scene (4). [9] Display system (2) according to one of the preceding claims, characterized by in that the adaptation arrangement (8) is designed to determine driving scene image data for the driving scene (4) on the basis of the environmental data and to superimpose the object image data into the driving scene image data, wherein the display arrangement (9) is designed to display the driving scene image data with the object image data. [10] Display system (2) according to one of the preceding claims, characterized by that the display arrangement (9) has an HMD for displaying the object image data and optionally additionally the driving scene image data. [11] Display system (2) according to claim 10, characterized by that the adaptation device (8) is designed to fade the object (3) into the driving scene when a viewing direction of the HMD is directed towards the object (3) and / or to fade the object (3) out of the driving scene when a viewing direction of the HMD is directed away from the object (3). [12] Display network (1) comprising a plurality of display systems (2) according to one of the preceding claims, characterized by that the display network (1) has a central data device (21), wherein classification data are stored as data records in the central data device (21). [13] Method for operating the display system (2) and / or the display network (2) according to one of the preceding claims, wherein object image data for an object (3) is produced or generated by the use of AI and the object image data is superimposed in the correct position into the driving scene (4). [14] Computer program for execution on a computer, wherein the computer program is designed to carry out and / or implement the method according to one of the preceding claims when executed. [15] A storage medium, wherein the computer program according to claim 14 is stored on the storage medium.

Images