Method and system for generating an augmented reality representation of a vehicle

Abstract

In a method for generating an augmented reality representation of a vehicle (102), a user input is received that characterizes the augmented reality representation. Based on the user input and using a generative model, first texture data are generated that correspond to a two-dimensional texture. Based on the first texture data, second texture data are generated that correspond to a texture adapted to the specific contours and surface details of a three-dimensional model of the vehicle (102). Furthermore, a display unit (104) for augmented reality is controlled based on the second texture data such that the adapted texture is projected onto the vehicle (102) in real time.

Description

The invention relates to a method and a system for generating an augmented reality representation of a vehicle.Augmented reality (AR), also augmented reality in German, is understood to mean the extension of a real environment by embedding virtual contents in the field of view of a user. With the aid of special terminals, for example AR glasses, smartphones or tablet computers, the virtual contents of the real environment itself or a representation of the real environment are presented superimposed, so that these are visually integrated into the real environment. Augmented reality is increasingly used in the automotive industry to improve customer experience and security. On the one hand, modern vehicles use AR-based head-up displays (HUDs) in order to be able to project important driving information, such as navigation instructions and speed limits, directly into the field of view of a driver. On the other hand, vehicles can be represented faithfully in so-called virtual showrooms by means of augmented reality. This allows potential customers to present vehicles in many different configurations without requiring the vehicles to be physically present.It is an object of the invention to specify a method and a system for generating an augmented reality representation of a vehicle, which can generate a multiplicity of different representations and in the process use little memory space and / or bandwidth.This object is achieved by a method having the features of claim 1 and by a system having the features of the independent device claim. Advantageous further developments are specified in the dependent claims.In the proposed method for generating an augmented reality representation of a vehicle, a user input is received which characterizes the augmented reality representation. Based on the user input and using a generative model, first texture data corresponding to a two-dimensional texture is generated. Based on the first texture data, second texture data is generated that corresponds to a texture matched to the specific contours and surface details of a three-dimensional model of the vehicle. Furthermore, a display unit for augmented reality is controlled on the basis of the second texture data in such a way that the adapted texture is projected onto the vehicle in real time.The method serves to allow a user to adapt the visual appearance of the vehicle to his wishes with the aid of augmented reality. For this purpose, the user input is first received, which characterizes the desired appearance. For example, the user may specify a particular theme to determine how to design the vehicle's appearance. The user input can be effected in particular in text form, in spoken form and / or in the form of an interaction with a user interface. Based on the user input, the two-dimensional texture is then first generated. The two-dimensional texture is flat and must be matched to the contours and surface details of the vehicle before it can be projected onto the vehicle. For this purpose, the second texture data are generated using a suitable texture mapping method and the three-dimensional model of the vehicle. This adapted texture is then projected onto the vehicle with the aid of the display unit for augmented reality in order to enable a visual integration of the adapted texture into the real environment.The generative model is used for generating the texture and thus the appearance of the vehicle. In particular, the generative model is a text-to-image model that has been trained to generate a texture corresponding to the desired visual appearance of the vehicle based on the user input. Known methods can be used for the training. In particular, however, a known, already trained generative model can also be used. Generative models have inherent noise, so that the output varies even with the same input. For the method, this means that each generated representation of the vehicle is unique. The method can thus meet the need of the user for personalizing his vehicle. However, since the various representations are generated individually and not provided, the method requires little memory space and / or bandwidth for this personalization.In one embodiment, the three-dimensional model of the vehicle is generated based on vehicle image data corresponding to images of the vehicle taken from different perspectives, respectively. In this embodiment, the three-dimensional model of the vehicle is photogrammetrically generated. Images of the vehicle are acquired from different perspectives, from which the three-dimensional model is generated by means of known photogramtric methods. The three-dimensional model can thus be created individual to the vehicle, whereby, for example, optical modifications of the vehicle made by the user can be taken into account when creating the augmented reality representation of the vehicle.In another embodiment, the images of the vehicle are captured using the display unit. For example, the images forming the basis for the three-dimensional model of the vehicle can be captured with a camera of AR spectacles. This enables, in particular, three-dimensional models to be created in real time while the user views the vehicle through his display unit. The process of creating the three-dimensional model is thus no longer perceptible to the user and the method is thus more user-friendly.In another embodiment, the three-dimensional model of the vehicle is generated from CAD data. In this embodiment, the three-dimensional model of the vehicle is provided in particular by the manufacturer of the vehicle. This has the advantage that steps no longer have to be taken on the user side in order to create the three-dimensional model of the vehicle.In another embodiment, the generative model is one of the following machine learning methods: a neural network, in particular a recurrent neural network, a transformer-based model, in particular a generative pre-trained transformer (GPT), a generative adversarial network (GAN), or a variational autoencoder (VAE). The foregoing machine learning methods are capable of generating complex and detailed images based on, for example, text inputs. They are thus best suited to generating the two-dimensional texture that is projected onto the vehicle in order to adapt the visual appearance of the vehicle to the desires of the user. For training the aforementioned machine learning methods, image datasets may be used as training data. These image datasets comprise in particular a large number of images covering a wide variety of objects, scenes and styles. From such image datasets, the machine learning methods may be trained to recognize and reproduce the complex patterns of the images in order to be able to generate highly detailed images similar to the originals.In another embodiment, the second texture data is generated using a UV mapping method. Two-dimensional coordinates of the texture, the so-called UV coordinates, are each assigned to points on the three-dimensional model of the vehicle. This determines how the two-dimensional texture fits on the three-dimensional model and how it extends over its surface. This allows for a precise control over how the two-dimensional texture is projected onto the complex surface of the vehicle, thereby achieving a more realistic and detailed representation.In a further embodiment, the display unit is AR spectacles, VR spectacles with AR function, a smartphone or a tablet computer. Moreover, other devices may be used as the display unit, for example, AR contact lenses. The AR glasses may be either so-called smart glasses that substantially have a display function or so-called smart glasses that are portable computers. Portable AR devices, such as AR glasses, inadvertently incorporate into the user's daily life. Mobile terminals, such as smartphones or tablet computers, can represent cost-effective alternatives to specialized AR devices.The invention further relates to a system for generating an augmented reality representation of a vehicle. The system comprises an input unit configured to receive a user input characterizing the augmented reality representation. A processing unit of the system is configured to generate, based on the user input and using a generative model, first texture data corresponding to a two-dimensional texture. The processing unit is further configured to generate, based on the first texture data, second texture data corresponding to a texture adapted to the specific contours and surface details of a three-dimensional model of the vehicle. The system further comprises a control unit which is designed to control an augmented reality display unit on the basis of the second texture data in such a way that the adapted texture is projected onto the vehicle in real time.In particular, the input unit is designed to receive the user input in text form, in the form of a voice input and / or in the form of a user interaction with a graphical user interface. For example, the input unit is a screen of the vehicle having a touch screen function or a mobile terminal connected to the system. In this case, a smartphone or a tablet computer, for example, can be used as the mobile terminal, on which an application is executed that can interact with the system. In particular, the input unit and the display unit for augmented reality can be part of an AR device, for example AR spectacles such as smart glasses. The processing unit may be implemented in various ways. For example, the processing unit may be part of the vehicle itself. Alternatively, the processing unit may also be a processing unit remote from the vehicle and may comprise, for example, one or more servers maintained by an operator of the system itself. Cloud computing offers a particularly advantageous implementation possibility. This allows the texture data to be generated in a cloud without the operator having to take care of the maintenance or scaling of the servers. This is particularly cost-effective, since calculations are performed on the basis of the actually used computing time and no possibly unused servers have to be kept available.The proposed system has the same advantages described above as the claimed method. In particular, the system can be further developed with the features of the dependent claims directed to the method. Furthermore, the method described above can be further developed with the features described in this document in connection with the system.Exemplary embodiments of the invention are explained in more detail below with reference to the figures. Shown therein are: FIG. 1 shows a schematic illustration of a system for generating an augmented reality illustration of a vehicle; and FIG. 2 shows a flow chart of a method for generating an augmented reality representation of a vehicle.FIG. 1 shows a schematic illustration of a system 100 for generating an augmented reality illustration of a vehicle 102.The system 100 allows a user to customize the visual appearance of the vehicle 102 to his or her wishes using augmented reality. For this purpose, the system 100 generates an augmented reality representation of the vehicle 102 and projects it onto the vehicle 102 using a display unit 104 for augmented reality, so that the augmented reality representation of the vehicle 102 appears visually integrated into the real environment for the user.The system 100 comprises an input unit 106 for receiving a user input characterizing the augmented reality representation. Based on the user input, the user specifies how to adjust the visual appearance of the vehicle 102. For example, the user specifies a particular motif, pattern, and / or theme that the vehicle 102 representation is to be generated based on. The input unit 106 can be part of a mobile terminal, for example, which allows the user to input the user input in particular in spoken form and / or in text form. Purely by way of example, the input unit 106 and the display unit 104 are shown in FIG. 1 as independent units. For example, the user input is via a smartphone, a tablet computer, or an input unit of the vehicle 102, such as a screen having a touchscreen function. However, the input unit 106 and the display unit 104 can also be units of the same mobile terminal, for example AR spectacles. This allows the interaction between the user and the system 100 to occur only via a single device. This improves and simplifies user experience because the need to switch between different devices is eliminated. In order to be able to receive the user input, a corresponding application can be provided on the mobile terminal, which can communicate with the remaining components of the system 100. The input unit 106 can furthermore be designed to store the user input in the form of data in a memory module 108 of the system 100, so that the user input can be further processed by further units of the system 100. This is shown in FIG. 1 by an arrow.The system 100 comprises a processing unit 110. The processing unit 110 may be a part of the same physical unit as the input unit 106 and / or the display unit 104. Preferably, however, the processing unit 110 is formed by a processing unit remote from the input unit 106 and the display unit 104, for example, one or more servers or a cloud computing environment. In the exemplary embodiment shown in FIG. 1, the processing unit 110 is designed to receive the user input from the memory module 108 of the system 100 in order to process it further. Alternatively or additionally, the processing unit 110 can also receive the user input in the form of data directly from the input unit 106.To create the augmented reality representation of the vehicle 102, the processing unit 110 first generates first texture data corresponding to a two-dimensional texture based on the user input and using a generative model. The two-dimensional texture includes one or more images corresponding to, for example, the subject, pattern, and / or theme set by the user in the user input. The generative model used to create the two-dimensional texture is, in particular, a machine learning model, preferably a text-to-image model, that can receive the user input as input. The generative model has been trained, for example, on the basis of comprehensive image data sets which correspond to a multiplicity of images of the most varied types, in particular different objects, scenes and styling directions. In order to prevent undesired or forbidden motifs from being generated by the generative model, the image datasets used for the training can be carefully selected and corrected, for example, in order to exclude specific contents. Additionally, filters and policies may be implemented during the training process and imaging.The two-dimensional texture is flat and therefore cannot be easily projected onto the vehicle 102 without the two-dimensional texture being distorted. Therefore, the processing unit 110 further processes the first texture data to generate second texture data corresponding to a texture matched to the specific contours and surface details of a three-dimensional model of the vehicle 102. In this case, the two-dimensional texture can also be modified, for example stretched or compressed, and / or at least partially newly generated in order to adapt it to the specific contours and surface details of the three-dimensional model. The three-dimensional model of the vehicle 102 can be created, for example, by the manufacturer of the vehicle 102 on the basis of CAD data and provided to the processing unit 110. Further, the three-dimensional model of the vehicle 102 may be generated from images of the vehicle 102 using photogrametric techniques, for example, by the processing unit 110. If the AR device used has a camera, this can be used to capture the images of the vehicle 102 and to transmit them to the processing unit 110 in the form of image data. The processing unit 110 may further store or buffer the first texture data and / or the second texture data in the memory module 108 of the system 100.The system 100 further comprises a control unit 112 which is designed to control the display unit 104. In particular, the control unit 112 may be a part of the same physical unit as the display unit 104. For example, the control unit 112 of the system 100 is a control unit 112 of AR glasses. To project the adjusted texture onto the vehicle 102, the controller 112 receives the second texture data from the processing unit 110 or from the memory module 108 of the system 100. The control unit 112 then controls the display unit 104 based on the second texture data such that the adjusted texture is projected onto the vehicle 102 in real time. As a result, for example, the adapted texture is projected onto a side door or the engine hood of the vehicle 102. Alternatively, the projection can also change the appearance of the entire vehicle 102. For this purpose, a specially trained generative model is possibly required that can generate such a texture. The projection of the adjusted texture onto the vehicle 102 changes its appearance as the user has specified in the user input. Thus, the system 100 allows the user to personalize the vehicle 102 according to his or her intentions.By the system 100 described with reference to FIG. 1, a method for generating an augmented reality representation of a vehicle 102 is executed. This method is described in more detail below with reference to FIG. 2.FIG. 2 shows a flow chart of the method for generating an augmented reality representation of a vehicle 102 according to an exemplary embodiment.The method generates an augmented reality representation of a vehicle 102 and projects onto the real-world vehicle 102 to alter the appearance of the vehicle 102 to the user. The method can be carried out, for example, by the system 100 described with reference to FIG. 1 and is described below purely by way of example with reference to this system 100.The method is started in step S 200. In step S 202, the user input is received. As the user input, the user may name, for example, colors, patterns, motifs, and / or style directions in which the vehicle 102 is to be fashioned. This can be done, for example, in spoken form and / or as a text input. The method may also be configured to allow the user to select from a variety of options to customize his input. For this purpose, the user can interact, for example, with a user interface which presents the plurality of options to the user. In particular, the user input may also be filtered using predetermined policies to prevent certain inputs and / or form a branded aesthetic framework.In optional step S 204, images of vehicle 102 are acquired from different perspectives. For example, the three-dimensional model of the vehicle 102 is created from these images by means of photogrametric methods. Alternatively, the three-dimensional model can be created and provided from CAD data of the vehicle 102, for example. Step S 204 may be performed at any point of the method before the three-dimensional model is needed for creating the adjusted texture.In step S 206, the first texture data is generated based on the user input and using the generative model. The generative model has the user input as input and generates the two-dimensional texture as output. For example, using a text-to-image model, an image is generated as the two-dimensional texture from the user input input input to text form or converted from speech form to text form. The randomness of the output inherent in generative models thereby causes different two-dimensional textures to be generated even with identical user input. Thus, each augmented reality representation of the vehicle 102 generated using this method is unique. This makes it possible, with the aid of the method described in this document, to provide a multiplicity of different augmented reality representations of the vehicle 102 without these having to be created in advance and thus requiring a large amount of storage space and / or consuming bandwidth.In step S 208, the two-dimensional texture is adapted such that it can be projected onto the vehicle 102 without being depicted in distorted form. For this purpose, in particular a UV mapping method is carried out. In this case, two-dimensional coordinates of the texture are first defined, the so-called UV coordinates. Points on the three-dimensional model of the vehicle 102 are then assigned to the UV coordinates. The UV mapping method determines how the two-dimensional texture is to be projected onto the three-dimensional model. Furthermore, the two-dimensional texture can also be changed in step S 208, for example stretched, compressed, cut and / or joined together anew. The two-dimensional texture can also be at least partially re-generated, for example because parts of the texture are missing and otherwise holes would result in the representation. For this purpose, step S 206 can be repeated using the original user input or a correspondingly modified user input. Through step S 208, the adjusted texture is generated from the two-dimensional texture, which is adjusted to the contours and surface details of the vehicle 102.In step S 210, the adjusted texture is projected onto the vehicle 102 in real time using the display unit 104, such that the user views the vehicle 102 in the appearance determined by his user input. By superimposing on the user's field of view, depending on the adjusted texture, various aspects of the vehicle 102, such as color or texture, may appear modified. In this case, the spatial orientation and movement of the display unit 104 determine how and where the adapted texture is projected onto the vehicle 102. Thus, an immersive experience arises for the user, in which the appearance of the vehicle 102 selected by him is seamlessly incorporated into the real environment.The method is then ended in step S 212.In the exemplary embodiment described with reference to FIGS. 1 and 2, at least the input unit 106, the processing unit 110 and the control unit 112 form the system 100 for generating an augmented reality representation of a vehicle 102. Further elements and features shown in FIGS. 1 and 2 and mentioned in the preceding description can be part of the system 100, in particular the display unit 104. Method steps described with reference to the system 100 can likewise be part of the claimed method.List of reference characters100 System 102 Vehicle 104 Display unit 106 Input unit 108 Memory module 110 Processing unit 112 Control unit

Claims

Method for generating an augmented reality representation of a vehicle (102), wherein a) a user input is received, which characterizes the augmented reality representation; b) first texture data are generated on the basis of the user input and using a generative model, which texture data correspond to a two-dimensional texture; c) second texture data are generated on the basis of the first texture data, which texture data correspond to a texture adapted to the specific contours and surface details of a three-dimensional model of the vehicle (102); and d) an augmented reality display unit (104) is controlled on the basis of the second texture data such that the adapted texture is projected onto the vehicle (102) in real time.The method of claim 1, wherein the three-dimensional model of the vehicle (102) is generated based on vehicle image data corresponding to images of the vehicle (102) taken from different perspectives, respectively.The method of claim 2, wherein the images of the vehicle (102) are captured using the display unit (104).The method of claim 1, wherein the three-dimensional model of the vehicle (102) is generated from CAD data.The method according to any of the preceding claims, wherein the generative model is one of the following machine learning methods: a neural network, in particular a recurrent neural network, a transformer-based model, in particular a generative pre-trained transformer (GPT), a generative adventrial network (GAN) or a variational autoencoder (VAE).The method of any preceding claim, wherein the second texture data is generated using a UV mapping method.The method of any preceding claim, wherein the display unit (104) is an AR glasses, an VR glasses with AR functionality, a smartphone, or a tablet computer.A system (100) for generating an augmented reality representation of a vehicle (102), comprising an input unit (106) configured to receive a user input characterizing the augmented reality representation; a processing unit (110) configured to generate, based on the user input and using a generative model, first texture data corresponding to a two-dimensional texture and generate, based on the first texture data, second texture data corresponding to a texture adapted to the specific contours and surface details of a three-dimensional model of the vehicle (102); and a control unit (112) configured to drive an augmented reality display unit (104) based on the second texture data such that the adapted texture is projected onto the vehicle (102) in real time.

Images