Visualization of a vehicle using mixed reality

Abstract

A procedure (200) comprises steps of determining a geographical position of a motor vehicle (105) and transmitting the determined geographical position to a predetermined location (110); determining a geographical position of an observer (120); querying a geographical position of a motor vehicle (105) in the area of ​​the geographical position of the observer (120) at the predetermined location (110); and displaying a representation of the motor vehicle (105) to the observer (120) using mixed reality.

Description

[0001] The present invention relates to the creation of a view of a motor vehicle. In particular, the invention relates to the visualization of a motor vehicle using mixed reality.

[0002] An immediate impression presented to a person can be enriched with additional information, for example, using smart glasses. The blending of real and virtual elements is also known as mixed reality (MR). Depending on the intensity, number, or strength of the virtual impressions, a distinction can be made between augmented reality (AR) and augmented virtuality (AV) at the transition between real reality and virtual reality (VR).

[0003] For various applications, such as a multimedia reference work, an interactive user manual, a dynamic warning sign, or an interactive vehicle rental service, a system can be implemented that provides support to the user in the form of mixed reality. A space in which such services function is either locally limited, for example, to a section of a room or a building, or only relatively little additional information is provided to the user.

[0004] Publication DE 20 2017 102 575 U1 discloses a tablet machine and an augmented reality device characterized in that the augmented reality device is configured to establish a communication link between a user and the tablet machine, wherein the user controls the tablet machine by entering at least one command and the command is transmitted to the tablet machine via the communication link using the augmented reality device, wherein the augmented reality device is configured to generate at least one projection of an object, which is visible to the user through at least one substantially transparent screen of the augmented reality device, and the commands are entered by the user interacting with the projection.

[0005] Publication DE 10 2014 010 309 A1 discloses a device comprising a display unit to be worn by a user on the head, which is designed to shield the user's eyes from light from the surrounding environment and to display a virtual object arranged in a virtual space stereoscopically in front of the eyes, and a control device designed to generate graphic data representing the virtual object in the virtual space and to output it to the display unit for display.

[0006] German patent application DE 10 2019 208 712 A1 discloses a method for displaying an identified vehicle in an environment using a display device of a mobile device. If object data of at least one detected object in the detected environment matches identification data of the vehicle received via a communication link, at least one display symbol characterizing the identified vehicle is superimposed on the at least one detected object when the environment is displayed.

[0007] Publication DE 10 2017 202 763 A1 discloses a method for evaluating vehicle operating information that is measured and stored by a vehicle during its operation, wherein a filter unit filters out raw vehicle data from the vehicle operating information, wherein the vehicle operating information and the raw vehicle data include position data of the vehicle, wherein a fusion unit adds geographical environment data from the surroundings of the position data to the raw vehicle data in order to generate simulation data, wherein the simulation data is displayed by means of a user interface.

[0008] One of the problems underlying the present invention is to provide an improved technique for visualizing a motor vehicle using mixed reality. The invention solves this problem by means of the subject matter of the independent claims. Dependent claims describe preferred embodiments.

[0009] According to a first aspect of the present invention, a method is proposed which includes steps of determining a geographical position of a motor vehicle and transmitting the determined geographical position to a predetermined location; determining a geographical position of an observer; querying a geographical position of a motor vehicle in the area of ​​the observer's geographical position at the predetermined location; and displaying a representation of the motor vehicle to the observer using mixed reality.

[0010] In practice, the vehicle's geographical location can be stored at the point of view, and a viewer within that area can be presented with a mixed reality representation of the vehicle. In this way, a real vehicle can be enhanced and integrated into the viewer's mixed reality. The vehicle can be the central focus of the viewer's attention or be presented as a background element or scenic detail. The viewer can thus have an enhanced, immersive experience when viewing the mixed reality.

[0011] It is generally preferred that the geographical position of the vehicle also includes an orientation of the vehicle. Such a combination is also called a pose, and the representation of the vehicle can be based on the pose in a positionally correct manner.

[0012] Preferably, a region around the viewer's geographic position is predetermined, whereby the representation of the motor vehicle is only displayed to the viewer if the motor vehicle's geographic position lies within this region. The region can, in particular, comprise a circle or a sphere with a predetermined radius. Whether the motor vehicle is actually visible in the viewer's mixed reality can depend on additional factors, such as an obstacle between the viewer's position and the position of the motor vehicle.

[0013] It is particularly advantageous that the system is configured to receive and, if necessary, provide the geographic positions of numerous vehicles. A user can obtain information about a large number of vehicles in their area and, accordingly, display numerous representations of actual vehicles. If a sufficient number of vehicles participate in the process, a traffic flow can be visualized, or a journey within a traffic flow can be simulated. The simulation can be based on real-world data, optionally provided in real time. Optionally, the geographic positions of the vehicles can also be stored, allowing for the display of vehicle representations relative to the user's geographic location and a selected point in the past.This can be helpful, for example, to determine an optimal route between two points in a real-life situation.

[0014] The viewer's geographical location can be real. In one embodiment, a real view of a motor vehicle is enhanced for the viewer with additional information about the vehicle, such as its speed or model. In another embodiment, the viewer's geographical location is chosen. This allows the viewer to gain an improved impression of a place that may be very far from their actual location. For example, the viewer's actual location may be in one city, while the chosen geographical location, with respect to the representations of motor vehicles, is in a second city.

[0015] The predetermined location can provide a reference to a visual feature of the motor vehicle, with the representation being determined based on this reference. The visual feature can include, for example, a color, shape, type, or current state of the motor vehicle. The state can further include, for example, whether a vehicle door, window, hatch, or convertible top is currently open or closed. Furthermore, the predetermined location can provide information associated with the motor vehicle. The representation can be determined based on this information, or the information can be displayed within the representation area. Examples of such information include the year of manufacture, mileage, or speed of the motor vehicle.

[0016] In one embodiment, the vehicle transmits a signal regarding the optical feature or information to the predetermined location. This signal or information can be sent once, on demand, or with each transmitted position to the predetermined location. The vehicle can decide which information is provided and when. Furthermore, information can be provided that is not accessible, or only accessible with difficulty, outside the vehicle.

[0017] In another embodiment, the predetermined location determines the optical feature based on the identification of the motor vehicle. For example, the motor vehicle may be assigned an identification number, and the predetermined location may provide manufacturer information about the motor vehicle based on this identification number. This manufacturer information may include, in particular, the type, color, or equipment specifications. According to a further embodiment, information may be associated with the identification number, which the predetermined location then makes available to the viewer.

[0018] If motor vehicles participating in the process are distributed over a very large geographical area, the effort required to transmit and manage a correspondingly large number of vehicle position data points can be considerable. According to a further embodiment of the present invention, the predetermined location is assigned to a predetermined geographical area. The location then provides information only for a geographical position that lies within this geographical area. This effectively prevents excessive requests or reports from overwhelming the location. The processing or storage capacity of the predetermined location can be well planned, and anticipated traffic volume developments in the area can be taken into account early on. If the processing capacity of the predetermined location becomes insufficient, the size of the predetermined geographical area can be reduced.

[0019] Preferably, several predetermined locations are provided, each assigned to a different geographic area. These geographic areas can overlap by a predetermined amount to, for example, better represent the movement of a motor vehicle traveling from one geographic area to another. If a predetermined location receives a query regarding a geographic position outside its assigned geographic area, it can return a reference to a responsible predetermined location.

[0020] A geographical area of ​​any size can be managed more effectively using a large number of predefined locations, as described. The individual geographical areas can be easily adjusted to better reflect changes in local traffic density, for example.

[0021] The vehicle can transmit its geographic position to a predetermined entity within whose assigned geographic area the vehicle's geographic position lies. A predetermined entity that receives a geographic position from a vehicle that is not within its assigned geographic area can forward the received information to a competent predetermined entity and / or provide the vehicle with a reference to the competent entity.

[0022] The method can be carried out using a system consisting of several components. According to a further aspect of the present invention, a device, which is also referred to herein as the first device or predetermined location, comprises a first interface for communication with a control device of a motor vehicle; a second interface for communication with a viewing device; and a processing unit. The viewing device can also be referred to as the user device. The processing unit is configured to receive a geographical position of a motor vehicle; to receive a geographical position of a viewing device; and to provide the geographical position of the motor vehicle if it is within the range of the geographical position of the viewing device.

[0023] The first device can be implemented, for example, as a server, as local infrastructure, or as a service, particularly in a cloud. Preferably, the first device is assigned to a predetermined geographic area, as described in more detail above.

[0024] It is further preferred that the predetermined location be configured to receive and preferably store the geographic positions of a large number of motor vehicles. A stored geographic position of a motor vehicle can be made available to a viewer or viewing device under the described conditions.

[0025] According to yet another aspect of the present invention, a further device is proposed, which is also referred to herein as the second device or viewing device. The second device is preferably assigned to a person, who is also referred to herein as the viewer or user.

[0026] The second device comprises a means for determining a geographical position; a wireless interface for communication with a predetermined location or a first device described herein; an output device for displaying information by means of mixed reality to a person or viewer; and a processing device. The processing device is configured to transmit a specific geographical position to the predetermined location; and to receive a geographical position of a motor vehicle from the predetermined location; as well as to display a representation of the motor vehicle in the area of ​​its own geographical position.

[0027] The representation of the vehicle can be displayed in a positionally accurate representation relative to the viewer's own geographic location. It should be noted that, in a first embodiment, the viewer's geographic location can be determined by a suitable sensor or freely chosen based on other information. Optionally, the viewer's geographic location can be in a place that a real-world observer cannot reach, or can only reach with difficulty, for example, a hovering position above a major city.

[0028] The present invention further proposes a system comprising a first device, a second device, and a motor vehicle configured to determine its geographical position and transmit it to a predetermined location. A corresponding control device may be provided on board the motor vehicle for this purpose. Optionally, the system may include a plurality of motor vehicles. In one embodiment, the system comprises several first devices, preferably assigned to different geographical areas. The system may also include any plurality of second devices. A second device is configured to provide a person or viewer with an immersive impression of a location by means of mixed reality. One aspect of the mixed reality relates to the presentation of a representation of one of the aforementioned motor vehicles.

[0029] A method described herein can be carried out using one or more devices or a system described herein. A processing unit comprised of a device is preferably electronic and can, for example, include an integrated circuit, a programmable logic device, or a programmable microcomputer. The method, or a part thereof, can be implemented as a configuration or as a computer program product with program code means for the processing unit. The configuration or the computer program product can be stored on a computer-readable data carrier. Features or advantages of the method can be transferred to a device described herein, or vice versa.

[0030] The invention will now be described in more detail with reference to the attached drawings, in which: Fig. 1. a system; and Fig. 2. A flowchart of a process is illustrated.

[0031] Fig. Figure 1 shows a schematic representation of a system 100 comprising a motor vehicle 105, a first device 110, and a second device 115. The first device 110 is also referred to herein as the predetermined location or central location; the second device 115 is assigned to an observer 120 and is also referred to herein as the observer device 115.

[0032] A control device 125 is preferably provided on board the motor vehicle 105. The control device 125 comprises a processing unit 130, a wireless interface 135, and a device 140 for determining a geographic position 145 of the motor vehicle 105. The interface 135 can, for example, use a radio technology such as WLAN or cellular communication. The device 140 for determining the geographic position of the motor vehicle 105 can, in particular, include a receiver for signals from a global satellite navigation system (GNSS). The control device 125 can be specifically adapted for the purposes described herein. However, a control device 125 that is already installed on board a motor vehicle 105 for other reasons can also be used for the technology described herein.

[0033] The predetermined location 110 preferably comprises a processing unit 150, an interface 155, and an optional data storage device 160. The predetermined location 110 is configured to communicate with a motor vehicle 105 or its control unit 125, and with a viewing device 115. The predetermined location 110 can receive a message from the control unit 125 containing the geographical position 145 of the motor vehicle 105 and optionally further information. This further information may include a reference to information associated with the motor vehicle 105. This information may be of an optical nature and, for example, relate to the color or shape of the motor vehicle 105. The information may also include an identification of the motor vehicle 105, based on which the predetermined location 110 can obtain further information.In yet another embodiment, virtually any information about the motor vehicle 105 can be provided to the predetermined location 110 in order to facilitate a subsequent visualization of the motor vehicle 105 by combining or comparing information.

[0034] The viewing device 115 comprises an output device 165, a processing device 170, and an interface 175. The interface 175 enables the viewing device 115 to communicate with the predetermined location 110. The interface 175 can be wired or wireless, in particular using technologies such as WLAN, Bluetooth, or cellular networks. The output device 165 is configured to display additional information about the viewer's environment to the viewer 120 in a mixed reality manner. For this purpose, the output device 165 is typically mounted on the viewer's head and configured to provide a visual output to the viewer 120. The viewing device 115 may include a position sensor, an accelerometer, a camera, or a similar sensor that can determine the position and / or orientation of the viewer's head.The relative position of an object in the vicinity of the viewer 120 is preferably displayed in such a way that, to the viewer 120, it appears to be stationary in its vicinity with respect to its own head movement or orientation. In the illustrated embodiment, the processing device 170 is, purely by way of example, comprised of a mobile device 180, which can be assigned to the viewer 120.

[0035] It should be noted that a geographic position 185, which forms the basis for a display using the output device 165, does not necessarily correspond to the actual geographic position of the viewer 120. Rather, the geographic position 185 can be freely chosen, for example, within the context of a display process, a simulation, or a game. Of course, it is also possible to set the geographic position 185 to the actual geographic position of the viewer 120. The actual geographic position can be determined using a suitable sensor, which could, for example, be included in the mobile device 180. The sensor could, in particular, include a GNSS receiver.

[0036] It is proposed that the motor vehicle 105 or the control device 125 transmit the geographical position 145 of the motor vehicle 105 to the predetermined location 110. The transmission can occur once, on demand, periodically, or continuously. The predetermined location 110 can receive and store the positions of multiple motor vehicles 105. The viewing device 115 can, with respect to its own geographical position 185, send a query to the predetermined location 110 regarding nearby motor vehicles 105. The predetermined location 110 can determine which motor vehicles 105 meet this condition and transmit previously stored or determined information regarding one or more motor vehicles 105 to the viewing device 115. The viewer 120 can then be presented with visual representations of the corresponding motor vehicles 105 by means of the output device 165 in a mixed reality manner.

[0037] Fig. Figure 2 shows a flowchart of an exemplary method 200 according to the present invention. The illustrated method 200 involves several participants. For ease of understanding, steps associated with a motor vehicle 105 or its control device 125 are shown in the upper right area. Steps associated with the predetermined location 110 are shown in the lower right area. Steps associated with a viewing device 115 are shown in the left area. Symbols facilitate the assignment of the groups to the participants.

[0038] In step 205, the geographical position 145 of the vehicle 105 is determined. Optionally, in step 210, an indication of a visual characteristic of the vehicle 105 is determined. This characteristic does not necessarily have to correspond to the actual characteristic of the vehicle 105. Optionally, an identification of the vehicle 105 can also be determined. In step 215, the collected information is transmitted from the vehicle 105 to the predetermined location 110. The aforementioned information, in particular the geographical position, is preferably transmitted with sufficient frequency to allow the movement of the vehicle 105 within its environment to be tracked.

[0039] In step 220, information about vehicle 105 is received by the predetermined location 110. This information, particularly its geographical position, can be stored. Optionally, further information related to vehicle 105 can be obtained. Information concerning vehicle 105 can also be compared or merged. Determining further information based on received data is also possible.

[0040] In step 225, the viewing device 115 determines or selects a geographical position 185. In step 230, the position 185 is transmitted as part of a request or demand to the predetermined location 110.

[0041] The predetermined location 110 receives the geographical position 185 of the viewing device 115 in a step 235.

[0042] In step 240, motor vehicles 105 can now be determined whose geographical positions 145 lie within a predetermined area or at a predetermined distance from the geographical position 185 of the viewing device 115. If the request or demand received by the viewing device 115 in step 235 relates to a point in the past, the positions 145 of motor vehicles 105 at the specified time can be determined accordingly. In step 245, additional information about a previously determined motor vehicle 105 can be provided. Such information can, in particular, include an optical feature provided by the motor vehicle 105 in step 210. The information can now be transmitted to the viewing device 115. The information provided constitutes a response to the request from the viewing device 115 in step 230.

[0043] In step 250, the viewing device 115 can receive information about a motor vehicle 105. The information preferably includes the geographical position 145 of the motor vehicle 105 and preferably also other information.

[0044] In step 255, a relative position of the motor vehicle 105 with respect to the observer 120 or the geographical position 185 of the observation device 115 can be determined. The relative position can, in particular, include a direction, a distance, and / or an orientation of the motor vehicle 105.

[0045] In step 260, an optical representation of the motor vehicle 105 can be determined. The representation can be determined based on information associated with the motor vehicle 105. Such information can be displayed directly if it relates to an optically perceptible feature of the motor vehicle 105. In further embodiments, information can be determined, for example, graphically, symbolically, or in text form within the area of ​​the representation of the motor vehicle 105. Furthermore, the representation can be determined based on the information. For example, the size of the representation can be determined depending on the distance between positions 145 and 185.

[0046] In step 265, the specific representation can be output to the viewer 120 in the manner of a mixed reality. The representation of the motor vehicle 105 can be displayed in the viewer's 120 environment. The representation is two-dimensional or three-dimensional and, more preferably, in relation to the viewer's 120's real environment. It is particularly preferred that a displayed representation of a motor vehicle 105 is independent of the viewer's head position. The representation preferably appears to the viewer 120 to be correctly oriented with respect to its environment. Furthermore, the movement of the motor vehicle 105 can be realistically displayed in the vicinity of the viewer's geographical position 185. Reference sign 100 System 105 motor vehicles 110 first device, predetermined position 115 second device, viewer device, user device 120 viewers, users, persons 125 Control device 130 processing facilities 135 wireless interface 140 Position determination device 145 geographical position of the motor vehicle 150 processing units 155 Interface 160 data storage 165 Output device 170 processing unit 175 interface 180 mobile devices 185 geographical position 200 procedures 205 Determine geographical position 210 Determine indication of optical feature 215 pieces of information Receive and save 220 geographical positions 225 Select geographical location Submit 230 selected positions 235 selected geographical position received Determine 240 motor vehicles in the area of ​​the position 245 Provide information about specific motor vehicles Receive 250 pieces of information about a motor vehicle 255 Determine the relative position of the motor vehicle 260 Determine the representation of the motor vehicle 265 Representation using mixed reality

Claims

Method (200) comprising the following steps: - Determining a geographic position (145) of a motor vehicle (105) and transmitting the determined geographic position (145) to a predetermined location (110); - Determining a geographic position (185) of an observer (120); - Querying a geographic position (145) of a motor vehicle (105) in the area of ​​the geographic position (185) of the observer (120) at the predetermined location (110); and - Displaying a representation of the motor vehicle (105) to the observer (120) using mixed reality. Method (200) according to claim 1, wherein the predetermined location (110) is configured to receive and, if necessary, provide geographical positions of a plurality of motor vehicles (105). Method (200) according to one of the preceding claims, wherein the geographical position of the viewer (120) is selected. Method (200) according to one of the preceding claims, wherein the predetermined location (110) provides a reference to an optical feature of the motor vehicle (105); and wherein the representation is determined on the basis of the reference, wherein optionally the motor vehicle (105) transmits a reference to the optical feature to the predetermined location (110); and wherein the predetermined location (110) can determine the optical feature on the basis of an identification of the motor vehicle (105). Method (200) according to one of the preceding claims, wherein the predetermined location (110) is assigned to a predetermined geographical area; and wherein the location (110) provides information only with respect to a geographical position that lies within the geographical area; wherein optionally several predetermined locations (110) are provided, to which different geographical areas are assigned. Method (200) according to claim 5, wherein the motor vehicle (105) transmits its geographical position to a predetermined location (110) in whose assigned geographical area the geographical position of the motor vehicle (105) is located. Device (110) comprising: - a first interface (155) for communication with a control device of a motor vehicle (105); - a second interface (155) for communication with a viewing device (115); - a processing unit (150) configured to receive a geographical position of a motor vehicle (105); to receive a geographical position of a viewing device (115); and to provide the geographical position of the motor vehicle (105) if it is within the range of the geographical position of the viewing device (115). Device (110) according to claim 6, wherein the predetermined location (110) is configured to store geographical positions of a plurality of motor vehicles (105). Device (115) comprising: - a device (170) for determining a geographical position; - a wireless interface (175) for communication with a predetermined location (110); - an output device (165) for displaying information by means of mixed reality to a viewer (120); and - a processing device (170) configured to transmit a specific geographical position (185) to the predetermined location (110); to receive a geographical position (145) of a motor vehicle (105) from the predetermined location (110); and to display a representation of the motor vehicle (105) in the area of ​​its own geographical position. System (100) comprising: - at least one device (110) according to claim 7 or 8; - at least one device (115) according to claim 9; and - at least one motor vehicle (105) equipped to determine its geographical position (145) and transmit it to the predetermined location (110).

Images