Augmented reality display

Abstract

The present disclosure relates to augmented reality displays. The present disclosure provides an AR system that utilizes pre-generated 3D models of the world to improve rendering of 3D graphics content for AR views of a scene (e.g., AR views of the world in front of a moving vehicle). By utilizing pre-generated 3D models, the AR system can use various techniques to enhance the rendering capabilities of the system. The AR system can obtain pre-generated 3D data (e.g., 3D tiles) from a remote source (e.g., cloud-based storage) and can utilize this pre-generated 3D data (e.g., 3D meshes, textures, and other geometric information) to augment local data (e.g., point cloud data collected by vehicle sensors) to determine more information about a scene than is possible from the local data, including information about occluded or distant regions of the scene.

Description

[0001] This application is a divisional application of the invention patent application filed on September 22, 2017, with application number 201780053115.9 and the invention title "Augmented Reality Display". Technical Field

[0002] This disclosure relates to augmented reality displays. Background Technology

[0003] Remote sensing technology provides various systems with information about their external environment. A wide range of technological applications rely on remote sensing systems and devices to operate. Furthermore, as more systems attempt to utilize larger amounts of data to perform different tasks in dynamic environments, remote sensing provides environmental data that can aid in decision-making. For example, control systems that guide machine operations can use remote sensing devices to detect objects within the workspace. Augmented reality (AR) systems, for instance, can use remote sensing devices to provide depth information about objects in the environment. In some cases, laser-based sensing technologies, such as optical ranging and detection (LiDAR), can provide high-resolution environmental data, such as depth maps, which can represent the proximity of different objects to the LiDAR sensor.

[0004] Real-time augmented reality faces various challenges as a primary display technology in vehicles traveling at varying speeds and angles in constantly changing environments. Weather conditions, sunlight, and vehicle kinematics are just a few factors that can affect rendering but also limit the overall functionality of the system. This is especially true because onboard sensors have a fixed range and algorithms are typically needed to optimize queries that impact overall quality and response time. Summary of the Invention

[0005] This disclosure describes methods and systems, for example, for augmented reality (AR) displays in vehicles. This disclosure describes an implementation of an AR system that utilizes a pre-generated stereo reconstruction or 3D model of the world to assist in anchoring and improving the rendering of AR scenes. By utilizing the stereo reconstruction of the world, the implementation of the AR system can use a variety of techniques to enhance the system's rendering capabilities. In the implementation, the AR system can acquire pre-generated 3D data (e.g., 3D tiles) derived from stereo reconstructions of the world generated over time using real-world images acquired from a large number of sources, and can use this pre-generated 3D data (e.g., 3D meshes, textures, and combinations of other geometric information) to determine more information about the scene than can be obtained from local sources (e.g., point cloud data acquired by vehicle sensors), from which AR rendering can benefit.

[0006] This disclosure describes an implementation of an AR system that can use three-dimensional (3D) mesh map data (e.g., 3D tiles reconstructed from aerial / street photography) to enhance or supplement information from vehicle sensors (e.g., LiDAR or cameras) on a head-up display. The 3D tiles can be used to compensate for sensor limitations (e.g., areas of the real environment obscured by buildings or terrain, or beyond the range of the real environment) to extend AR to the entire real environment in front of the vehicle (i.e., within the driver's field of vision). For example, a route can be displayed, including portions of the route obscured by objects or terrain in the real environment.

[0007] The implementation allows 3D elements to be projected onto the terrain without performing height queries or referencing point cloud data acquired by onboard sensors. Elements beyond the capabilities / range of onboard sensors can be rendered in augmented reality scenes. Besides being obstructed by occluded objects in the environment, sensors are limited by factors such as vehicle distance and speed. Pre-generated 3D mesh map data can compensate for these limitations because static images are available without the need for scanning and referencing point clouds.

[0008] In some implementations, sensor data can be used to provide AR content of the nearby real environment, where pre-generated 3D mesh map data is used to provide AR content for distant objects and occluded parts of the real environment.

[0009] In some implementations, pre-generated 3D mesh map data can be used to accelerate point cloud queries for nearby objects; more limited queries based on the 3D mesh map data are possible. Therefore, point cloud queries for local areas can be optimized based on 3D mesh map data for local areas.

[0010] In some implementations, if sensor data is unavailable or missing / limited for some reason (e.g., sensor malfunction, dirt on the sensor, fog, heavy rain, snow, darkness / night (for camera information), inside a tunnel or garage, obstructed by other vehicles, etc.), pre-generated 3D mesh map data may still be available and can be used to populate the missing AR content as well as more remote content.

[0011] In some implementations, knowledge of the normals to visible surfaces in a scene provided in pre-generated 3D data, along with the position of a light source (e.g., the sun), allows the AR system to determine the orientation of the surfaces relative to the light source. Using this information, the AR system can adjust the presentation of content within the AR content as elements are rendered into the augmented reality scene, making the content easier to see.

[0012] In some implementations, animated elements (e.g., virtual representations of vehicles, pedestrians, etc.) in a 3D rendered scene can be created based on pre-generated 3D data to respond to terrain and the types of ground that make up the terrain. For example, if a vehicle in the scene turns and drives behind a building, the virtual image of the vehicle can be displayed as driving up a hill behind the building, thus out of the field of view of the onboard sensors.

[0013] Pre-generated 3D mesh map data can be applied to the entire real-world environment, including the 360° views around the vehicle, behind obstructions, and beyond the horizon. Therefore, in some implementations, 3D mesh map data can be used to provide information about the environment, including unseen objects, to the sides and rear of the vehicle.

[0014] In some implementations, 3D mesh map data can be used by AR systems in driving conditions with poor / limited visibility, such as heavy fog, snow, winding mountain roads, etc., where the sensor range can be limited, for example, when projecting the route in front of the vehicle onto an AR display. For example, 3D mesh map data can be used to enhance sensor data by displaying upcoming curves or intersections. Attached Figure Description

[0015] Figure 1 This is a high-level flowchart of a method for enhancing AR displays using 3D data from stereoscopic reconstruction, based on some implementation schemes.

[0016] Figure 2 An adaptive augmented reality (AR) system and display according to some implementation schemes are shown.

[0017] Figure 3 Examples of processing 3D mesh map data and local sensor data according to some implementation schemes to generate virtual content for AR displays are illustrated.

[0018] Figure 4 3D blocks according to some implementation schemes are shown.

[0019] Figure 5 A 3D mesh is shown according to some implementation schemes.

[0020] Figure 6 An exemplary adaptive AR display based on some implementation schemes is shown.

[0021] Figure 7 Another exemplary adaptive AR display according to some implementation schemes is shown.

[0022] Figure 8 This illustrates how virtual content is adjusted in an AR display based on a real-world scene, according to some implementation schemes.

[0023] Figure 9 The illustration shows virtual content displaying animated elements in an AR display, according to some implementation schemes.

[0024] Figure 10 An example is given of an AR view of the environment used to provide passengers in a vehicle with 3D mesh map data and local sensor data, according to some implementation schemes.

[0025] Figure 11 This is a high-level flowchart of a method for adjusting AR displays using 3D grid map data, based on some implementation schemes.

[0026] Figure 12 This is a flowchart of a method for stabilizing virtual content on an AR display, according to some implementation schemes.

[0027] This specification includes references to "one embodiment" or "implementation". The use of the phrase "in one embodiment" or "in an embodiment" does not necessarily refer to the same embodiment. Specific features, structures, or characteristics may be combined in any suitable manner consistent with this disclosure.

[0028] "Comprising". This term is open-ended. As used in the appended claims, this term does not exclude additional structures or steps. Consider the following cited claim: "An apparatus comprising one or more processor units..." Such claims do not exclude the apparatus from including additional components (e.g., network interface unit, graphics circuitry, etc.).

[0029] "Configured to". Various units, circuits, or other components may be described or stated as being "configured to" perform one or more tasks. In such a context, "configured to" is used to imply a structure by indicating that the unit / circuit / component includes a structure (e.g., a circuit) that performs this one or more tasks during operation. Thus, a unit / circuit / component may be configured to perform the task even when the specified unit / circuit / component is currently inoperable (e.g., not switched on). Units / circuits / components used with the language "configured to" include hardware—e.g., circuits, memory storing program instructions that can be executed to perform the operation, etc. Referring to a unit / circuit / component as being "configured to" perform one or more tasks is explicitly intended to exclude reference to 35 U.S.SC §112(f) for that unit / circuit / component. Furthermore, "configured to" may include a general structure (e.g., a general circuit) manipulated by software and / or firmware (e.g., an FPGA or a general-purpose processor executing software) in a manner capable of performing one or more tasks to be solved. "Configured to" may also include adjusting the manufacturing process (e.g., a semiconductor manufacturing facility) to manufacture equipment (e.g., an integrated circuit) suitable for performing one or more tasks.

[0030] "First," "second," etc. As used herein, these terms serve as labels for the nouns preceding them and do not imply any type of ordering (e.g., spatial, temporal, logical, etc.). For example, a buffer circuit may be described herein as performing write operations on a "first" value and a "second" value. The terms "first" and "second" do not necessarily imply that the first value must be written before the second value.

[0031] "Based on." As used herein, this term describes one or more factors that influence the determination. This term does not exclude additional factors influencing the determination. That is, the determination may be based solely on these factors or at least partially on them. Consider the phrase "A is determined based on B." In this case, B is the factor influencing the determination of A, and such phrases do not exclude the possibility that the determination of A may also be based on C. In other instances, A may be determined solely on B. Detailed Implementation

[0032] This disclosure describes methods and systems, for example, for augmented reality (AR) displays used in vehicles. The disclosure describes implementations of AR systems that utilize pre-generated stereo reconstructions of the world to assist in anchoring and improving the rendering of AR scenes. By utilizing stereo reconstructions of the world, implementations of the AR system can use a variety of techniques to enhance the system's rendering capabilities. In one implementation, the AR system can acquire pre-generated 3D data derived from stereo reconstructions of the world generated from real-world images acquired over time from a large number of sources, and use this pre-generated 3D data (e.g., a combination of 3D meshes, textures, and other geometric information) to determine more information about the scene than can be obtained from local sources (e.g., point cloud data acquired by vehicle sensors), from which AR rendering can benefit. Furthermore, the pre-generated 3D data can be in a manageable format that can be used by the AR system to render local environments without querying point cloud data acquired at high speed by vehicle sensors. The pre-generated 3D data can be used as a wizard for point cloud queries, allowing queries to be focused and limited to the area requiring data. Pre-generated 3D data unlocks AR rendering capabilities and allows AR systems and displays for vehicles to exceed the capabilities of systems that use conventional onboard sensor-specific methods that query large amounts of point cloud data in real time.

[0033] In some implementations, pre-generated 3D data from stereoscopic 3D imaging provides a mesh reconstruction of the real-world scene in front of and / or around the vehicle or viewer, as well as geometric information that can be used to determine surface angles and lighting information without querying point cloud data acquired by local sensors. Knowing the vehicle's position and field of view, the AR system can obtain 3D tiles with appropriate information for the scene.

[0034] By utilizing pre-generated 3D data, AR systems can render elements in augmented reality scenes beyond the capabilities of available onboard sensors (e.g., LiDAR, cameras, etc.). LiDAR and other onboard sensors typically have range limitations and may also be limited by other factors such as vehicle speed, vehicle motion (e.g., turning or bouncing), and obstructions (buildings, other vehicles, terrain, etc.). Stereo imaging methods can overcome these limitations because static imaging can be obtained from pre-generated 3D data without querying point cloud data captured by onboard sensors.

[0035] The advantage of using pre-generated 3D data in AR systems is that it allows 3D elements to be projected onto the terrain without having to perform numerous height queries or reference large amounts of point cloud data collected by onboard sensors. Furthermore, 3D meshes allow AR systems to accurately detect occluded parts of the scene, such as routes behind buildings or mountains, and render virtual content in the scene accordingly.

[0036] In addition to the 3D geometry of the terrain, the normals of visible surfaces in the scene provided in the pre-generated 3D data, and knowledge of the position of light sources (e.g., the sun), allow the AR system to determine the orientation of surfaces relative to the light source (e.g., the sun). When using this information to render elements into an augmented reality scene, the AR system can perform color correction of virtual content based on lighting angles, move virtual content away from surfaces with glare, and / or mitigate sunlight or glare that might make parts of the rendered AR scene difficult to see.

[0037] In some implementations, animated elements (e.g., virtual representations of vehicles, pedestrians, etc.) in a 3D rendered scene can be created based on pre-generated 3D data to respond to terrain and the types of ground that make up the terrain. For example, if a vehicle in front of a vehicle with an AR system turns and drives behind a building, the virtual image of the vehicle can be displayed as driving up a hill behind the building, thus outside the field of view of the vehicle's onboard sensors.

[0038] As used herein, vehicles can include any type of land vehicle, such as cars, trucks, motorcycles, all-terrain vehicles, buses, trains, etc. However, it should be noted that the AR systems and methods described herein are also applicable to aircraft, helicopters, small boats, ships, etc. Furthermore, the AR systems and methods described herein are also applicable to mobile devices, such as smartphones, tablets or tablet devices, laptops or laptops, and AR / VR head-mounted displays (HMDs), such as glasses, goggles, or helmets that pedestrians or passengers in vehicles may carry or wear.

[0039] Figure 1This is a high-level flowchart of a method for enhancing AR displays using stereoscopically reconstructed 3D data, according to some implementation schemes. Elements 1010 and 1020 can be performed, for example, by a web-based 3D data system comprising one or more computing systems that acquire images (e.g., from previously collected aerial and street photography, images captured by vehicles equipped with AR system examples and / or cameras or still-life cameras, images captured by personal devices such as smartphones or tablets, etc.), stereoscopically reconstruct and otherwise process the images to generate data including 3D mesh maps of the ground and objects, surface normals, textures and other geometric information, location information (e.g., GPS coordinates), height information, timestamps, etc. The 3D data can be stored, for example, as 3D tiles in a back-end storage system such as a cloud-based storage device, where each 3D tile represents a 3D portion of the real world and is labeled with appropriate information. A front-end server and API are provided for retrieving 3D data from the back-end storage device.

[0040] As indicated in 1010, a 3D data system can acquire images of the real world. As shown in 1020, the 3D data system can stereoscopically reconstruct the images to generate 3D data (e.g., 3D tiles), and can store this 3D data (e.g., tagged with location information, height information, timestamps, etc.) in a storage system (e.g., a cloud-based storage device). As indicated by the arrow returning from 1020 to 1010, acquiring image data and generating 3D data from the acquired images can be a continuous process, wherein new images are acquired and processed to add them to the storage device or update existing 3D data when the images become available.

[0041] As noted in 1030, the AR system in the vehicle can acquire 3D data (e.g., as 3D tiles) based on the vehicle's current position and direction of travel. For example, the AR system can obtain location information (e.g., GPS coordinates) from the vehicle's positioning system, and can also obtain direction information from the vehicle. The AR system can then query the front end of a cloud-based storage device for 3D data based on the location and direction information, for example via a wireless communication link used to connect the vehicle to the Internet / cloud, to obtain 3D tiles for the area in front of or around the vehicle. 3D tiles can be obtained for the local environment within the vehicle's sensor range and for the environment in an extended range beyond the sensor range. In some embodiments, the AR system can locally store or cache previously acquired 3D tiles, for example for frequently passed or recently traveled routes, and thus can first check the cache to determine whether the 3D data corresponding to the current position in the cache is available before querying the cloud. In some embodiments, the AR system can pre-acquire 3D tiles based on the vehicle's current position, direction of travel, and speed. Therefore, the AR system can proactively retrieve 3D tiles that may be needed in the future from cloud-based storage devices, while simultaneously processing previously acquired 3D tiles for the current location and / or upcoming locations.

[0042] As noted in 1040, an AR system can query sensor data based on pre-generated 3D data to obtain 3D information about the local environment within the sensor range. The vehicle may include sensors (e.g., LiDAR, cameras, etc.) capable of generating a large number of data points (referred to as a point cloud) of the vehicle's local environment. The AR system can query this point cloud to obtain and / or generate local 3D information (e.g., the position and geometry (shape and size) of stationary and moving objects, the extent of objects, the motion / direction of moving objects, etc.). In some embodiments, pre-generated 3D data can be used to reduce or accelerate queries to the point cloud of nearby objects. For example, if pre-generated 3D data is available for a region, queries to that region of the local environment can be reduced or eliminated entirely. Furthermore, if sensor data is insufficient or missing due to occlusion, sensor limitations, or environmental conditions such as rain or fog, the acquired 3D data can be used to fill in these regions if pre-generated 3D data is available for them.

[0043] As noted in 1050, an AR system can generate 3D graphics (also referred to herein as virtual content) at least in part based on pre-generated 3D data and local 3D information obtained from sensor data. For example, virtual content may include route markings for vehicle travel, indications of occluded or partially occluded objects or terrain features (e.g., occluded buildings, occluded parts of roads or vehicle travel routes, other vehicles or pedestrians traveling behind buildings or terrain features, etc.). Virtual content may also include informative labels about objects or features in the environment, such as labels for identifying and providing information about visible and / or occluded buildings or terrain features, roads, routes, etc. The information in the labels may be obtained, for example, from existing 3D map information of the environment, obtained from a mapping / navigation system. Virtual content may also include information about the vehicle, such as location information (e.g., GPS coordinates), direction of travel, time of day, speed, etc. In some embodiments, 3D mesh data obtained from the cloud and / or from local storage may be aligned and combined with 3D mesh data generated from sensor data to generate a 3D mesh of the environment in front of or around the vehicle. The 3D graphics can then be positioned appropriately within an image generated by the AR system based on a 3D mesh. In some embodiments, surface normals, texture information, and / or colors of objects or surfaces in the environment included in pre-calculated 3D data can be combined with ambient lighting information (e.g., the sun's current position relative to the vehicle) to reposition at least a portion of the 3D graphics, change the color and / or intensity of at least a portion of the 3D graphics, or otherwise alter at least a portion of the 3D graphics. In some embodiments, the color and / or brightness (e.g., blue sky) of a region in the environment can be detected based on 3D mesh information and / or information acquired from vehicle sensors and used to reposition and / or change the color of the 3D graphics content. For example, if blue 3D graphics content is to be projected onto a blue sky area, the color of the content can be changed to provide contrast, or the content can be moved to another area. In some embodiments, terrain geometry information (e.g., the slope of an obscured road) can be used to change the orientation of animated elements (such as a graphical representation of a vehicle or pedestrian that has driven behind a building or terrain feature) such that the animated elements follow the slope of the obscured terrain feature.

[0044] As noted at 1060, the AR system projects 3D graphics onto an AR display, such as a portion of a vehicle's windshield. In some implementations, the image containing virtual content can be aligned with the field of view in front of the vehicle, for example, by aligning the image with a real-world image captured by the vehicle's camera or by aligning the image based on positioning and location information acquired from the vehicle's positioning and location components, and the aligned image can be projected onto a portion of the windshield by the AR system's projector assembly to display the 3D graphic content as a "head-up" AR display.

[0045] As indicated by the arrow returning from 1060 to 1030, the method can be a continuous process in which the AR system acquires and processes pre-generated 3D data and sensor data to update the 3D graphical content in the AR view of the scene as the vehicle moves through the environment.

[0046] Although Figure 1 An AR system is described as projecting virtual content, representing a view of the front of a vehicle, onto the vehicle's windshield as a "head-up display." However, in some implementations, the AR system may similarly generate 3D graphical content for views in other directions (side, or rear), at least in part, based on pre-generated 3D data and vehicle sensor data. This generated content can be projected onto other surfaces within the vehicle (e.g., side or rear windows) or provided to other devices supporting AR displays, such as smartphones, laptops, or tablets / tablets held by passengers, displays mounted in the vehicle (e.g., behind the front seats or in the dashboard), or head-mounted AR displays worn by passengers in the vehicle (e.g., AR glasses, goggles, helmets, etc.). On these other devices, the 3D graphical content provided by the AR system may, for example, be overlaid on a camera view of the environment obtained from an in-vehicle camera.

[0047] Figure 2 An adaptive augmented reality (AR) system and display are shown according to some implementation schemes, enabling the implementation of various AR methods described herein. Figure 2 In this context, region 20 corresponds to locally located elements (e.g., in a vehicle), while region 10 corresponds to remotely located elements that can be accessed by the AR system 122 in vehicle 20 via a wireless communication connection (e.g., on a "cloud" network).

[0048] In some implementations, remote elements may include a 3D data system 102, a 3D data repository 110, and one or more front-end servers 104. The network-based 3D data system 102 may include, but is not limited to, one or more computing systems that receive data input 114 (e.g., from previously collected aerial and street photography, images captured by vehicles equipped with AR system 122 examples and / or cameras or still-life cameras, images captured by personal devices such as smartphones or tablets, etc.), stereoscopically reconstruct and otherwise process the data input 114 to generate data including a 3D mesh map of the ground and objects, surface normals, textures and other geometric information, location information (e.g., GPS coordinates), height information, timestamps, etc. 3D data may be stored, for example, as 3D tiles in the back-end 3D data repository 110, such as a cloud-based storage device, where each 3D tile represents a 3D portion of the real world and is labeled with appropriate information. The front-end server 104 may provide an API for retrieving 3D data from the 3D data repository 110.

[0049] In some implementations, vehicle 20 may include, but is not limited to, an AR system 122, real-world sensors 130 (e.g., LiDAR sensors, cameras, etc.), positioning components 140 (e.g., GPS or other location tracking systems, compasses, etc.), and motion sensors 150 (e.g., accelerometers, active suspension sensors, etc.). Vehicle 20 also includes at least one display 160, which may be a surface (e.g., a windshield) or a screen (e.g., a display mounted on a dashboard), onto which virtual content generated by the AR system 122 may be projected or otherwise displayed.

[0050] In some implementations, the AR system 122 may include, but is not limited to, a controller 124, a communication interface 127 for querying the front-end server 114 to obtain 3D data 112, a memory for storing 3D data and other information, and a projector 126 for projecting virtual content 125 onto a display 160.

[0051] In some embodiments, controller 124 may include, but is not limited to, one or more of various types of processors, CPUs, image signal processors (ISPs), graphics processing units (GPUs), encoders / decoders, memory, and / or other components. Controller 502 may, for example, generate virtual content 125 for projection onto display 160 by projector 126 based at least in part on sensor data 132 enhanced by 3D data 112, which is obtained from a 3D data repository 110 and / or from locally stored data 128 according to positioning data 142, as described herein. In some embodiments, controller 124 may also obtain and use motion data 152 to influence the generation and projection of virtual content 125.

[0052] Communication interface 127 can be implemented using any suitable remote wireless communication technology. Projector 126 can be implemented using any suitable projection technology and can be configured to project or scan frames of virtual content 125 onto display 160.

[0053] The display 160 can display a real-world view (e.g., in front of the vehicle 20) enhanced by 3D virtual content 125, which is generated by the controller 124 and projected onto the real-world view by the projector 126. For example, the displayed virtual content may include information labels for objects in the view, indications of obscured features or objects such as obscured portions of a route, indications of distant features such as stripes or lines indicating distant portions of a route, information about the vehicle 20 such as its direction of travel and speed, or any information generally useful or beneficial to the viewer.

[0054] Figure 3 The diagram illustrates how pre-generated 3D mesh map data and local sensor data are processed according to some implementation schemes to generate virtual content for AR displays. This processing can, for example, be performed by... Figure 2 The AR system shown is executed.

[0055] Figure 3 (A) A graphical illustration shows sensor data (e.g., point cloud) 32 obtained by the vehicle's sensors (e.g., lidar), and multiple example 3D tiles 34 obtained from a remote 3D data repository and / or from local storage based on positioning and orientation information. Figure 3As shown in (A), tiles 34 may have different sizes and may overlap with other tiles 34. Sensor data 32 may include a point cloud comprising a large number of data points indicating the depth, orientation, height, and possibly other information (e.g., illumination) of points on surfaces in or around the vehicle when detected by the sensor. However, sensor data 32 may be limited by the effective range of the sensor, the speed of the vehicle, or other factors such as weather conditions and occlusion 36. In some embodiments, 3D mesh map data provided by 3D tiles 34 may be used to fill in 3D data for portions of the scene that are out of range, occluded, or insufficient or missing from sensor data 32 due to other reasons (e.g., weather conditions). In some embodiments, 3D mesh map data provided by 3D tiles 34 may also be used to accelerate the querying of sensor data 32 for nearby objects; the number of queries to the point cloud required to generate a local map or 3D mesh of the environment within the sensor's range can be significantly reduced based on the 3D mesh map data.

[0056] Figure 3 (B) A 3D mesh 38 is graphically illustrated, generated by aligning and combining 3D mesh map data provided by 3D tiles 34 with local 3D meshes generated based on sensor data 32. In some embodiments, each point in mesh 38 may correspond to a 3D point in the scene, may be represented by 3D coordinates, and may also have other associated information such as depth or distance, texture, color, visible / occluded portions, etc. In some embodiments, occluded points or surfaces in 3D mesh 38 may be detected by image processing techniques, such as ray tracing techniques that direct light from a viewer's perspective into the scene; if light hits at least one intermediate point or surface defined by the 3D mesh before hitting another point or surface, that other point or surface may be marked as occluded. Figure 3 (B) An example of 3D virtual content indicating a vehicle's route is also illustrated graphically. As shown, 3D virtual content indicating the visible portion of the route (which is within the range of the vehicle's sensors) can be generated based on grid 38. Furthermore, 3D virtual content indicating the occluded portion of the route (and therefore undetectable by the vehicle's sensors) can be generated based on grid 38. Additionally, 3D virtual content indicating the distant portion of the route (and therefore beyond the range of the vehicle's sensors) can be generated based on grid 38; it should be noted that at least a portion of the distant route may also be occluded (e.g., by trees or terrain) and therefore not visible in the real-world view on the AR display.

[0057] Figure 3(C) Graphically displaying virtual content generated from a 3D mesh projected onto a real-world scene, such as a vehicle's windshield. The display may show a real-world view (e.g., in front of the vehicle) enhanced by 3D virtual content generated by and projected onto the real-world view by an AR system. For example, the displayed virtual content may include information labels for objects in the view (e.g., labels identifying a route as "Route 12," and labels indicating terrain or other features such as "Mount Jones" and "Mount Smit"), indications of occluded features or objects such as occluded portions of the route, indications of distant features such as stripes or lines indicating distant portions of the route, information about the vehicle such as direction of travel, speed, time, etc., or any information generally useful or beneficial to the viewer.

[0058] Figure 4 , Figure 5 and Figure 6 The illustration visually demonstrates how pre-generated 3D mesh map data and local sensor data are processed according to some implementation schemes to generate virtual content for AR displays. This process can, for example, be performed by... Figure 2 The AR system shown is executed. Figure 4 , Figure 5 and Figure 6 Real-world examples involving scenes seen through a vehicle's windshield (in...) Figure 6 (as shown in the image).

[0059] Figure 4 Examples of some implementation schemes are shown in the reference. Figure 3 (A) shows and describes a 3D tile superimposed on a real-world image. Figure 4 As shown, a large number of 3D tiles 34 can be obtained from a remote 3D data repository and / or from local storage devices based on the vehicle's positioning and orientation information. These 3D tiles can have different sizes and can overlap with other tiles. The 3D tiles can be used to enhance or populate the vehicle's sensor data and / or optimize queries on point cloud data acquired by sensors.

[0060] Figure 5 Examples of some implementation schemes are shown in the reference. Figure 3 (B) shows and describes the same Figure 6 The 3D mesh corresponding to the real-world view shown. Figure 5 This illustrates that meshes can be very complex in practice. Figure 5The diagram also shows virtual content (route strips) overlaid on a grid, comprising nearby visible and occluded portions, as well as distant portions. Each portion can be represented by different colors, patterns, or graphic markers. For example, the occluded and distant portions of the route can be represented by stripes of a different color than the nearby visible portions used to indicate the route.

[0061] Figure 6 An exemplary adaptive AR display based on some implementation schemes is shown. Figure 6 This illustrates the projection onto a real-world scene, such as onto the windshield of a vehicle, in area 160. Figure 5 Virtual content generated from a 3D mesh. Viewers can see the real world enhanced by the 3D virtual content (e.g., through the windshield in front of a vehicle), which is generated by an AR system and projected onto the real-world view by the AR system. For example, the displayed virtual content may include information labels for objects in the view, indications of obscured features or objects such as stripes or colored circles indicating obscured portions of a route, indications of distant features such as stripes or colored circles indicating distant portions of a route, indications of nearby visible features such as a portion of a route indicated by a circle or stripe directly in front of the vehicle, information about the vehicle such as direction of travel, speed, time, etc., or any information that is generally useful or beneficial to the viewer.

[0062] Figure 7 Another example of adaptive AR display according to some implementation schemes is shown. In this example, circles indicate suggested routes that the viewer might want to take, and gray circles indicate portions of the route obscured by real-world features such as buildings. It should be noted that, in addition to direction, the graphical indication of the obscured route can also indicate other information to the viewer, such as changes in altitude, turns, road conditions (speed bumps, buildings, etc.), or any information about the route that might be of interest to or useful to the viewer. This example shows a route involving a left turn, a hill climb, and then a right turn in front of the vehicle.

[0063] Figure 8 The diagram graphically illustrates how virtual content is adjusted in an AR display based on a real-world scene, according to some embodiments. In some embodiments, surface normals, texture information, and / or colors of objects or surfaces in the environment included in pre-calculated 3D data can be combined with ambient lighting information (e.g., the sun's current position relative to a vehicle) to reposition at least a portion of the 3D graphics, change the color and / or intensity of at least a portion of the 3D graphics, or otherwise alter at least a portion of the 3D graphics. For example, in Figure 8In this context, the texture information of surface 800 can indicate that the surface is highly reflective (e.g., glass). The normal of surface 800, the position of the light source (sun), and the vehicle's current position and orientation can be used to determine whether sunlight will be reflected from surface 800 into the viewer's eyes, causing glare. If so, virtual content A can be moved to another location, and / or its rendering (color, intensity, size, etc.) can be altered to make the content more visible to the viewer. In some implementations, the color and / or brightness of areas in the environment can be detected based on pre-generated 3D data and / or information acquired from vehicle sensors, and used to reposition and / or change the color of the 3D graphical content. For example, in Figure 8 In this scenario, if virtual content B is blue and is to be projected onto a blue sky area, the content's color can be changed to provide contrast, or the content can be moved to another area. In some implementations, the position of a light source (e.g., the sun) in the view can be determined, for example, by using the time of day, the date of the year, and the position and direction of a vehicle, or by using information acquired from a light sensor or camera. This positional information can be used to reposition or change the appearance of the virtual content. For example, in... Figure 8 In the process, the position of the sun can be determined, and the virtual content C can be moved or changed so that the content is not close to the position of the sun or is at the position of the sun.

[0064] Figure 9 The diagram graphically illustrates virtual content displayed as animated elements in an AR display according to some embodiments. In some embodiments, if a visible object in the view (e.g., vehicle 900) is moving behind an occluded object (e.g., a building or hill), a graphical representation of an animated virtual element 902, such as a vehicle or pedestrian, can be rendered or displayed as moving behind the object in the AR view. In some embodiments, terrain geometry determined based on pre-generated 3D data (e.g., the slope of an occluded road) can be used to alter the rendering of the animated element 902 that is moving behind a building or terrain feature, such that the animated element 902 follows the contour (e.g., the slope) of the occluded terrain feature. For example, in Figure 9 In the animation, element 902A is shown continuing to the left on a generally flat road behind the building on the left, while element 902B is shown continuing to the right on a sloping uphill road behind the building on the right.

[0065] Figure 10 Examples of AR views used to provide passengers in a vehicle with an environment based on 3D mesh map data and local sensor data are illustrated in some implementation schemes. Figure 10As shown, the pre-generated 3D data 812 can be applied to the entire real-world environment around the vehicle 800 (360°), behind obstructions, and beyond the horizon. Therefore, in some implementations, the 3D data 812 can be used to provide information about the environment, including unseen objects, to the sides and rear of the vehicle 800.

[0066] While implementations are typically described as projecting virtual content onto the windshield of vehicle 800 to provide an AR view of the scene in front of vehicle 800, in some implementations, AR system 822 may generate 3D graphic content for views in other directions (side, or rear) based at least in part on pre-generated 3D data 812 and vehicle sensor data 824, and may project the generated content onto other surfaces in the vehicle (e.g., side or rear windows) or provide the generated 3D graphic content to other devices that support AR displays, such as smartphones, laptops, or tablets / tablet devices held by passenger 890, displays mounted in the vehicle (e.g., mounted behind the front seats or in the dashboard), or head-mounted AR displays worn by passenger 890 in the vehicle (e.g., AR glasses, goggles, helmets, etc.). On these other devices, the 3D graphic content provided by the AR system may, for example, be overlaid on a camera view of the environment obtained from an onboard camera. Figure 10 An example is shown where two passengers 890A and 890B are facing each other; therefore, passenger 890A is facing rearward in vehicle 800. Both passengers hold or wear AR-enabled devices 892. AR system 822 can provide an AR view to passenger 890 via a wired or wireless (e.g., Bluetooth) connection to the respective device 892. For example, rearward-facing passenger 890A may wish to view the scene in front of vehicle 800, so AR system 822 can provide an AR view of the scene in front of passenger 890A via device 892A. In some embodiments, passenger 890 can point their respective device 892 in a certain direction to obtain an AR view in that direction, or can select the view direction via an interface. In some embodiments, at least a portion of the virtual content can be interactive; for example, passenger 890 can select virtual elements to obtain more information about visible or occluded objects or terrain features in the AR view. In some embodiments, AR system 822 may include projectors for other windows (e.g., side windows) of vehicle 800, so that passenger 890 can view the AR-enhanced scene view from the other windows.

[0067] Figure 11 This is a high-level flowchart of a method for adjusting an AR display using pre-generated 3D mesh map data, based on some implementation schemes. As indicated at 1100, the AR system can, for example, obtain data from... Figure 2The system uses a cloud-based 3D data repository to obtain pre-generated 3D data (e.g., 3D tiles). These 3D tiles can collectively form a global 3D mesh or 3D model of the scene, including distant (beyond sensor range) and occluded portions. As indicated at 1102, the AR system can also acquire and process sensor data collected by vehicle sensors (e.g., LiDAR) to generate a 3D mesh or 3D model of the local (within sensor range) environment. As indicated by the arrows from 1100 to 1102, the pre-generated 3D data can be used to reduce or optimize the querying of sensor data, thereby accelerating sensor data processing. As indicated at 1110, the global and local 3D meshes can be aligned and combined by the AR system to generate a 3D mesh or 3D model of the scene extending from outside the vehicle to the horizon or beyond; the 3D mesh includes near, far, and occluded areas of the scene. As indicated at 1120, the AR system can then generate virtual content, at least in part, based on the 3D mesh described herein. As indicated at 1130, the AR system can then project virtual content onto an AR display (e.g., onto the windshield of a vehicle). As indicated by the arrow returning from element 1130, this method can be a continuous process for updating the virtual content projected onto the AR display.

[0068] Stable virtual content

[0069] Figure 12 This is a flowchart illustrating a method for stabilizing virtual content on an AR display, according to some implementation schemes. A vehicle driver may frequently travel the same routes, such as a commute to work. A variable in the environment is the road surface conditions of the road the driver is traveling on. As indicated at 1200, vehicle motion (e.g., bouncing) can be recorded along the frequently traveled routes. Figure 2 As shown, the vehicle may include motion sensors (e.g., accelerometers, active suspension sensors, etc.) that can provide motion information to the AR system. In some embodiments, the AR system may record this motion information in local storage or by providing the information to a 3D data system for recording in a 3D data repository (e.g., a cloud-based storage device). As indicated at 1210, while subsequently traversing the route, the AR system may generate virtual content to be displayed, at least in part, based on the 3D mesh as described above. In some embodiments, the recorded vehicle motion information can be used to adjust the position of the virtual content based on the recorded motion information of the route. For example, if the recorded motion information indicates a bump at a given location, the virtual content may be moved up and down according to the recorded vehicle motion to stabilize the virtual content in the driver's line of sight. As indicated at 1220, the stabilized virtual content may be projected onto the AR display.

[0070] Although specific embodiments have been described above, these embodiments are not intended to limit the scope of this disclosure, even where only a single embodiment is described with respect to a particular feature. The examples of features provided in this disclosure are intended to be illustrative and not limiting, unless otherwise stated. The scope of this disclosure includes any feature or combination of features disclosed herein (expressly or implicitly), or any extension thereof, regardless of whether it alleviates any or all the problems addressed herein. Therefore, new claims may be made for any such combination of features during the execution of this patent application (or a patent application claiming priority thereto). Specifically, referring to the appended claims, features of dependent claims may be combined with features of independent claims, and features from the respective independent claims may be combined in any suitable manner rather than solely by the specific combinations listed in the appended claims.

[0071] In different implementations, the various methods described herein may be implemented in software, hardware, or a combination thereof. Furthermore, the order of the blocks of the methods may be changed, and various elements may be added, reordered, combined, omitted, modified, etc. Various modifications and changes will be apparent to those skilled in the art who benefit from this disclosure. The various implementations described herein are intended to be illustrative and not restrictive. Many variations, modifications, additions, and improvements are possible. The boundaries between various components and operations are somewhat arbitrary, and specific operations are shown in the context of a particular exemplary configuration. Other allocations of functionality are contemplated, which may fall within the scope of the appended claims. Finally, the structure and functionality of discrete components presented in the exemplary configuration may be implemented as a combined structure or component. These and other variations, modifications, additions, and improvements may fall within the scope of the implementations as defined in the following claims.

Claims

1. A display system, comprising: Display devices; and Controller, the controller includes: One or more processors; and The memory stores instructions that, when executed on or across the one or more processors, cause the one or more processors to: Obtain pre-generated 3D data for at least a portion of a real-world scene, wherein the pre-generated 3D data includes a pre-generated 3D mesh for a corresponding region of the scene, wherein the pre-generated 3D mesh includes occluded portions of the scene, the occluded portions being obscured from the angle of one or more sensors by objects or terrain in the scene, and wherein the pre-generated 3D mesh includes distant portions of the scene outside the range of the one or more sensors. Determine one or more of the pre-generated 3D meshes, wherein the one or more of the pre-generated 3D meshes include portions of local regions within the range of the one or more sensors; Point cloud data obtained from the one or more sensors is used to generate a local 3D mesh for portions of the local region that are not included in the pre-generated 3D mesh; The local 3D mesh and the pre-generated 3D mesh are used to generate a 3D model of the scene, the 3D model of the scene including the occluded part of the scene and the distant part of the scene; Virtual content of the scene is rendered at least partially based on the 3D model, wherein the virtual content of the scene includes the occluded portion of the scene, the occluded portion having a corresponding indication in the rendered virtual content that the occluded portion of the scene is occluded by objects or terrain in the scene, and wherein the virtual content of the scene includes the distant portion of the scene, the distant portion having a corresponding indication in the rendered virtual content that the distant portion of the scene is outside the range of the one or more sensors; and The rendered virtual content is provided to the display device.

2. The display system of claim 1, wherein the point cloud data comprises a plurality of data points, wherein each data point indicates the depth, orientation, and height of a point on a surface within the local region detected by the one or more sensors.

3. The display system of claim 1, wherein the memory further comprises instructions that, when executed on or across the one or more processors, cause the one or more processors to: The occluded portions of the scene are detected, at least in part, based on ray tracing techniques performed on the pre-generated 3D mesh.

4. The display system of claim 1, wherein the pre-generated 3D data comprises one or more pre-generated 3D tiles generated based on one or more photographs of the real-world scene.

5. The display system as claimed in claim 1, wherein, In order to generate the local 3D mesh, the controller is configured to query the point cloud to obtain data points for portions of the local region that are not included in the one or more pre-generated 3D meshes, wherein no query is performed for portions of the local region that are included in the one or more pre-generated 3D meshes.

6. The display system as claimed in claim 1, wherein, The virtual content includes information tags for objects in the scene.

7. The display system as claimed in claim 1, wherein, The pre-generated 3D data includes surface normals for one or more surfaces in the scene, and wherein, in order to render virtual content of the scene at least in part based on the 3D model, the memory also includes instructions that cause the one or more processors to: Determine the direction of light from the light source in the scene; The reflection of light from the surface in the viewer's direction is determined based on the direction of the light and the surface normal of the surface; and A portion of the virtual content intended to be displayed on or near the surface will be moved to another location within the scene.

8. The display system of claim 1, wherein the pre-generated 3D data includes information including surface normals, textures, and colors for one or more surfaces in the scene, and wherein, in order to render virtual content of the scene at least in part based on the 3D model, the controller is configured to change the rendering or position of portions of the virtual content based on the surface information.

9. The display system of claim 1, wherein a corresponding indication for the obscured portion of the scene has a first indication type, and wherein a corresponding indication for the distant portion of the scene has a second indication type different from the first indication type.

10. The display system of claim 1, wherein the display device is integrated into the windshield of the vehicle.

11. The display system of claim 10, wherein the virtual content includes one or more of an indication of a route in the vicinity of the range of the one or more sensors or an indication of a route in the distance outside the range of the one or more sensors.

12. The display system of claim 10, wherein, The virtual content includes a graphical representation of a moving object occluded by objects or terrain features in the scene, wherein the controller is configured to render the graphical representation of the moving object such that the moving object corresponds to the outline of a terrain geometry determined based on the pre-generated 3D data.

13. The display system of claim 10, wherein, The memory also includes instructions that cause the one or more processors to: Record the movement of the vehicle caused by road conditions along its route; and During subsequent travel along the route, the position of the virtual content on the window is adjusted based on the recorded movement of the vehicle along the route.

14. A method performed by a display system, comprising: The display system obtains pre-generated 3D data for at least a portion of the environment, wherein the pre-generated 3D data includes a pre-generated 3D mesh for a corresponding area of ​​the environment, and wherein the pre-generated 3D mesh includes occluded portions of the scene, the occluded portions being obscured from the angle of one or more sensors by objects or terrain in the scene, and wherein the pre-generated 3D mesh includes distant portions of the scene outside the range of the one or more sensors. The display system determines one or more of the pre-generated 3D meshes, wherein the one or more of the pre-generated 3D meshes include portions of local regions within the range of the one or more sensors; The display system uses point cloud data obtained from the one or more sensors to generate a local 3D mesh for portions of the local region not included in the pre-generated 3D mesh; and The display system uses the local 3D mesh and the pre-generated 3D mesh to generate a 3D model of the environment, the 3D model of the environment including the occluded part of the scene and the distant part of the scene; The display system renders virtual content of the scene at least in part based on the 3D model, wherein the virtual content of the scene includes the occluded portion of the scene, the occluded portion having a corresponding indication in the rendered virtual content that the occluded portion of the scene is occluded by objects or terrain in the scene, and wherein the virtual content of the scene includes the distant portion of the scene, the distant portion having a corresponding indication in the rendered virtual content that the distant portion of the scene is outside the range of the one or more sensors; as well as The rendering virtual content is provided to the display device by the display system.

15. The method of claim 14, wherein the point cloud data comprises a plurality of data points, wherein each data point indicates the depth, orientation, and height of a point on a surface within the local region detected by the one or more sensors.

16. The method of claim 14, wherein the 3D model includes the distant portion of the environment outside the range of the one or more sensors and the occluded portion of the environment that is obscured from the angle of the one or more sensors by objects or terrain in the environment.

17. The method of claim 14, wherein generating the local 3D mesh includes querying the point cloud to obtain data points for portions of the local region not included in the one or more pre-generated 3D meshes, wherein no query is performed for portions of the local region included in the one or more pre-generated 3D meshes.

18. The method of claim 14, further comprising: The occluded portions of the scene are detected, at least in part, based on ray tracing techniques performed on the pre-generated 3D mesh.

19. The method of claim 14, wherein rendering virtual content comprises: The direction in which the surface reflects the light in the viewer's direction is determined based on the direction of light in the environment and the surface normal of the surface in the 3D model. as well as Move a portion of the virtual content intended to be displayed on or near the surface to another location within the window.

20. The method of claim 14, wherein the 3D model includes information including surface normals, textures, and colors for one or more surfaces in the scene, and wherein the method further includes changing the rendering or position of portions of the virtual content based on the surface information.

21. The method of claim 14, wherein the corresponding indication for the occluded portion of the scene has a first indication type, and wherein the corresponding indication for the distant portion of the scene has a second indication type different from the first indication type.

22. The method of claim 14, wherein the virtual content includes one or more of an indication of a route in the vicinity of the range of the one or more sensors or an indication of a route in the distance outside the range of the one or more sensors.

23. The method of claim 14, wherein, The virtual content includes a graphical representation of a moving object occluded by an object or terrain feature from the perspective of one or more sensors, wherein the method further includes rendering the graphical representation of the moving object such that the moving object corresponds to the outline of the terrain geometry.

24. The method of claim 14, further comprising: Record the movement of the vehicle caused by road conditions along its route; as well as During subsequent travel along the route, the position of the virtual content on the window is adjusted based on the recorded movement of the vehicle along the route.

Images