Procedural World Generation

Abstract

To solve a problem that existing techniques for generating a simulated environment require intensive and time-consuming manual generation in computer terms.SOLUTION: The present invention provides a system that causes to perform operation including aligning a first region associated with a road mesh and a second region associated with supplement data related to a simulated environment, determining an error between the road mesh and the supplement data, performing at least one of mixing the road mesh and the supplement data for generating data mixed to reduce the error or deforming the road mesh to substantially align the road mesh with the supplement data, and outputting the simulated environment that is used by an autonomous robot computing device and is corrected.SELECTED DRAWING: Figure 5

Description

Technical Field

[0001] The present disclosure relates to the procedural generation of worlds.

Background Art

[0002] (Priority Application) This PCT international application is a continuation and claim of priority of U.S. Patent Application No. 16 / 163,478, filed on October 17, 2018, which claims priority of U.S. Provisional Patent Application No. 62 / 716,839, entitled "Procedural World and Agent Generation", filed on August 9, 2018. Also, this PCT international application is a continuation and claim of priority of U.S. Patent Application No. 16 / 163,466, filed on October 17, 2018. The entire contents of all the foregoing applications are incorporated herein by reference.

[0003] A simulated world environment ("simulation environment") includes the rendering of various environments. Such renderings may include, for example, roads, vehicles, pedestrians, and the like. The simulation environment may be useful for enhancing training, testing, and / or system certification. Existing techniques for generating simulation environments require computationally intensive and time-consuming manual generation.

Brief Description of the Drawings

[0004] The detailed description is set forth with reference to the accompanying drawings. In the drawings, the leftmost digit of a reference number identifies the drawing in which the reference number first appears. The use of the same reference number in different drawings indicates similar or identical components or features.

[0005]

Figure 1

Figure 2A

Figure 2B

Figure 2C

Figure 2D

Figure 2E

Figure 2F

Figure 3

Figure 4

Figure 5

Figure 6

[0006] The techniques described herein are directed to various aspects of procedural world generation. That is, the techniques described herein are directed to procedurally generating a simulated world for use in testing, validating, or training systems and / or components used by vehicles that navigate, plan, and / or make decisions. In at least some of the examples described herein, the simulated world so generated can be generated to represent the environment of the real world and, in at least some examples, is as accurate as possible. The techniques described herein explain how such simulated environments can be generated.

[0007] In one example, the techniques described herein are directed to receiving sensor data from various sensor systems in an actual environment. The sensor systems may include, but are not limited to, light detection and ranging (LIDAR) sensors, radio detection and ranging (RADAR) sensors, ultrasonic transducers, sound navigation and ranging (SONAR) sensors, time-of-flight (ToF) sensors, position sensors (e.g., global positioning system (GPS), compass, etc.), inertial sensors (e.g., inertial measurement unit, accelerometer, magnetometer, gyroscope, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.). By using the sensor data, the techniques described herein can generate, receive, and / or otherwise access road network data associated with the actual environment, and a road mesh associated with the actual environment. The techniques described herein can associate the road network data with the road mesh and generate a simulated environment. That is, data representing the actual environment can be used to generate the simulated environment. In one example, the simulated environment can be supplemented with a third data (e.g., data from a third party), which may be referred to herein as "complementary data". The techniques described herein are further directed to procedurally rendering details of objects and surfaces into the simulated environment. The resulting simulated environment can be used to test, authenticate, and / or train systems and / or components used by computing devices of autonomous robots such as autonomous vehicles for navigation, planning, and / or decision making.

[0008] The simulation environment can be used to enhance training, testing, and / or authenticate a system (e.g., one or more components of an artificial intelligence (AI) stack) installed in an autonomous vehicle. For example, in at least one illustration, the simulation environment may be useful for a training system installed in and used by an autonomous vehicle (e.g., a model used in such a system), for example, when actual data is not readily available, when it is not safe to test in the actual environment, and / or to generate a larger amount of data that would otherwise be available. In at least one illustration, the simulation environment may be used to generate training data for rare or low-frequency situations and / or objects. Further, the simulation environment may be useful for testing the performance of an autonomous vehicle (e.g., when the model and / or system is running thereon) when, for example, the actual environment is not available or safe, or when ground truth is not available otherwise. Further, in one illustration, the sensor data associated with the simulation environment may be more accurate than the sensor data associated with the actual environment (e.g., due to occlusion, noise, drift, etc.), and thus the simulation environment may be used to verify observations obtained in relation to the actual environment. In one illustration, the simulation environment may be used for calibration (e.g., of one or more sensor systems installed in an autonomous vehicle). As described above, the techniques described herein are directed towards generating and using simulation environments in various situations.

[0009] The techniques described herein provide various computational efficiencies. For example, by using the procedural rendering techniques described herein, a computing device can require fewer computational resources and the simulated world can be generated faster than what is available via conventional techniques. Conventional techniques are not scalable. For example, generating a simulated environment for a new geographical location can take days or even months using conventional techniques. Generating the dozens, hundreds, and thousands of new simulated environments required for a training system, test system, and / or certification system (e.g., one or more components of an AI stack) on a self-driving vehicle (e.g., before such a self-driving vehicle is deployed in a new actual environment) can take months or even years, limiting the capabilities of such a training system, test system, and / or certification system (e.g., one or more components of an AI stack) on a self-driving vehicle before entering a new actual environment. The techniques described herein differ from the prior art in that they utilize sensor data collected from an actual environment, complement that data with voxel data, and generate a substantially accurate simulated environment more efficiently (e.g., with respect to the corresponding actual environment) than what is available via conventional techniques. Further, the techniques described herein enable the generation of large-scale, scalable simulated environments in less time and with fewer computational resources by randomizing and / or parameterizing the addition of object and / or surface details to customize the appearance of the simulated environment.

[0010] Furthermore, the technology described herein is directed towards improvements in safety. That is, the simulated environments resulting from the generation techniques described herein can be used in test systems, training systems, and certification systems mounted on autonomous vehicles, and ensuring such systems enables the safe operation of the autonomous vehicle when deployed in the actual environment. That is, the simulated environments resulting from the generation techniques described herein can be used to test, train, and certify the planner system, which may be used for the autonomous vehicle to navigate along a trajectory in the actual environment. Thus, such training, testing, and certification made possible by the technology described herein can provide an opportunity to ensure that the autonomous vehicle can operate safely in the actual-world environment. Therefore, the technology described herein improves safe and impactful navigation.

[0011] Figure 1 illustrates a schematic diagram 100 representing the generation of a procedural world as described herein. In one example, one or more computing devices can procedurally render a simulated environment as described herein.

[0012] In at least one example, data collection device 102 can utilize sensor system 104 to collect sensor data 106 associated with the actual environment. As described above, sensor system 104 may include, but is not limited to, LIDAR sensors, RADAR sensors, ultrasonic transducers, SONAR sensors, ToF sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.). Sensor system 104 may output sensor data 106, which may be received by a computing device. In one example, data collection device 102 may be an autonomous vehicle traversing the actual environment as illustrated in FIG. 1. However, data collection device 102 may be any computing device capable of collecting sensor data 106 within the actual environment.

[0013] A computing device may receive, generate, and / or otherwise access road network data 108 and / or road mesh 110, which may be based at least in part on sensor data 106. In at least one example, the road network data 108 may be a two-dimensional (2D) representation or a three-dimensional (3D) representation that indicates, for example, one or more of driving lane elements, bicycle lane elements, parking lane elements, crosswalk elements, intersection elements, lane split elements, traffic signal elements, stop sign elements, stop line elements, yield sign elements, yield line elements, driveway elements, speed bump elements, jaywalking areas (e.g., virtual crosswalks), railroad crossing points (e.g., well-known railroads), passenger pickup points, sign location elements, geopfence elements, and the like. In one example, the road network data 108 may be encoded with information indicating attributes of particular portions of the road network data. For example, a road line in the road network data 108 can be encoded with information indicating that the road line is associated with a bicycle lane element, a parking lane element, or a crosswalk element.

[0014] The road mesh 110 may include 3D tiles (which may be output by a positioning system as described below). Additional details associated with such road network data 108 and / or road mesh 110 are described in U.S. Patent Application No. 15 / 927,806, filed on March 21, 2018, and U.S. Patent Application No. 15 / 913,647, filed on March 6, 2018, the entire contents of both of which are incorporated herein by reference. The computing device may associate the road network data 108 with the road mesh 110 to generate a simulation environment 112. Such integration may include, in at least some instances, projecting the road network data 108 (as 2D data or 3D data) onto the road mesh 110. That is, the techniques described herein are directed to generating a simulation environment (e.g., simulation environment 112) based on real-world data (e.g., sensor data 106).

[0015] The resulting simulated environment 112 may include accurate heights and surface details (e.g., considering the corresponding actual environment). However, in one example, for instance, when constructing the road mesh 110, there may be deficiencies (e.g., incomplete data) in the simulated environment due to occlusions (e.g., parked cars, narrow alleys, etc.). In at least one example, the computing device can access (e.g., and fill in the deficiencies) a second alternative data source to complement the existing simulated environment. For example, in at least one example, the computing device can access data from a third - party source and / or system 114, utilize such complementary data 116, and complement the existing simulated environment 112. By being able to go beyond the current data set (e.g., road network data 108 and road mesh 110), the complementary data 116 provides information related to the actual environment that the data collection device 102 would not otherwise be able to access due to occlusions and / or other deficiencies associated with the data collection technology. In at least one example, the complementary data 116 may include data such as the data evaluation model (DEM) data of the United States Geological Survey (USGS). The USGS's DEM data may include a data set having raster elevation data (e.g., digital elevation maps). The USGS's DEM data may not be as accurate as the related data (e.g., road network data and road mesh), but the USGS's DEM data is often more complete. That is, since the USGS's DEM data may have no deficiencies, such data can be used to complement data sets where data is missing (e.g., due to occlusions or other deficiencies in data collection). In additional or alternative examples, the complementary data 116 may include tree map data associated with the actual environment, color image data associated with the actual environment, map data associated with the environment, etc.

[0016] Furthermore, the computing device can further complement the simulated environment 112 by leveraging data associated with the characteristics of objects in the environment. In at least one example, the computing device can access a stored object footprint data storage 118 that stores stored object data 120 representing the footprint of a building or other stationary object. In one example, such a footprint can be associated with annotations regarding height, classification (e.g., residential, commercial, etc.), and the like. The computing device can utilize the footprint and associated annotations (e.g., stored object data 120) as a guide mesh for generating a facade portion and a rule set. For example, in one example, the computing device can associate a rule set with the footprint of an individual stored object. The rule set can indicate surface details and / or textures associated with different parts of the object. Such a rule set can be associated with the footprint of an individual stored object randomly or based on one or more parameters (e.g., height, classification, etc.). Such a rule set can indicate a method for generating a facade portion of the object corresponding to the footprint of the stored object. For example, the rule set can include a reference to a texture that may be used to generate the facade portion. As a non-limiting example, the rule set can indicate using a specific mesh, texture, etc. (e.g., building classification) for the first floor of a commercial office building, and using different meshes, textures, etc. for the second floor (and subsequent floors) of such a building. Therefore, the execution of the rule set can add surface details (e.g., facade) to objects within the simulated environment. The addition of such details enables the simulated environment of the actual appearance to be procedurally generated.For example, the details of the facade may affect the shading function and / or the reflection of the window, which can add complexity to the simulation environment and thus represent the actual situation.

[0017] In at least one example, the building footprint, height, texturing, and classification may be randomly defined. In examples where the data associated with these footprints does not have an indication of the position within the map (for example, when the footprint is randomly determined, when obtained from a map-independent footprint data storage), such footprints may be aligned or, if not, placed such that at least one facade is placed to align with the lanes in the road network and spaced according to one or more rules (for example, placed at a specific distance from the lane, based on classification, at a minimum or maximum distance from other buildings, oriented in a specific direction, etc.). When such a set of rules is applied, a building with a plausible appearance can be automatically generated (for example, without human modeling) in the simulation environment 112. By using such rules, it is possible to procedurally generate simulation environments with different appearances without a significant investment of the designer's time and / or computational resources. That is, using such rules increases the efficiency with which complex simulation environments can be generated (for example, via relatively simple rules).

[0018] In one example, a computing device can utilize texturing data to add surface details to the simulated environment 112, for example, during real-time rendering. In such an example, the computing device can access a surface detail data storage 130, which stores surface detail data 132. The surface detail data 132, which is also referred to as "texturing data", may include details (such as defects, patches, markings, etc.) that can be added to objects within the simulated environment 112, making such objects appear unique (without significantly increasing the workload of the technician or without additional calculations required for algorithmically customizing otherwise). In at least one example, the computing device utilizes sparse virtual textures to enable rendering of the simulated environment 112 in a single draw, which improves performance and reduces computing resources. In such an example, each surfel (e.g., a surface element) may be associated with unique data (such as an identifier), whereby individual surfels can be assigned, addressed, and further assigned. Further, in one example, the computing device can add a plurality of brushstroke-like decals to each surface of an object rendered within the simulated environment 112. In at least one example, the various decals are applied to various area and structure classifications (e.g., realistic dirt, grime, graffiti, debris, etc. may be applied to the facade of a building in an alleyway), and can modify any procedural-based texturing (e.g., apply a texture pattern over a surface given a related classification). The techniques described herein enable a designer to model several different textures, which can be used throughout the simulated environment 112. Adding surface details to the simulated environment 112 can increase the diversity within the simulated environment 112 and between the simulated environment 112 and other simulated environments.

[0019] The resulting simulation environment 134 may be output for use by a simulation computing system. In at least one example, the simulation environment 134 may be useful for enhancing a training system, a test system, and / or an authentication system (e.g., one or more components of an AI stack) mounted on an autonomous vehicle. For example, in at least one example, the simulation environment 134 may be useful for a training system used on an autonomous vehicle (e.g., a model used in such a system), for example, when actual data is not readily available, when it is not safe to test in an actual environment, and / or to generate a larger size of data that would otherwise be available. In at least one example, the simulation environment 134 may be used to generate training data for situations and / or objects that occur rarely or at a low frequency. Further, the simulation environment 134 may be useful for testing the performance of an autonomous vehicle (e.g., when a model and / or system is running thereon), for example, when either the actual environment is not available or is not safe, or when ground truth is otherwise not available. By having a simulation environment, accurate ground truth measurements can be determined for such authentication without the need for human-based annotation of data. Further, in one example, the sensor data associated with the simulation environment 134 may be more accurate than the sensor data associated with the actual environment (e.g., due to occlusion, noise, drift, etc.), and thus, the simulation environment 134 may be used to verify observations obtained in relation to the actual environment. In one example, the simulation environment 134 may be used for calibration (e.g., of one or more sensor systems mounted on an autonomous vehicle).

[0020] Figures 2A-2F illustrate non-limiting examples of various aspects of a procedure for rendering a simulation environment, as described herein.

[0021] As described above, the data collection device 102 is capable of generating sensor data 106 associated with the actual environment via the sensor system 104. The computing device may receive the sensor data 106 and may receive, generate, and / or otherwise access road network data 108, as illustrated in FIG. 2A. In at least one example, the road network data 108 may be a 2D or 3D representation showing one or more of, for example, driving lane elements, bicycle lane elements, parking lane elements, crosswalk elements, intersection elements, lane split elements, traffic signal elements, stop sign elements, stop line elements, yield sign elements, yield line elements, driveway elements, speed bump elements, jaywalking areas (e.g., virtual crosswalks), rail crossing points (e.g., well-known rails), passenger pickup points, sign location elements, geopfence elements, etc. Further, the computing device is capable of generating a road mesh 110. In at least one example, as illustrated in FIG. 2B, the sensor data 106 may include LIDAR data, which may be used to generate a 3D point cloud representing the actual environment. In at least one example, the computing device is capable of generating a road mesh 110 based on the 3D point cloud. The road mesh 110 may include 3D tiles (which may be output by a positioning system as described below). The computing device can associate the road network data 108 with the road mesh 110 and generate a simulated environment 112. Such integration may include, in at least some instances, projecting the road network data 108 (as 2D or 3D data) onto the road mesh 110.

[0022] The resulting simulation environment 112 may include accurate heights and surface details (e.g., considering the corresponding actual environment). However, in one example, when constructing the road mesh 110, for example, there may be gaps in the simulation environment (e.g., incomplete data) due to occlusions (e.g., parked cars, narrow alleys, etc.). In at least one example, the computing device can access (e.g., and fill in the gaps) a second alternative data source to complement the existing simulation environment. FIG. 2C illustrates a non-limiting example of complementary data 116 corresponding to the same portion of the actual environment as represented in FIGS. 2A and 2B. For example, in at least one example, the computing device can access data from a third-party source and / or system 114 (e.g., by integrating road network data 108 and / or the road mesh 110 with the complementary data 116) and utilize such complementary data 116 to complement the existing simulation environment 112. As described above, the complementary data 116 may include USGS DEM data associated with the actual environment, tree map data associated with the actual environment, color image data associated with the actual environment, map data associated with the environment, and the like.

[0023] Furthermore, the computing device can further complement the simulated environment 112 by leveraging data associated with the characteristics of objects in the environment. In at least one example, the computing device can access a stored object footprint data storage 118, which stores stored object data 120 representing the footprint of a building or other stationary object. In one example, such a footprint can be associated with annotations regarding height, classification (e.g., residential, commercial, etc.), and the like. As described above, the computing device can utilize the footprint and associated annotations (e.g., stored object data 120) as a guide mesh for generating a facade portion. In at least one example, the footprint, height, texturing, rule set, and / or classification of a building may be defined randomly. In examples where the data associated with these footprints does not have an indication of position within the map (e.g., when the footprint is randomly determined, when obtained from a map-independent footprint data storage), such footprints may be aligned or otherwise placed such that at least one facade is placed to align with a lane in the road network and spaced according to one or more rules (e.g., placed at a specific distance from the lane, based on classification, a minimum or maximum distance from other buildings, oriented in a specific direction, etc.). As illustrated in FIG. 2D, when such a rule set is applied, a building with a plausible appearance can be automatically generated within the simulated environment 112.

[0024] In one example, the computing device can utilize texturing data to add surface details to the simulated environment 112, for example, during real-time rendering. In such an example, the computing device can access a surface detail data storage 130, which stores surface detail data 132. The surface detail data 132, also referred to as "texturing data", may include details (such as defects, patches, markings, etc.) that can be added to objects within the simulated environment 112, making such objects uniquely visible (without significantly increasing the workload of the engineer). As illustrated in FIG. 2E, in at least one example, the computing device utilizes sparse virtual textures to enable rendering of the simulated environment 112 in a single draw, which improves performance and reduces computing resources. In such an example, each surface may be associated with unique data (such as identification), whereby individual surfaces may be assigned, addressed, and further assigned. In FIG. 2E, although depicted as a single alphanumeric for illustrative purposes, such identification is not meant to be so limiting. Further, in one example, the computing device can add a plurality of brushstroke-like decals to each surface of an object rendered within the simulated environment 112. In at least one example, the various decals are applied to various area and structure classifications (for example, realistic dirt, grime, scribbles, debris, etc. may be applied to the facade of a building in an alleyway), modifying any procedural-based texturing (for example, applying a texture pattern over a surface given a related classification).

[0025] FIG. 2F illustrates a non-limiting example of a resulting simulation environment 134, as described herein, which may be for enhancing a training system, a test system, and / or an authentication system (e.g., one or more components of an AI stack) mounted on an autonomous vehicle. That is, in one example, the resulting simulation environment 134 may be used in a training algorithm, a test algorithm, and / or an authentication algorithm used by an in-vehicle system of an autonomous vehicle, which may be used to control the autonomous vehicle.

[0026] FIG. 3 is a block diagram illustrating an exemplary system for procedurally rendering a simulation environment.

[0027] In at least one example, vehicle 302 may include one or more vehicle computing devices 304, one or more sensor systems 306, one or more emitters 308, one or more communication connections 310, at least one direct connection 312, and one or more drive systems 314. For illustrative purposes, vehicle 302 may be an autonomous vehicle configured to operate according to a level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle that does not expect a driver (or passenger) to always control the vehicle and is capable of performing all safety-critical functions for the entire trip. In such an example, vehicle 302 may be configured to control all functions from start to stop, including all parking functions, and thus may be empty. This is merely an example, and the systems and methods described herein may be incorporated into any ground, air, or water vehicle, including those that need to be always manually controlled by a driver to those that are partially or fully autonomously controlled. That is, in the example described, vehicle 302 is an autonomous vehicle, but vehicle 302 may be any other type of vehicle.

[0028] In at least one example, vehicle 302 may be a data collection device (e.g., of data collection device 102). In additional or alternative examples, one or more components of the AI stack described above may be associated with vehicle 302. That is, the simulation environment described herein can be used to train, test, and / or validate one or more components described below with reference to vehicle 302.

[0029] Vehicle computing device 304 may include a processor 316 and a memory 318 communicatively coupled to processor 316. In the example described, memory 318 of vehicle computing device 304 stores a positioning system 320, a perception system 322, a prediction system 324, a planning system 326, and one or more system controllers 328. Further, memory 318 may include storage 230, which may store maps, models, and the like. The map may be any number of data structures modeled in two dimensions, three dimensions, or N dimensions that can provide information about environments such as, but not limited to, topology (such as intersections), lanes, mountains, roads, terrain, and general surroundings. The map may be associated with an actual environment or a simulation environment. The model may include models trained by a machine as described below.

[0030] In at least one example, positioning system 320 can determine the pose (e.g., position and orientation) of vehicle 302 relative to a local map and / or a global map, based at least in part on sensor data received from sensor system 306 and / or map data associated with (e.g., of) a map. In at least one example, positioning system 320 can include or be associated with a calibration system that performs operations that substantially simultaneously perform calibration (determining various associated intrinsic and extrinsic parameters of any one or more of sensor system 306), positioning, and mapping. Additional details associated with such systems are described in U.S. Patent Application No. 15 / 675,487, filed Aug. 11, 2017, which is related to U.S. Patent Application No. 15 / 675,853, filed Aug. 11, 2017, the entire contents of both of which are incorporated herein by reference. As noted above, positioning system 320 can output road network data and / or a road mesh based on sensor data received by sensor system 306.

[0031] In at least one example, the perception system 322 is capable of performing object detection, segmentation, and / or classification based at least in part on sensor data received from the sensor system 306. In at least one example, the perception system 322 is capable of receiving raw sensor data (e.g., from the sensor system 306). In other examples, the perception system 322 is capable of receiving processed sensor data (e.g., from the sensor system 306). For example, in at least one example, the perception system 322 is capable of receiving data from a vision system that receives and processes camera data (e.g., an image). In at least one example, the vision system utilizes one or more image processing algorithms to enable performing object detection, segmentation, and / or classification on objects identified within the image. In one example, the vision system may associate a bounding box (or other semantic information such as instance segmentation) with the identified object and may associate a confidence score associated with the classification of the identified object. In one example, the object may be colored based on the perceived class when rendered via a display. In at least other examples, similar processes (detection, classification, segmentation, etc.) may be performed by the perception system 322 for one or more other modalities (e.g., LIDAR, RADAR, ToF sensors, etc.).

[0032] The prediction system 324 may access sensor data from the sensor system 306, map data associated with a map (e.g., a map that may be present in the storage 230), and / or perception data output from the perception system 322 (e.g., processed sensor data), and may output a prediction associated with one or more objects in the environment of the vehicle 302. In at least one example, the planning system 326 may be capable of determining a route and / or trajectory for use in controlling the vehicle 302 based at least in part on sensor data received from the sensor system 306 and / or any decisions made by the perception system 322. Additional details of the positioning system, perception system, prediction system, and / or planning system may be found to be applicable in U.S. Patent No. 9,612,123, issued April 4, 2017, and U.S. Patent Application No. 15 / 632,208, filed June 23, 2017, the entire contents of both of which are hereby incorporated by reference. In one example (e.g., if the vehicle 302 is not an autonomous vehicle), one or more of the foregoing systems and / or components may be excluded from the vehicle 302. Although the above systems are described as being "mounted" on the vehicle 302, in other implementations, the systems may be remotely located and / or accessible to the vehicle 302.

[0033] In at least one example, the positioning system 320, perception system 322, prediction system 324, and / or planning system 326 may be capable of processing sensor data as described above and transmitting their respective outputs to the computing device 334 via the network 332. In at least one example, the positioning system 320, perception system 322, prediction system 324, and / or planning system 326 may be capable of transmitting their respective outputs to the computing device 334 at a particular frequency after a predetermined period of time has elapsed, such as substantially in real time.

[0034] In at least one example, the vehicle computing device 304 may include one or more system controllers 328, which may be configured to control the steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle 302. These system controllers 328 may be capable of communicating with and / or controlling a system corresponding to the drive system 314 and / or other components of the vehicle 302.

[0035] In at least one example, the sensor system 306 may correspond to the sensor system 104 and may include a LIDAR sensor, a RADAR sensor, a ToF sensor, an ultrasonic transducer, a SONAR sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., an inertial measurement unit, an accelerometer, a magnetometer, a gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, etc.), a microphone, a wheel encoder, an environmental sensor (e.g., a temperature sensor, a humidity sensor, a light sensor, a pressure sensor, etc.), and the like. The sensor system 306 may include various instances of each of these or other types of sensors. For example, the LIDAR sensor may include individual LIDAR sensors located at the corners, front, rear, sides, and / or top of the vehicle 302. As another example, the camera sensor may include various cameras disposed at various locations inside and / or outside the vehicle 302. The sensor system 306 may provide inputs to the vehicle computing device 304. In one example, the sensor system 306 may be capable of preprocessing at least a portion of the sensor data before transmitting the sensor data to the vehicle computing device 304. In at least one example, the sensor system 306 may be capable of transmitting sensor data to the computing device 334 at a specific frequency after a predetermined period of time, such as substantially in real time, via the network 332.

[0036] Also, vehicle 302 may include one or more emitters 308 that emit light and / or sound as described above. The emitter 308 in this example includes internal voice and visual emitters and communicates with the occupants of vehicle 302. For purposes of illustration and not limitation, the internal emitters may include speakers, lights, symbols, display screens, touch screens, tactile emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, headrest positioners, etc.), and the like. The emitter 308 in this example also includes external emitters. For purposes of illustration and not limitation, the external emitters in this example may include light emitters (e.g., indicator lights, signs, light arrays, etc.) that communicate visually with pedestrians, other drivers, other nearby vehicles, etc., and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that communicate audibly with pedestrians, other drivers, other nearby vehicles, etc. In at least one example, the emitter 308 may be disposed at various locations external and / or internal to vehicle 302.

[0037] Also, vehicle 302 may include a communication connection 310 that enables communication between vehicle 302 and other local or remote computing devices. For example, communication connection 310 may be capable of facilitating communication between vehicle 302 and other local computing devices on drive system 314 and / or. Also, communication connection 310 may be capable of enabling the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Also, communication connection 310 may be capable of enabling vehicle 302 to communicate with a remotely operated computing device or with other remote services.

[0038] The communication connection 310 may include a physical and / or logical interface that connects the vehicle computing device 304 to a network such as another computing device or network 332. For example, the communication connection 310 may enable Wi-Fi-based communication via a frequency defined by the IEEE802.11 standard, short-range radio frequencies such as Bluetooth®, or any suitable wired or wireless communication protocol that enables each computing device to interface and connect with other computing devices.

[0039] The direct connection 312 may directly connect the drive system 314 and other components of the vehicle 302.

[0040] In at least one example, the vehicle 302 may include a drive system 314. In one example, the vehicle 302 may have a single drive system 314. In at least one example, when the vehicle 302 has multiple drive systems 314, the individual drive systems 314 may be disposed at both ends of the vehicle 302 (e.g., at the front and rear). In at least one example, the drive system 314 may include a sensor system that detects the state of the drive system 314 and / or the state of the surroundings of the vehicle 302. For purposes of illustration and not limitation, the sensor system may include a wheel encoder (e.g., a rotary encoder) that senses the rotation of the wheels of the drive module, an inertial sensor (e.g., an inertial measurement unit, an accelerometer, a gyroscope, a magnetometer, etc.) that measures the position and acceleration of the drive module, a camera or other image sensor, an ultrasonic sensor that audibly detects objects in the vicinity of the drive module, a LIDAR sensor, a RADAR sensor, etc. Some sensors, such as wheel encoders, may be specific to the drive system 314. In some cases, a sensor system in the drive system 314 may overlap or complement a corresponding system of the vehicle 302 (e.g., the sensor system 306).

[0041] The drive system 314 may include a high-voltage battery, a motor that propels the vehicle 302, an inverter that converts direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including a hydraulic or electric actuator, a suspension system including hydraulic and / or pneumatic components, a stability control system that reduces traction loss and distributes braking force to maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior perimeter of the vehicle), and one or more other systems (e.g., other electrical components such as a cooling system, a safety system, an on-board charging system, a DC / DC converter, a high-voltage junction, a high-voltage cable, a charging system, a charging port, etc.). Further, the drive system 314 may include a drive module controller, which can receive data from the sensor system, perform preprocessing, and control the operation of various vehicle systems. In one example, the drive module controller may include a processor and a memory communicatively coupled to the processor. The memory may store one or more modules that perform various functions of the drive module 314. Additionally, the drive module 314 may also include a communication connection that enables communication between each drive module and other local or remote computing devices.

[0042] In one example, the vehicle computing device 304, the sensor system 306, the emitter 308, and the communication connection 310 may be implemented external to the actual vehicle as, for example, a simulated vehicle or a simulated system that "traverses" a simulated environment. That is, the vehicle computing device 304, the sensor system 306, the emitter 308, and the communication connection 310 may be used as a simulated autonomous vehicle for the purposes of simulation as described above.

[0043] As described above, vehicle 302 can send sensor data to computing device 334 via network 332. That is, in one example, vehicle 302 may be data collection device 102 as described above with reference to FIG. 1. For the purposes of this discussion, the computing device described above with reference to FIG. 1 may refer to vehicle computing device 304 and / or computing device 334. In one example, vehicle 302 can send raw sensor data to computing device 334. In other examples, vehicle 302 can send processed sensor data and / or a representation of the sensor data (e.g., data output from positioning system 320, perception system 322, prediction system 324, and / or planning system 326) to computing device 334. In one example, vehicle 302 can send sensor data to computing device 334 at a specific frequency after a predetermined period of time, such as at a time close to real-time.

[0044] The computing device 334 is capable of receiving (raw or processed) sensor data from the vehicle 302 and / or one or more data collection devices 336 (which may include other vehicles such as the vehicle 302), and similarly, is capable of receiving data from one or more third-party sources and / or systems 338. In at least one example, the computing device 334 may include a processor 340 and a memory 342 communicatively coupled to the processor 340. In the illustrated example, the memory 342 of the computing device 334 stores an analog system 344, a training system 346, an evaluation system 348, a map storage 350 (e.g., one or more maps, road network data, road meshes, etc.), a training data storage 352 (e.g., storing training data accessible to the training system 346), a model storage 354 (e.g., a model output by the training system 346), a footprint data storage 356 of stored objects, and a surface detail data storage 358. In one example, one or more systems and / or storage repositories may be associated with the vehicle 302 instead of, or in addition to, being associated with the memory 342 of the computing device 334.

[0045] The simulation system 344 is capable of generating a simulation environment. In at least one example, the simulation system 344 can generate a simulation environment via procedural generation (e.g., algorithmic data creation) as described above with reference to FIGS. 1-2G. Further details are described below. In at least one example, the simulation system 344 can access the stored object footprint data storage 356 and / or the surface detail data storage 358 and procedurally render the simulation environment. In one example, the stored object footprint data storage 356 may correspond to the object footprint data storage 118 described above with reference to FIG. 1, and with reference to FIG. 1, as described above, the surface detail data storage 358 may correspond to the surface detail data storage 130. In one example, the stored object footprint data storage 356 and / or the surface detail data storage 358 may be stored in the memory 342 as illustrated in FIG. 3. In additional or alternative examples, the stored object footprint data storage 356 and / or the surface detail data storage 358 may be stored remotely and accessible to the computing device 334, and / or the data stored therein may be provided to the computing device 334 from a third-party source and / or system 338. In one example, the stored object data and / or the surface texture data can be generated substantially in real time.

[0046] In at least one example, as described below, the simulation system 344 is capable of procedurally rendering objects within a simulated environment. That is, in one example, the above data (e.g., 3D tiles, road network data, complementary data, etc.) still has defects when compared to the corresponding actual environment. In such an example, the simulation system 344 can utilize various heuristics to render objects within the simulated environment. In non-limiting examples of such objects, street light poles and / or poles connecting street light poles to traffic signals, parking signs (e.g., which can be rendered based on parking lanes determined from road network data), parking meters (e.g., which can be rendered based on parking lanes determined from road network data), stop signs (e.g., which can be rendered based on stop lines determined from road network data), and the like are included. Additional details are described below with reference to FIG. 6.

[0047] In at least one example, the evaluation system 348 can use the perception system 322 (or another system that inputs data into the perception system 322, such as a vision system, a LIDAR system, etc.) to evaluate how realistic a simulated environment, or a part thereof, is with respect to the corresponding actual environment. In one example, in two environments (e.g., an actual environment vs. a simulated environment), they may look different to a human, but may be perceived in the same way (e.g., based on the activation of a neural network) by a robotic system (e.g., an autonomous vehicle) as defined herein. In at least one example, the evaluation system 348 can analyze data using a machine-trained model and evaluate the authenticity of the simulated environment. For example, in at least one example, the evaluation system 348 can analyze a first intermediate output of a neural network associated with a system (e.g., a vision system, a LIDAR system, etc.) (e.g., based on a simulated environment) with a second intermediate output of a neural network (e.g., based on the corresponding actual environment), and determine a similarity metric (e.g., a difference) that can represent how similar the activation of the neural network associated with the simulated environment is when compared to the activation of the neural network associated with the corresponding actual environment.

[0048] In at least some examples, such activation can be compared by discretizing regions of the input space into corresponding grids and constructing a histogram of activation in the relevant grids for the input data and the comparison data. Once determined, the histogram can be analyzed, for example, by a support vector machine (SVM), where the distance (e.g., a statistical distance) is used to determine how similar the two data sets are. In one example, data types of different sensors can be associated with different parameters of interest (e.g., different parameters that are adjusted to improve the realism of the simulated environment). For example, in a vision system, the parameters of interest may be brightness, exposure, etc., and in a LIDAR system, the parameters of interest may be angle, distance, intensity, sensor modality, etc.

[0049] In at least one example, the training system 346 can train a data model that learns which parameters are important for the perception system 322 (e.g., which parameters are important for the perception system 322 to be able to perceive the simulated environment as it would the actual environment). That is, in at least one example, the training system 346 can train a data model and evaluate realism based on one or more identified parameters. Additional details related to the training and / or use of such models are described in U.S. Patent Application No. 16 / 163,435, filed on October 17, 2018, concurrently herewith and in the same manner, the entire contents of which are incorporated herein by reference.

[0050] As described above, in at least one example, the evaluation system 348 can analyze data using a model trained by a machine (e.g., as described above) and evaluate the authenticity of the simulated environment. That is, the evaluation system 348 can analyze the simulated environment and determine whether such a simulated environment activates the neural network in the same way as the corresponding actual environment activates the neural network. In at least one example, the evaluation system 348 utilizes a model trained by a machine and compares a first intermediate output associated with the actual environment with a second intermediate output associated with the corresponding simulated environment to determine a similarity metric (e.g., difference, distance, etc.), which is a representation of the similarity between the first intermediate output and the second intermediate output. In one example, the first intermediate output and the second intermediate output may be derived from an image, a part of the data (e.g., corresponding to an individual object associated with the data), etc. For example, in at least one example, the first intermediate output may be associated with a first perceptual object in an image associated with the actual environment, and the second intermediate output may be associated with a second perceptual object in an image associated with the simulated environment. If the similarity metric (e.g., difference, distance, etc.) does not meet the threshold (e.g., the first intermediate output and the second intermediate output are similar), the evaluation system 348 can determine that the simulated environment realistically represents the actual environment (e.g., the activation of the neural network is similar). However, if the similarity metric (e.g., difference, distance, etc.) meets or exceeds the threshold, the evaluation system 348 can adjust one or more parameters and observe the change in one or more metrics. For example, the evaluation system 348 can adjust parameters such as brightness, exposure, etc. to improve the realism.

[0051] As described above, the simulation environment may be useful for enhancing training, testing, and / or an authentication system (e.g., one or more components of an AI stack) mounted on a self-driving vehicle such as vehicle 302. In at least one example, the simulation environment may be useful for a training data model where training data from the actual environment is insufficient (e.g., for rare objects, rare situations, etc.). In such an example, the resulting data model may be provisioned or accessible by vehicle 302, and vehicle 302 may utilize the data model that classifies objects in real time (e.g., while driving in the actual environment or otherwise operating). That is, the perception system 322 may utilize the onboard data model (trained based on simulated data associated with the simulation environment) to classify objects in near real time.

[0052] As a non-limiting example, training data from the actual environment is insufficient to train the vehicle 302 to recognize rare events / objects (e.g., types of traffic signals that are not frequently seen). In at least one example, by comparing the simulated environment with the actual environment, the data model can learn that certain parameters are important for training the traffic signal classifier. For example, such parameters may include discoloration of the light bulb, shadows, lens distortion, signal fouling, filament burnout, brightness changes, rotation of the light bulb, intensity of the light bulb, and the like. Based on the identification of the parameters, the training system 346 can adjust the simulated environment associated with the traffic signal and can train the traffic signal classifier based on the adjusted simulated environment. Such a classifier may be provisioned or accessible by the vehicle 302, and the vehicle 302 can utilize a data model that classifies traffic signals in real time. For example, the perception system 322 can utilize an onboard classifier (trained based on simulated data used to generate the simulated environment) to classify traffic signals in substantially real time. That is, as described above, in at least one example, the classifier may be trained with simulated data and used to evaluate actual data. In one example, the classifier may be trained with actual data and authenticated with simulated data. In such an example, the identified differences may be used to improve the classifier. In at least one example, such rare examples may be identified, for example, by training a traffic signal detector based on simulated image data, a detector operating on actual data, and determining where detections are missed. Similarly, determining that the simulated parameters are incorrect may include training an algorithm (e.g., the same detector as above) with actual data, running such a detector with simulated data, and detecting missed objects.

[0053] Furthermore, the simulation environment may be useful for validating and / or updating the positioning algorithm used by the positioning system 320. For example, in an actual environment, a GPS sensor may experience position drift and as a result accumulate errors. Thus, to validate the positioning algorithm used to position the vehicle 302, the evaluation system 348 may use a simulation environment where the pose of the vehicle 302 is known at various times (including all times), and evaluate the sensor data associated with the corresponding actual environment (e.g., by relying on the simulated pose as the ground truth of the position and / or orientation) to validate the positioning algorithm. In such an example, the sensor system 306 may generate sensor data associated with the simulation environment, and the sensor data may be analyzed by the perception system 322. The output of the perception system 322 (e.g., associated with a position in the actual environment) may be validated taking into account the sensor data associated with the corresponding position in the simulation environment. That is, the sensor data associated with a position in the simulation environment may function as the ground truth for the corresponding position in the actual environment. As an example, LIDAR data recorded in relation to a simulation environment (e.g., when the pose of the vehicle 302 is known) may be compared to LIDAR data recorded in relation to the corresponding position in the actual environment, and the positioning algorithm may be updated as appropriate. Additionally, the simulation environment may be useful for validating the RADAR or other sensors of the sensor system 306. In one example, the simulation environment may be capable of providing ground truth data for calibrating the sensors (e.g., of the sensor system 106). Other examples include, but are not limited to, validation of rolling shutters in simulation, calibration of various sensors (e.g., one or more intrinsic or extrinsic), etc. As will be appreciated, the techniques described herein may be used for validating, calibrating, training, etc. various other systems, subsystems, etc.

[0054] The processor 316 of the vehicle 302 and the processor 340 of the computing device 334 may be any suitable processor capable of processing data and executing instructions that perform operations as described herein. By way of example and not limitation, processors 316 and 340 may include one or more central processing units (CPUs), graphics processing units (GPUs), or any other device or portion of a device that processes electronic data and converts that electronic data into other electronic data that may be stored in registers and / or memory. In one example, associated circuitry (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be considered processors so long as they are configured to implement the encoded instructions.

[0055] Memories 318 and 342 are examples of non-transitory computer-readable media. Memories 318 and 342 may store an operating system and one or more software applications, instructions, programs, and / or data, and implement functions resulting from the methods and various systems described herein. In various implementations, the memory may be implemented using any suitable memory technology such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash type memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, and those illustrated in the accompanying drawings are merely examples relevant to the description herein.

[0056] While FIG. 3 is illustrated as a distributed system, in an alternative example, components of vehicle 302 may be associated with computing device 334, and / or components of computing device 334 may be associated with vehicle 302. That is, vehicle 302 may perform one or more of the functions associated with computing device 334, and vice versa.

[0057] FIGS. 4-6 are flowcharts illustrating exemplary methods that include the techniques described herein. The methods illustrated in FIGS. 4-6 are described with reference to system 300 illustrated in FIG. 3 for convenience and ease of understanding. However, the methods illustrated in FIGS. 4-6 are not limited to being executed using system 300. Further, system 300 described herein is not limited to executing the methods illustrated in FIGS. 4-6.

[0058] Methods 400-600 are illustrated as a set of blocks in a logical flow graph, where they represent a series of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, a block represents computer-executable instructions stored on one or more computer-readable media that, when executed by a processor, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform a particular function or implement a particular abstract data type. The order in which the operations are described is not intended to be construed as a limitation, and any number of the recited blocks may be combined in any order and / or in parallel to implement the process. The order in which the operations are described is not intended to be construed as a limitation, and any number of the recited blocks may be combined in any order and / or in parallel to implement the process. In one illustration, one or more blocks of the process may be entirely removed. Further, methods 400-600 may be combined with each other, in whole or in part, or with other methods.

[0059] FIG. 4 illustrates an exemplary process 400 for generating a simulation environment and / or using the simulation environment for training, testing, authentication, etc.

[0060] Block 402 indicates accessing road network data. The vehicle computing device 304 can receive, generate, and / or otherwise access road network data. In one example, the road network data may be at least partially based on sensor data. In at least one example, the road network data may be, for example, a 2D or 3D representation showing one or more of driving lane elements, bicycle lane elements, parking lane elements, crosswalk elements, intersection elements, lane split elements, traffic signal elements, stop sign elements, stop line elements, yield sign elements, yield line elements, driveway elements, speed bump elements, jaywalking areas (e.g., virtual crosswalks), railroad crossing points (e.g., well-known railroads), passenger pickup points, sign location elements, geopence elements, etc. As described above, in one example, the road network data may be encoded with information indicating attributes of specific portions of the road network data. In at least one example, the simulation system 344 may access the road network data. As described above, in one example, the road network data may be stored in the map data storage 350.

[0061] Block 404 indicates accessing a road mesh. In at least one example, the vehicle computing device 304 may generate a road mesh 110. In at least one example, the vehicle computing device 304 may receive LIDAR data, which may be used to generate a 3D point cloud that is a representation of the actual environment. In at least one example, the vehicle computing device 304 is capable of generating the road mesh 110 based on the 3D point cloud. As described above, the road mesh 110 may include 3D tiles. The simulation system 344 may access the 3D tiles, which may be stored in the map storage 350 in one example.

[0062] Block 406 shows generating a simulation environment by combining road network data and a road mesh. The simulation system 344 can associate the road network data with the road mesh to generate a simulation environment. Such an association may include, in at least some instances, projecting the road network data (as 2D or 3D data) onto the road mesh. This can be done by determining the center of the road segment and substantially aligning the segment with the corresponding area from the road mesh in the projection. Any outliers (e.g., surfaces that do not align with the road mesh after projection, such as a sidewalk projected onto a tree due to errors in alignment or map generation) may be determined and smoothed as necessary. The resulting simulation environment may include accurate height and surface details (e.g., considering the corresponding actual environment).

[0063] Block 408 indicates accessing complementary data from an alternative data source. In one example, for instance, when constructing a road mesh, there may be deficiencies (e.g., incomplete data) in the simulated environment due to occlusions (e.g., parked cars, narrow alleys, etc.). In at least one example, the simulation system 344 can access a second alternative data source that complements the existing simulated environment (e.g., and fills in the deficiencies). For example, in at least one example, the simulation system 344 can access data from a third - party source and / or system 338 and, as shown in block 410, utilize such data to complement the existing simulated environment by combining the complementary data into the simulated environment. In at least one example, as described above, the complementary data may include USGS DEM data associated with the actual environment, tree map data associated with the actual environment, color image data associated with the actual environment, map data associated with the environment, etc. In one example, the complementary data does not align naturally with the 3D tiles used in the generation of the simulated environment. Details related to the alignment of the road mesh and the complementary data are described below with reference to FIG. 5.

[0064] Block 412 shows the rendering of an object in a simulated environment. The simulation system 344 can further complement the simulated environment by leveraging data associated with the characteristics of the object within the environment. In at least one example, the simulation system 344 may access data representing the footprint of a building or other stationary object, which may be stored in the stored object footprint data storage 356. In one example, such a footprint may be associated with annotations regarding height, classification (e.g., residential, commercial, etc.), rule sets (e.g., texturing), etc. The simulation system 344 may utilize the footprint and associated annotations as a guide mesh for generating a facade portion. In at least one example, the footprint, height, texturing, and classification of a building may be randomly defined. In one example, a predefined set of textures, heights, or footprints may be defined. The set resulting from one or more random arrays of textures, heights, footprints, etc. is used as the set of buildings to be added to the simulated environment in sequence. In examples where the data associated with these footprints does not have an indication of position within the map (e.g., when the footprint is randomly determined, obtained from a map-independent footprint data storage), such footprints may be aligned or otherwise placed such that at least one facade is arranged to align with a lane in the road network and is spaced according to one or more rules (e.g., placed at a specific distance from the lane, at a minimum or maximum distance from other buildings based on classification, oriented in a specific direction, etc.). When such a rule set is applied, a building with a plausible appearance can be automatically generated (e.g., without human modeling), making it possible to generate a simulated environment including the building.

[0065] Block 414 shows the details of the rendering surface associated with the object within the simulated environment. In one example, the simulation system 344 can utilize texturing data to add surface details to the simulated environment, for example, during real-time rendering. Such texturing adds details (defects, patches, markings, etc.) to the objects within the simulated environment and makes such objects unique (without significantly increasing the workload of the engineer). In at least one example, the simulation system 344 may utilize sparse virtual textures and render the simulated environment in one draw, which improves performance and reduces computational resources. In such an example, each surface may be associated with unique data (such as identification), whereby individual surfaces may be assigned, addressed, and further assigned. Additionally, in one example, the simulation system 344 can add multiple brushstroke-like decals to each surface of the objects rendered in the simulated environment. In at least one example, the various decals may be applied to various area and structure classifications (for example, realistic dirt, grime, graffiti, debris, etc. may be applied to the facade of a building in an alleyway), and to do so, any procedural-based texturing is changed (for example, applying a texture pattern over the surface given the relevant classification).

[0066] In at least one example, the simulation system 344 can parameterize the surface of the simulation environment and create a unique texture coordinate space. In such an example, each surface may be associated with a set of pixels that individually reference a texture, which may be stored in the surface detail data storage 358. Thus, at runtime, the simulation system 344 may utilize a single draw call to render (e.g., to a shading system) the details (e.g., textures) associated with the portion of the simulation environment that can be viewed through the viewport of the simulation computing device. In one example, the simulation environment may be associated with multiple textures (e.g., associated with different properties of materials), and each texture may be rendered at runtime via an individual draw call. Such techniques reduce the workload on the processor 340, thereby providing runtime computational efficiency. In at least one example, if the properties of a texture are changed, such changes may be stored in the surface detail data storage 358 in relation to the texture, and at runtime, the changes may be reflected in the texture being rendered. That is, the script can update the texture data associated with the texture in the surface detail data storage 358, thereby affecting the update of the texture rendered in the simulation environment at runtime without the need to reexamine all the techniques.

[0067] Block 416 indicates that it outputs a simulation environment. In at least one example, the simulation system 344 is capable of outputting a simulation environment. As described above, the simulation environment may be useful for training, testing, and / or enhancing an authentication system (e.g., one or more components of an AI stack) installed in an autonomous vehicle such as vehicle 302. In at least one example, the simulation environment may be useful for a training data model where training data from the actual environment is insufficient (e.g., for rare objects, rare situations, etc.). In such an example, the resulting data model may be provisioned to vehicle 302 or be accessible by vehicle 302, and vehicle 302 can utilize the data model that classifies objects in real time (e.g., while driving in the actual environment or otherwise operating). That is, the perception system 322 can utilize the installed data model (trained based on simulation data associated with the simulation environment) to classify objects almost in real time. In at least some examples, such a data model trained using data associated with the simulation environment can output objects in the actual environment based on actual sensor data. Trajectories (and corresponding controls) can then be determined to enable the autonomous vehicle to safely navigate the actual environment based at least in part on the output of such a data model trained with simulation data.

[0068] As described above, in at least one example, the evaluation system 348 can analyze perceptual data using a model trained by a machine (e.g., as described above) and evaluate the realism of the simulated environment. In at least one example, the evaluation system 348 utilizes a model trained by a machine and compares a first intermediate output associated with an actual environment (e.g., associated with a layer of a neural network) with a second intermediate output associated with a corresponding simulated environment (e.g., associated with the same layer of the neural network) to determine a similarity metric (e.g., difference, distance, etc.), which is a representation of the similarity between the first intermediate output and the second intermediate output. In one example, the first intermediate output and the second intermediate output can correspond to an image, a part of the data (e.g., corresponding to an individual object associated with the data), etc. If the similarity metric (e.g., difference, distance, etc.) does not meet the threshold (e.g., the first intermediate output and the second intermediate output are similar), as described above, the evaluation system 348 may determine that the simulated environment realistically represents the actual environment and may output the simulated environment. However, if the similarity metric (e.g., difference, distance, etc.) meets or exceeds the threshold, the evaluation system 348 can adjust one or more parameters and observe the change to one or more metrics. For example, the evaluation system 348 can adjust parameters such as brightness, exposure, etc. to improve realism. When the similarity metric (e.g., difference, distance, etc.) is lower than the threshold (e.g., the first intermediate output and the second intermediate output are similar), the evaluation system 348 may determine that the simulated environment realistically represents the actual environment.

[0069] Figure 5 illustrates an exemplary process 500 for combining complementary data into a simulated environment.

[0070] Block 502 shows the alignment of the complementary data with the road mesh associated with the simulation environment. As described above, in one example, the simulation system 344 can complement the road mesh (e.g., 3D tiles) with the complementary data. First, the simulation system 344 can roughly align the complementary data with the road mesh. The complementary data may not naturally align with the 3D tiles of the road mesh used to generate the simulation environment.

[0071] Block 504 is shown to determine an error between the complementary data and the road mesh associated with the simulation environment. In at least one example, the simulation system 344 can periodically measure the error between the complementary data and the road mesh. In at least one example, such measurements may be made in areas where the complementary data is predicted to be accurate, such as, for example, large flat spaces (e.g., parking lots, large intersections, etc.). Such spaces may correspond to areas of the actual environment that meet or exceed a threshold area (e.g., "large") and / or are associated with a maximum elevation change over an area below a threshold (e.g., "flat"). In at least one example, such measurements may be made in areas visible in the road mesh and the complementary data, and / or areas that are unobstructed and / or otherwise free of objects such as trees, buildings, etc. (e.g., aerial scans from USGS). In at least one example, the error may be measured from a specified position within the road mesh. In one example, the specified position may be a centerline, which may be derived from the road network data and / or the road mesh. For example, as shown in the road mesh, the center of the driving lane may be designated as the centerline from which the error can be measured. In at least one example, the error may be measured from the determined center point of such an area. That is, the road mesh may be considered the basis, and the error may be measured from the basis. As a non-limiting example, the error may be associated with the difference (e.g., Euclidean distance) between the road mesh and the complementary data. In one example, the error may be a height error and measure the vertical distance between the road mesh and the complementary data. The error may be a single measurement, the average of multiple measurements, the maximum value over an area, the minimum value over an area, a comprehensive error, or another statistically significant measurement representing the difference and / or distance between the complementary data and the road mesh.

[0072] Block 506 indicates determining whether the error meets or exceeds a threshold. In at least one illustration, as shown in block 508, the simulation system 344 can compare the error with the threshold, and based on the determination that the error does not meet or exceed the threshold, the simulation system 344 is not limited to, but can perform, fusion techniques such as overall fusion techniques, local fusion techniques, linear fusion techniques, fusion techniques of smooth curves (with a specific radius), etc., and fuses the complementary data with the road mesh. In at least one illustration, such fusion techniques may be accompanied by attenuation, whereby the error becomes less important as the simulation system 344 moves away from the selected area. That is, at the centerline of the road mesh, as described below, the error may be corrected or reduced, but other parts of the simulation environment that are continuously further away from the centerline may be fused and the error corrected or otherwise reduced.

[0073] Based on the determination that the error meets or exceeds the threshold, the simulation system 344 can apply a deformation lattice to substantially align the complementary data with the road mesh, as shown in block 510, before fusion. Adjustment to either data from a third-party source or a map from a mapping system can drive such an error in an area (e.g., a large and flat area) used to determine the error and can propagate it throughout the rest of the simulation environment. In at least one illustration, the simulation environment system 344 can adjust the complementary data to locally align the complementary data with the road mesh. In additional or alternative illustrations, the simulation system 344 can adjust the road mesh to locally align the road mesh with the complementary data. For example, in at least one illustration, the simulation system 344 can apply centroid weighting to reduce the error (or other interpolation in the data). As a result, the simulation system 344 can output a refined simulation environment that is more complete than the resulting simulation environment by integrating the road network data and the road mesh based on the complementary data. That is, the deficiencies within the simulation environment can be filled, thereby enhancing the simulation environment with the complementary data.

[0074] In one illustration, the flow may first proceed from block 506 to block 508. At block 508, the deformation may be applied locally based on the error in a large and flat area such as an intersection. The interpolation may be applied between the errors from the first intersection (or area, etc.) to all adjacent intersections. In at least one illustration, such interpolation may be linear, although all other interpolations (e.g., polynomial, bicubic, etc.) are considered. For these areas of the road mesh that are away from the center point of such a selected area, centroid coordinates (or other interpolations such as bicubic) are applied to adjust the road mesh to the complementary data.

[0075] In these illustrations where the flow first proceeds to block 508, the flow can then proceed to block 510. In such illustrations, even though the road mesh is being adjusted to the supplementary data, additional errors may be found between the mesh and the supplementary data. In such illustrations, block 510 may include further deformations (or otherwise adjusting one or more of the mesh or the supplementary data). Such deformations can be performed according to local deformations (e.g., if the road mesh data follows an S-curve where it is 100% the source of height information at the center of the selected area and 0% of the height information beyond a certain radius from the center, or otherwise). By locally deforming the data (either the mesh or the supplement), it is possible to smoothly transition between the boundaries of the mesh data and the supplementary data.

[0076] FIG. 6 illustrates an exemplary process 600 for procedurally rendering an object such as a traffic signal pole within a simulated environment, as described herein. In at least one illustration, the simulation system 344 is capable of procedurally rendering an object within the simulated environment. That is, in one illustration, the above data (e.g., 3D tiles, road network data, supplementary data, etc.) still has defects when compared to the corresponding actual environment. In such illustrations, the simulation system 344 can utilize various heuristics to render an object within the simulated environment. Process 600 shows an example of such a process.

[0077] Block 602 shows determining the position of a traffic signal within the simulated environment. In at least some illustrations, for example, data regarding the position of the traffic signal may be provided or otherwise determined in association with the road network data and / or the road mesh. In at least one illustration, the simulation system 344 is capable of determining the position of the traffic signal based on such data. However, such data may be absent of traffic information.

[0078] Block 604 shows generating at least one plane passing through the traffic signal. In at least one example, the simulation system 344 can generate one or more planes passing through the traffic signal (e.g., at least one plane having a surface normal indicating the direction of illumination of the light source). In one example, the simulation system 344 can generate two or more planes (e.g., a first plane parallel to the road under the traffic signal and a second plane perpendicular to the road under the traffic signal).

[0079] Block 606 shows determining a sidewalk proximate to the traffic signal from the road network data. In at least one example, the simulation system 344 can identify a sidewalk proximate to the traffic signal, for example, based on the road network data as described above.

[0080] Block 608 shows determining the point closest to the sidewalk. In at least one example, the simulation system 344 can determine the point closest to the surrounding sidewalk. Such a system can incorporate various path planning algorithms (e.g., A*, D*, etc., incorporating Manhattan constraints (forcing 90-degree constraints) and / or minimizing the number of curves of streetlight poles) to find the shortest path from the location of the traffic signal to the closest sidewalk point.

[0081] Block 610 shows a streetlight pole and a rendering of the pole that connects the streetlight pole to a traffic signal within a simulated environment. The simulation system 344 may utilize such determined routes to render the streetlight pole, which may be associated with the pole that connects the streetlight pole to the traffic signal. In at least one example, the simulation system 344 is able to access data from a third-party source and / or system 338, which indicates rules such as the position, orientation, style, etc. of the streetlight pole and the pole. For example, as a non-limiting example, such data may indicate that the streetlight pole should be placed approximately 20 feet to the right of the traffic signal.

[0082] Process 600 is directed to rendering a streetlight pole and the pole that connects the streetlight pole to a traffic signal, but in additional or alternative examples, process 600 may be directed to adding other objects to the simulated environment. Non-limiting examples include parking signs (e.g., which may be rendered based on parking lanes determined from road network data), parking meters (e.g., which may be rendered based on parking lanes determined from road network data), stop signs (e.g., which may be rendered based on stop lines determined from road network data), etc. In such examples, the simulation system 344 is able to access data from a third-party source and / or system 338, which indicates rules such as the position, orientation, style, etc. of the object. In at least one example, the rendering of such objects within the simulated environment may be parameterized, and the simulation system 344 may follow the rules as indicated by a third-party source and / or system 338.

[0083] (Exemplary clause) A. The computer-implemented method includes receiving sensor data from a plurality of data collection devices in an actual environment, accessing road network data associated with the actual environment, generating a road mesh associated with the actual environment based at least in part on the sensor data, integrating the road network data with the road mesh to generate a simulated environment, accessing a data storage of footprints of stored objects, selecting footprints of stored objects from the data storage of footprints of stored objects, rendering at least one object corresponding to the footprint of the stored object into the simulated environment, rendering details of a surface associated with the at least one object, and outputting for at least one of navigation, planning, or decision-making for at least one of testing, authenticating, or training an algorithm used by an autonomous robot computing device.

[0084] B. In the computer-implemented method described in paragraph A, the road network data includes a two-dimensional representation of the actual environment and includes a display of at least one of a driving lane element, a bicycle lane element, a parking lane element, or a crosswalk element.

[0085] C. In the computer-implemented method described in any of paragraphs A to B, the road mesh includes a plurality of three-dimensional tiles output from a mapping system.

[0086] D. In the computer-implemented method described in any of paragraphs A to C, integrating the road network data with the road mesh includes aligning road segments in the road network data with corresponding areas of the road mesh and projecting at least a portion of the road segments onto the road mesh.

[0087] In the computer-implemented method described in any of paragraphs A - D, the footprint of the stored object is associated with at least one of height, an annotation indicating classification, or a set of rules indicating the texture associated with the object, and the computer-implemented method further comprises rendering at least one object corresponding to the footprint of the stored object into the simulation environment, at least in part based on the annotation.

[0088] In the computer-implemented method described in any of paragraphs A - E, the surface details include at least one of a defect texture, a patch texture, or a marking texture.

[0089] In the computer-implemented method described in any of paragraphs A - F, the surface details are added using a sparse virtual texture in a single draw.

[0090] The system comprises at least one processor and computer-readable instructions that, when executed by the at least one processor, cause the at least one processor to access road network data associated with the actual environment, generate a road mesh associated with the actual environment, associate the road network data with the road mesh to generate a simulation environment, procedurally render at least one object into the simulation environment at least in part based on at least one rule, and output for at least one of testing, authenticating, or training an algorithm used by an autonomous robotic computing device, control of the autonomous robotic computing device.

[0091] I. In the system described in paragraph H, the operation further includes accessing a data storage containing at least one stored object footprint, selecting a footprint of a stored object from the data storage, and rendering at least one object corresponding to the footprint of the stored object into a simulation environment, at least in part based on at least one rule.

[0092] J. In the system described in paragraph I, the rules include the display of the distance between an individual object and a lane in road network data, the display of the distance between individual objects of a plurality of objects, or the direction of individual objects of a plurality of objects.

[0093] K. In the system described in paragraph I, the footprint of at least one stored object is associated with an annotation indicating the characteristics of the object, where the characteristics include height or classification.

[0094] L. In the system described in paragraph K, the footprint of at least one stored object is associated with a rule set for texturing the object, and the rule set is associated with the footprint of at least one stored object based on the characteristics.

[0095] M. In the system described in any of paragraphs H - L, the operation further includes rendering the details of the surface associated with at least one object using a sparse virtual texture.

[0096] In the system described in any of paragraphs H to M, the operation further includes determining the positions of traffic signals, at least in part, based on road network data; generating at least one plane passing through the traffic signals; determining sidewalks adjacent to the traffic signals from the road network data; determining the point closest to the sidewalk using a planning algorithm; and rendering, in a simulated environment, streetlight poles and poles to connect the streetlight poles to the traffic signals.

[0097] The non-transitory computer-readable instructions, when executed, cause one or more processors to access road network data associated with an actual environment, generate a road mesh associated with the actual environment, integrate the road network data with the road mesh to generate a simulated environment, procedurally render at least one object into the simulated environment, at least in part, based on at least one rule, and output the simulated environment via a computing device.

[0098] In the non-transitory computer-readable medium described in paragraph O, outputting the simulated environment includes outputting, via a simulated computing device, for use by an autonomous vehicle in at least one of testing, authenticating, or training an algorithm used by the autonomous vehicle, in relation to controlling the autonomous vehicle.

[0099] Q. In the non - transitory computer - readable medium described in any of paragraphs O - P, the operations further include accessing a data storage including the footprint of at least one stored object, where the footprint of at least one stored object is associated with at least one annotation indicating the height or classification of the corresponding object, selecting the footprint of the stored object from the data storage, and rendering at least one object corresponding to the footprint of the stored object into a simulated environment based at least in part on the display of the distance between an individual object and a lane in road network data, the display of the distance between individual objects of a plurality of objects, or the display of the orientation of individual objects of a plurality of objects.

[0100] R. In the non - transitory computer - readable medium described in paragraph Q, the operations further include associating a rule set with the footprint of a stored object based at least in part on at least one annotation, and rendering a texture associated with the object based at least in part on the rule set.

[0101] S. In the non - transitory computer - readable medium described in any of paragraphs O - R, the operations include accessing a data storage including the footprints of a plurality of stored objects, where the footprint of each stored object is associated with a respective annotation, and further include combining and rendering one or more objects corresponding to one or more of the footprints of the plurality of stored objects.

[0102] In the system described in any of paragraphs O to S, the operation further includes determining the positions of traffic signals, generating at least one plane passing through the traffic signals, determining sidewalks close to the traffic signals from the road network data, determining the point closest to the sidewalk using a planning algorithm, and rendering streetlight poles and connecting the streetlight poles in the simulated environment to the traffic signals.

[0103] U. The computer-implemented method includes receiving sensor data from a plurality of data collection devices in an actual environment, accessing road network data associated with the actual environment, where the road network data is at least partially based on the environment, determining a road mesh associated with the actual environment at least partially based on the sensor data, generating a simulated environment by associating the road network data with the mesh, where the simulated environment is incomplete with respect to the actual environment, accessing complementary data associated with the actual environment, where the complementary data provides information associated with the actual environment that is not available to the plurality of data collection devices, associating the complementary data with the simulated environment to complement the simulated environment as a modified simulated environment, and outputting for at least one of navigation, planning, or decision-making for at least one of testing, authenticating, or training an algorithm used by an autonomous robot computing device with the modified simulated environment.

[0104] V. In the computer-implemented method described in paragraph U, the complementary data includes a geospatial file format storing a raster-based digital elevation model.

[0105] W. In the computer-implemented method described in paragraph V, the geospatial file format is associated with the data evaluation model (DEM) standard of the United States Geological Survey (USGS).

[0106] In the computer-implemented method described in any of paragraphs U to W, the simulation environment is incomplete due to occlusion in sensor data associated with a parked vehicle or an alley.

[0107] Y. The computer-implemented method described in any of paragraphs U to X further includes: determining an error between a first portion of the complementary data and a second portion of the road mesh, wherein the first portion and the second portion are associated with the same area of the actual environment; determining whether the error meets or exceeds a threshold amount of error; and adjusting at least one of the complementary data or the road mesh.

[0108] Z. In the computer-implemented method described in paragraph Y, the error includes at least one of the average errors associated with the same area of the actual environment.

[0109] AA. The computer-implemented method described in paragraph Y further includes applying a deformation grid to one or more of at least a part of the complementary data or a corresponding part of the road mesh to substantially align the complementary data and the road mesh and reduce the error.

[0110] AB. The system includes one or more computer-readable instructions that, when executed by at least one processor, cause the at least one processor to receive sensor data from at least one data collection device in an actual environment, access at least one of road network data associated with the actual environment or a road mesh associated with the sensor data, generate a simulated environment based on at least one of the road network data or the road mesh, associate complementary data with the simulated environment to generate a modified simulated environment, and output to at least one that controls an autonomous robot computing device for at least one of testing, authenticating, or training an algorithm used by the autonomous robot computing device with respect to the modified simulated environment.

[0111] AC. In the system described in paragraph AB, the simulated environment is incomplete due to at least one occlusion in the actual environment.

[0112] AD. In the system described in any of paragraphs AB - AC, provide information associated with the actual environment in which at least one data collection device is otherwise unavailable due to at least one occlusion.

[0113] AE. In the system described in any of paragraphs AB - AD, the road network data includes a two-dimensional representation of the actual environment and includes a display of at least one of driving lane elements, bicycle lane elements, parking lane elements, or crosswalk elements.

[0114] AF. In the system described in any of paragraphs AB - AE, the road mesh includes a plurality of three-dimensional tiles output from a mapping system.

[0115] In the system described in any of paragraphs AB - AF, the operation further includes accessing road network data and a road mesh, and associating the road network data with the road mesh based at least in part on projecting the road network data onto the road mesh.

[0116] In the system described in any of paragraphs AB - AG, the complementary data includes elevation data collected by a third - party source or system.

[0117] In the system described in any of paragraphs AB - AH, the operation further includes measuring a height error between a first portion of the complementary data and a second portion of the road mesh, determining whether the height error meets or exceeds a threshold amount of error, and applying a deformation lattice to one or more of the first portion or the second portion to reduce the height error.

[0118] A non - transitory computer - readable medium storing instructions, when executed, causes one or more processors to perform operations including accessing at least one of road network data associated with an actual environment or a road mesh associated with an actual environment, generating a simulated environment based on at least one of the road network data or the road mesh, associating complementary data with the simulated environment to generate a modified simulated environment, and outputting to at least one that controls an autonomous robotic computing device for at least one of testing, authenticating, or training an algorithm used by the autonomous robotic computing device for the modified simulated environment.

[0119] In the non - transitory computer - readable medium described in paragraph AJ, the complementary data includes elevation data collected by a third - party source or system.

[0120] In the non - transitory computer - readable medium described in any of paragraphs AJ - AK, the complementary data includes a geospatial file format associated with the United States Geological Survey (USGS) Data Evaluation Model (DEM) standard.

[0121] In the non - transitory computer - readable medium described in any of paragraphs AJ - AL, measure the height difference between the first part of the complementary data and the second part of the road mesh to determine that the first part and the second part are associated with the same area of the actual environment without objects, and whether the difference meets or exceeds the difference threshold, and further include applying a deformation grid to one or more of the first part or the second part to reduce the error.

[0122] In the non - transitory computer - readable medium described in any of paragraphs AJ - AM, provide information associated with an actual environment in which at least one data - collection device is otherwise unavailable due to at least one occlusion.

[0123] While the exemplary clauses as described above are described with respect to a particular implementation, in the context of this document, it should be understood that this exemplary content can also be implemented through methods, devices, systems, computer - readable media, and / or other implementations.

[0124] (Conclusion) One or more examples of the techniques described in this specification are described, but various changes, additions, substitutions, and their equivalents are included within the scope of the techniques described in this specification.

[0125] In the illustrative description, reference is made to the accompanying drawings that form a part of this specification, which show detailed specific illustrations of the subject matter of the invention claimed as a way of explanation. It should be understood that other illustrations may be used and that changes or alternatives such as structural changes may be made. Such illustrations, changes or alternatives do not necessarily depart from the scope of the claimed subject matter of the intended invention. The steps in this specification may be provided in a particular order, but in some cases, the order may be changed so that, without changing the functionality of the systems and methods described, a particular input is provided at a different time or in a different order. The disclosed procedures may also be executed in a different order. Further, the various calculations in this specification need not be executed in the disclosed order, and other illustrations using alternative orders of calculation may be readily implemented. In addition to rearrangement, the calculations may also be broken down into sub-calculations and it is possible to obtain the same result.

Claims

1. one or more processors; When executed, the one or more processors are As a first alignment, aligning a first region associated with the road mesh and a second region associated with the supplemental data associated with the simulated environment; determining an error between the road mesh and the supplemental data based at least in part on the first alignment; performing at least one of blending the road mesh and the supplemental data to generate blended data to reduce the error based at least in part on the error, or transforming the road mesh to substantially align the road mesh with the supplemental data as a second alignment different from the first alignment; outputting a modified simulated environment for use by the autonomous robotic computing device based at least in part on said blending or said transforming; One or more non-transitory computer-readable media storing computer-executable instructions for performing operations including: A system comprising:

2. The system of claim 1 , wherein the supplemental data comprises a geospatial file format including elevation data.

3. 2. The system of claim 1, wherein the error comprises at least one of a single measurement, an average of multiple measurements, a maximum over an area, a minimum over an area, or a total error representing a difference between a first portion of the road mesh and a second portion of the supplemental data.

4. The operation includes: comparing the error to a threshold amount of error; said blending being based at least in part further on an error not exceeding a threshold amount of said error; or said modifying based at least in part on meeting or exceeding a threshold amount of error; and performing at least one of:

5. The system of claim 1 , wherein the actions further comprise generating a modified simulated environment based at least in part on the mixing or the transforming.

6. The system of claim 1 , wherein the error is a vertical distance error or a horizontal distance error.

7. The system of claim 1 , wherein the error comprises an error between a first portion of the road mesh and a second portion of the supplemental data.

8. 1. A computer-implemented method comprising: As a first alignment, aligning a first region associated with the road mesh and a second region associated with the supplemental data associated with the simulated environment; determining an error between the road mesh and the supplemental data based at least in part on the first alignment; performing at least one of the following steps based at least in part on the error: blending the road mesh and the supplemental data to generate blended data for reducing the error; or deforming the road mesh to substantially align the road mesh with the supplemental data as a second alignment different from the first alignment; outputting a modified simulated environment for use by an autonomous robotic computing device based at least in part on the blending or transforming; 23. A computer-implemented method comprising:

9. The computer-implemented method of claim 8 , wherein the supplemental data comprises a geospatial file format that includes elevation data.

10. The computer-implemented method of claim 8 , wherein the simulated environment is related to an actual environment determined from sensor data from at least one data collection device.

11. The computer-implemented method of claim 8 , wherein the road mesh comprises a plurality of three-dimensional tiles.

12. comparing the error to a threshold amount of error; said blending further based at least in part on an error not exceeding a threshold amount of said error; or said modifying based at least in part on said error meeting or exceeding a threshold amount of error; and performing at least one of the following:

13. 9. The computer-implemented method of claim 8, wherein determining the error between the road mesh and the supplementary data comprises periodically measuring the error between the road mesh and the supplementary data and determining an average of multiple measurements, a maximum over an area, or a minimum over an area.

14. The step of measuring the error includes the step of measuring the error at a specified position within the road mesh; The computer-implemented method of claim 8 , wherein the specified location comprises a road centerline.

15. The computer-implemented method of claim 8 , wherein the blending step includes at least one of global blending, local blending, linear blending, smooth curve blending for blending the road mesh and the supplemental data.

16. The computer-implemented method of claim 8 , wherein a level of blending is based at least in part on a distance between a centerline of the road mesh and an area of ​​the supplemental data.

17. The computer-implemented method of claim 8 , further comprising smoothing transitions between boundaries of the road mesh and boundaries of the supplemental data.

18. The computer-implemented method of claim 8 , wherein the supplemental data includes elevation data collected by a third party source or system.

19. 10. The computer-implemented method of claim 8, wherein the supplemental data comprises a geospatial file format associated with a United States Geological Survey (USGS) Data Evaluation Model (DEM) standard.

20. The computer-implemented method of claim 8 , wherein the supplemental data provides information associated with a real-world environment that is unavailable to at least one data gathering device due to at least one occlusion.

Images