Simulating viewpoint acquisition from a rendered environment with ground truth heuristics

Abstract

An aspect of this technical solution is to generate a three-dimensional (3D) model including a first surface corresponding to one or more physical paths through a physical environment according to one or more first environmental metrics, the one or more first environmental metrics indicating boundaries of the one or more physical paths, generate one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, the second environmental metrics indicating the one or more physical paths, identify one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicating a camera of the physical object configured to move along the one or more physical paths, and render one or more 2D images from the one or more corresponding portions of the 3D model, each 2D image corresponding to a respective one of the viewpoints.

Description

[Technical Field]

[0001] [Cross-reference to related patent applications] This application claims priority to U.S. Provisional Application No. 63 / 377,954, filed September 30, 2022, which is incorporated by reference herein in its entirety for all purposes.

[0002] This implementation relates generally to computer rendering, including but not limited to simulating viewpoint capture from a rendered environment with ground truth heuristics. [Background technology]

[0003] Consumers are increasingly demanding mobile systems that can more accurately navigate their surroundings. Mobile systems can require a significant amount of reliable input from the real world to learn to navigate their surroundings accurately. Training a mobile system to accurately navigate its surroundings can be time-consuming and requires a significant amount of input across a wide variety of conditions. However, the amount of such input required by many mobile systems exceeds the amount of input that can be effectively obtained from the real world. For example, input for a particular real-world location present in a tropical location cannot be obtained in snowy or blizzard conditions. Summary of the Invention

[0004] The technical solution relates to capturing simulated sensor data from a virtual environment that corresponds to at least a physical environment. For example, a computing system can generate a three-dimensional model that corresponds to one or more physical aspects of one or more physical environments in the real world. For example, the computing system can obtain one or more input models, each corresponding to the boundary or shape of an area measured from a "ground truth" physical environment having similar characteristics (e.g., roadways, medians). The computing system can generate a virtual environment that corresponds to those aspects. The computing system can capture one or more viewpoints in the virtual environment that correspond to the movement of an autonomous object in the physical environment that corresponds to the virtual environment. For example, the viewpoints can correspond to respective cameras, each having a specific orientation relative to the autonomous object, and can move within the virtual environment according to the position or orientation of the autonomous object. The computing system can also modify one or more aspects of the virtual environment to create variations of the virtual environment that each correspond to the ground truth roadways of the physical environment but that have not or cannot be captured by measuring or imaging the physical environment. Thus, a computing system may create many virtual instances of a physical environment, as well as many simulated viewpoints corresponding to the many virtual instances of the physical environment, thereby providing a technical improvement for generating a quantity of simulated viewpoints that exceeds at least the amount available in the real world and the amount that can be manually drawn. For example, a computing system may provide a technical improvement for enabling at least autonomous objects that can more accurately navigate the real world. Thus, embodiments herein provide a technical solution for the simulation of viewpoints captured from an environment rendered with ground truth heuristics.

[0005] In one embodiment, the system can include a memory and one or more processors. The system can retrieve a camera feed of an ego object navigating within a physical environment. Based on the camera feed, the system can generate a three-dimensional (3D) model that can include a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics, where the one or more first environmental metrics indicate boundaries of the one or more physical paths. The system can generate one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, where the second environmental metrics indicate the one or more physical paths. The system can identify one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicating a camera of the physical object configured to move along the one or more physical paths. The system can render one or more simulated environment 2D images, each corresponding to a respective one of the viewpoints, from the one or more corresponding portions of the 3D model of the physical environment. The system can train an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images.

[0006] The system can modify a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicative of a condition of the physical path.

[0007] The system may modify the topology of a portion of the first surface according to one or more third environmental metrics.

[0008] The system may modify an opacity of at least a portion of one of the geometric 2D objects, the geometric 2D object being located on a portion of the first surface, according to one or more third environmental metrics.

[0009] The system can generate one or more 3D objects that satisfy the localization heuristic at one or more corresponding locations within a second surface of the 3D model, excluding the first surface, according to a localization heuristic that indicates a type of physical object within the physical environment.

[0010] The system may include a type of physical object corresponding to at least one of a type of geography, a type of climate, or a type of architecture.

[0011] The system may generate one or more 3D objects that satisfy the environmental heuristics at one or more corresponding locations in the 3D model according to environmental heuristics that indicate atmospheric conditions that are indicative of weather in the physical environment.

[0012] The system can segment one area model into multiple area segments according to a block heuristic that indicates the amount of computational resources, where the area model represents a physical area that can include a physical environment, the area segments each correspond to a respective portion of the area model, and the 3D model corresponds to one of the area segments.

[0013] The system can execute a first subset of the region segments with a first computational resource, and can execute a second subset of the region segments with a second computational resource concurrently with execution by the first computational resource.

[0014] In another embodiment, a method can include searching a camera feed of an ego object navigating within a physical environment. The method can include generating a three-dimensional (3D) model that can include a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics, where the one or more first environmental metrics indicate boundaries of the one or more physical paths. The method can include generating one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics based on the camera feed, where the second environmental metrics indicate the one or more physical paths. The method can include identifying one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicative of a camera of the physical object configured to move along the one or more physical paths. The method can include rendering one or more 2D images from one or more corresponding portions of the 3D model of the physical environment, each 2D image corresponding to a respective one of the viewpoints. The method can include training an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images.

[0015] The method may include modifying a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicative of a condition of the physical path.

[0016] The method may include modifying a topology of a portion of the first surface according to one or more third environmental metrics.

[0017] The method may include modifying an opacity of at least a portion of one of the geometric 2D objects according to the one or more third environmental metrics, the geometric 2D object being located on a portion of the first surface.

[0018] The method may include generating, according to a localization heuristic indicative of a type of physical object in the physical environment, one or more 3D objects that satisfy the localization heuristic at one or more corresponding locations in a second surface of the 3D model, excluding the first surface.

[0019] The method may include a type of physical object corresponding to at least one of a type of geography, a type of climate, or a type of architecture.

[0020] The method may include generating one or more 3D objects that satisfy the environmental heuristics at one or more corresponding locations in the 3D model according to environmental heuristics that are indicative of atmospheric conditions that are indicative of weather in the physical environment.

[0021] The method may include segmenting an area model into a plurality of area segments according to a block heuristic that indicates an amount of computational resources, the area model representing a physical area that may include a physical environment, each area segment corresponding to a respective portion of the area model, and the 3D model corresponding to one of the area segments.

[0022] The method can include executing, by a first computational resource, a first subset of the region segments. The method can include executing, by a second computational resource, a second subset of the region segments concurrently with execution by the first computational resource.

[0023] In yet another embodiment, a non-transitory computer-readable medium can include one or more instructions stored thereon and executable by a processor. The processor can search a camera feed of an ego object navigating within a physical environment. The processor can generate a three-dimensional (3D) model that can include a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics, where the one or more first environmental metrics indicate boundaries of the one or more physical paths. The processor can generate one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, where the second environmental metrics indicate the one or more physical paths. The processor can identify one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicating a camera of the physical object configured to move along the one or more physical paths. The processor can render one or more simulated environment 2D images, each corresponding to a respective one of the viewpoints, from the one or more corresponding portions of the 3D model of the physical environment. The processor can train an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images.

[0024] The computer-readable medium can include one or more instructions executable by a processor, wherein the processor can modify a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicative of a condition of the physical path. [Brief explanation of the drawings]

[0025] These and other aspects and features of the present implementations are depicted by way of example in the figures described herein. The present implementations may be directed to, but are not limited to, the examples depicted in the figures described herein. Accordingly, this disclosure is not limited to any figure or portion thereof depicted or referenced herein, or any aspects described herein with respect to any figure depicted or referenced herein.

[0026] [Figure 1A] 1 illustrates components of an AI-enabled visual data analysis system, according to one embodiment.

[0027] [Figure 1B] 1 illustrates various sensors associated with an ego, according to one embodiment.

[0028] [Figure 1C] 1 illustrates components of a vehicle, according to one embodiment.

[0029] [Figure 2] FIG. 1 illustrates a flow diagram for simulating gaze capture from an environment rendered with ground truth heuristics, according to one embodiment.

[0030] [Figure 3] 1 illustrates a system architecture according to one embodiment.

[0031] [Figure 4] 1 illustrates a ground truth visualization according to one embodiment.

[0032] [Figure 5] 1 illustrates a road visualization according to one embodiment.

[0033] [Figure 6] 1 illustrates a visualization of a road surface, according to one embodiment.

[0034] [Figure 7A] 1 illustrates a visualization of road lines according to one embodiment.

[0035] [Figure 7B] 1 illustrates a visualization of road lines according to one embodiment.

[0036] [Figure 8A] 1 illustrates a visualization of the outer surface, according to one embodiment.

[0037] [Figure 8B] 1 illustrates a visualization of a populated exterior surface, according to one embodiment.

[0038] [Figure 9A] 1 illustrates a visualization of traffic objects according to one embodiment.

[0039] [Figure 9B] 1 illustrates a visualization of populated traffic objects according to one embodiment.

[0040] [Figure 10] 1 illustrates a populated traffic environment according to one embodiment.

[0041] [Figure 11A] 1 illustrates a visualization with a modified environment scenario, according to one embodiment.

[0042] [Figure 11B] 1 illustrates a visualization with a modified environment scenario, according to one embodiment.

[0043] [Figure 12] 1 illustrates a rendered video object according to one embodiment.

[0044] [Figure 13] 1 illustrates a segmented geographic model according to one embodiment.

[0045] [Figure 14] 1 illustrates a segmentation architecture according to one embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0046] Aspects of this technical solution are described herein with reference to drawings that are illustrative examples of this technical solution. The following figures and examples are not intended to limit the scope of this technical solution to this implementation or a single implementation; for example, other implementations according to this implementation are possible by replacing some or all of the elements described or illustrated. When specific elements of this implementation can be partially or completely implemented using known components, only the parts of such known components necessary for understanding this implementation are described, and detailed descriptions of other parts of such known components are omitted so as not to obscure this implementation. Terms in this specification and claims should not be considered to have a general or special meaning unless explicitly stated herein. Furthermore, this technical solution and this implementation encompass current and future known equivalents to known components referenced herein for description, illustration, or example.

[0047] For example, the system may render a plurality of different three-dimensional environments having one or more aspects corresponding to physical features of the physical environment. For example, the system may obtain a ground truth data model corresponding to one or more aspects of the physical environment detected from the physical environment. For example, the aspects of the physical environment may include, but are not limited to, boundaries between the roadway and surrounding land, surface features of the roadway, surface markings on the roadway, traffic signs or signals at specific locations relative to the roadway, and traffic patterns through the roadway.

[0048] The system can modify aspects of the virtual environment to generate more instances than exist at a particular physical location or state of the physical location. For example, the system can modify a particular roadway marking to change the display of traffic flow, modify the wear level on the roadway surface, the water level on the roadway surface, roadway markings, or traffic signs, or modify one or more objects surrounding the roadway to correspond to weather, biome, or urban density levels. The system can include dynamic objects that can be captured by one or more viewpoints. For example, dynamic objects can include dynamic traffic objects including traffic lights or gates. In another example, dynamic objects can include dynamic environment objects including trees, branches, traffic cones, or other obstacles in the roadway, or dynamic environment objects that can affect traffic patterns or vehicles, pedestrians, or any combination thereof. Thus, this technical solution can provide at least a technical improvement for creating numerous permutations of real-world environments that would otherwise be unavailable and unable to be manually detected or rendered.

[0049] The system may allocate the generation of multiple portions of a physical area to achieve at least a technical improvement for creating numerous permutations of real-world environments that would otherwise not be available and could not be manually detected or rendered. For example, the system may divide a large geographic area into multiple physical locations and allocate one or more instructions for rendering the corresponding physical locations to one or more processors or processor cores. For example, the system may allocate different instructions for rendering different corresponding physical locations according to one or more aspects of the physical locations or one or more modifications to one or more aspects.

[0050] FIG. 1A is a non-limiting example of system components capable of implementing the methods and systems discussed herein. For example, an analytics server may train an artificial intelligence (AI) model and use the trained AI model to generate occupancy datasets and / or maps for one or more ego. FIG. 1A illustrates components of an AI-enabled visual data analytics system 100. System 100 may include analytics server 110a, system database 110b, administrator computing device 120, egos 140a-b (collectively ego 140), ego computing devices 141a-c (collectively ego computing devices 141), and server 160. System 100 is not limited to the components described herein and may include additional or other components not shown for the sake of brevity, which components should be considered within the scope of the embodiments described herein.

[0051] The above components may be connected via a network 130. Examples of network 130 may include, but are not limited to, a private or public LAN, a WLAN, a MAN, a WAN, and the Internet. Network 130 may include wired and / or wireless communications according to one or more standards and / or over one or more transport media.

[0052] Communications over network 130 may occur according to various communication protocols, such as Transmission Control Protocol / Internet Protocol (TCP / IP), User Datagram Protocol (UDP), IEEE communications protocols, etc. In one example, network 130 may include wireless communications according to the Bluetooth® set of specifications or another standard or proprietary wireless communications protocol. In another example, network 130 may also include communications over cellular networks, including, for example, Global System for Mobile Communications (GSM), Code Division Multiple Access (CDMA), or Enhanced Data for Global Evolution (EDGE) networks.

[0053] System 100 illustrates an example of a system architecture and components that may be used to train and execute one or more AI models, such as AI model 110c. Specifically, as depicted in FIG. 1A and described herein, analytics server 110a may train AI model 110c using data retrieved from ego 140 (e.g., by using data streams 172 and 174) using methods described herein. Once AI model 110c is trained, each ego 140 may access the trained AI model 110c and execute it. For example, vehicle 141a with ego computing device 140a may send its camera feed to trained AI model 110c and determine the occupancy status of its surroundings (e.g., data stream 174). Furthermore, data captured and / or predicted by AI model 110c for ego 140 (during inference) may be used to improve AI model 110c. Thus, system 100 describes a continuous loop that can periodically improve the accuracy of AI model 110c. Furthermore, the system 100 loops in such a way that the data received by the ego 140 can be used in the learning phase as well as the inference phase.

[0054] Analytics server 110a may be configured to collect, process, and analyze navigation data (e.g., images captured while navigating) and various sensor data collected from ego 140. The collected data may then be processed and prepared into a training dataset. The training dataset may then be used to train one or more AI models, such as AI model 110c. Analytics server 110a may also be configured to collect visual data from ego 140. Using AI model 110c (trained using the methods and systems discussed herein), analytics server 110a may generate a dataset and / or an occupancy map for ego 140. Analytics server 110a may display the occupancy map on ego 140 and / or transmit the occupancy map / dataset to ego computing device 141, administrator computing device 120, and / or server 160.

[0055] Although FIG. 1A shows AI model 110c as a component of system database 110b, AI model 110c may be stored in a different or separate component, such as cloud storage or any other data repository accessible to analytics server 110a.

[0056] The analytics server 110a may also be configured to display an electronic platform showing various training attributes for training the AI ​​model 110c. The electronic platform may be displayed on the administrator computing device 120 to allow an analyst to monitor the training of the AI ​​model 110c. An example of an electronic platform generated and hosted by the analytics server 110a may be a web-based application or website configured to display the training dataset collected from the ego 140 and / or the training status / metrics of the AI ​​model 110c.

[0057] Analysis server 110a may be any computing device equipped with a processor and non-transitory machine-readable storage capable of performing the various tasks and processes described herein. Non-limiting examples of such computing devices may include workstation computers, laptop computers, server computers, etc. Although system 100 includes a single analysis server 110a, system 100 may include any number of computing devices operating in a distributed computing environment, such as a cloud environment.

[0058] Ego 140 may represent various electronic data sources that transmit data associated with a previous or current navigation session to analytics server 110a. Ego 140 may be any device configured for navigation, such as vehicle 140a and / or truck 140c. Ego 140 is not limited to being a vehicle and may include robotic devices as well. For example, ego 140 may include robot 140b, which may represent a general-purpose, bipedal, autonomous humanoid robot capable of navigating various terrains. Robot 140b may be provided with software that enables balance, navigation, perception, or interaction with the physical world. Robot 140b may also include various cameras configured to transmit visual data to analytics server 110a.

[0059] Although referred to herein as “ego,” ego 140 may or may not be an autonomous device configured for automatic navigation. For example, in some embodiments, ego 140 may be controlled by a human operator or by a remote processor. ego 140 may include various sensors, such as those depicted in FIG. 1B . The sensors may be configured to collect data as ego 140 navigates various terrains (e.g., roads). Analytics server 110a may collect data provided by ego 140. For example, analytics server 110a may obtain navigation session and / or road / terrain data (e.g., images of ego 140 navigating roads) from various sensors, such that the collected data is ultimately used by AI model 110c for training purposes.

[0060] As used herein, a navigation session corresponds to a trip in which ego 140 travels a route, regardless of whether the trip was autonomous or controlled by a human. In some embodiments, the navigation session may be for data collection and model training purposes. However, in some other embodiments, ego 140 may refer to a vehicle purchased by a consumer, and the purpose of the trip may be classified as daily use. A navigation session may begin when ego 140 travels more than a threshold distance (e.g., 0.1 miles, 100 feet) from a non-moving location or exceeds a threshold speed (e.g., greater than 0 mph, greater than 1 mph, greater than 5 mph). A navigation session may end when ego 140 is returned to a non-moving location and / or turned off (e.g., when the driver exits the vehicle).

[0061] Ego 140 may correspond to a group of egos monitored by analytics server 110a to train AI model 110c. For example, drivers of vehicles 140a may authorize analytics server 110a to monitor data associated with their respective vehicles. As a result, analytics server 110a may collect sensor / camera data using various methods discussed herein and generate a training dataset for training AI model 110c accordingly. Analytics server 110a may then apply the trained AI model 110c to analyze data associated with ego 140 and predict an occupancy map for ego 140. Furthermore, additional / ongoing data associated with ego 140 may be processed and added to the training dataset, allowing analytics server 110a to recalibrate AI model 110c accordingly. Thus, system 100 performs a loop in which navigation data received from ego 140 can be used to train AI model 110c. Ego 140 may include a processor that executes the trained AI model 110c for navigation purposes. During navigation, ego 140 can collect additional data about their navigation session and can use the additional data to calibrate AI model 110c. That is, ego 140 corresponds to an ego that can be used to train, run / use, and recalibrate AI model 110c. In a non-limiting example, ego 140 corresponds to a vehicle purchased by a customer that can navigate autonomously using AI model 110c and simultaneously improve AI model 110c.

[0062] Egos 140 may be equipped with various technologies that enable them to gather data from their surroundings and (potentially) navigate autonomously. For example, ego 140 may be equipped with an inference chip for running self-driving software.

[0063] Various sensors for each ego 140 may monitor and transmit collected data associated with different navigation sessions to analytics server 110a. FIGS. 1B-1C show block diagrams of sensors incorporated within ego 140, according to one embodiment. The number and location of each sensor discussed with respect to FIGS. 1B-1C may depend on the type of ego discussed in FIG. 1A. For example, robot 140b may include different sensors than vehicle 140a or truck 140c. For example, robot 140b may not include airbag activation sensor 170q. Additionally, the sensors for vehicle 140a and truck 140c may be in different locations than those shown in FIG. 1C.

[0064] As discussed herein, various sensors incorporated within each ego 140 may be configured to measure various data associated with each navigation session. Analytics server 110a may periodically collect the data monitored and collected by these sensors, which is processed according to methods described herein and used to train and / or execute AI model 110c to generate an occupancy map.

[0065] Ego 140 may include user interface 170a. User interface 170a may refer to the user interface of an ego computing device (e.g., ego computing device 141 of FIG. 1A). User interface 170a may be implemented as a display screen integrated with or coupled to a vehicle's interior, a head-up display, a touchscreen, etc. User interface 170a may include input devices such as a touchscreen, knobs, buttons, a keyboard, a mouse, a gesture sensor, a steering wheel, etc. In various embodiments, user interface 170a may be adapted to provide user input (e.g., as types of signals and / or sensor information) to other devices or sensors of ego 140 (e.g., the sensors shown in FIG. 1B), such as controller 170c.

[0066] User interface 170a may also be implemented with one or more logic devices that may be adapted to execute instructions, such as software instructions, that implement any of the various processes and / or methods described herein. For example, user interface 170a may be adapted to form a communication link, send and / or receive communications (e.g., sensor signals, control signals, sensor information, user input, and / or other information), or perform various other processes and / or methods. In another example, a driver may use user interface 170a to control the temperature of ego 140 or activate its features (e.g., autonomous driving or steering system 170o). Accordingly, user interface 170a may monitor and collect driving session data in conjunction with other sensors described herein. User interface 170a may also be configured to display various data generated / predicted by analytics server 110a and / or AI model 110c.

[0067] Orientation sensor 170b may be implemented as one or more of a compass, float, accelerometer, and / or other digital or analog device capable of measuring the orientation of ego 140 (e.g., the magnitude and direction of roll, pitch, and / or yaw relative to one or more reference orientations, such as gravity and / or magnetic north). Orientation sensor 170b may be adapted to provide orientation measurements at ego 140. In other embodiments, orientation sensor 170b may be adapted to provide roll, pitch, and / or yaw rate of ego 140 using a time series of orientation measurements. Orientation sensor 170b may be positioned and / or adapted to provide orientation measurements relative to a particular coordinate frame of ego 140.

[0068] Controller 170c may be implemented as any suitable logic device (e.g., a processing device, microcontroller, processor, application specific integrated circuit (ASIC), field programmable gate array (FPGA), memory storage device, memory reader, or other device or combination of devices) that can be adapted to execute, store, and / or receive appropriate instructions, such as software instructions that implement control loops for controlling various operations of ego 140. Such software instructions may also process sensor signals, determine sensor information, provide user feedback (e.g., via user interface 170a), interrogate devices regarding operating parameters, select operating parameters for devices, or implement methods for performing any of the various operations described herein.

[0069] Communications module 170e may be implemented as any wired and / or wireless interface configured to communicate sensor data, configuration data, parameters, and / or other data and / or signals to any feature shown in FIG. 1A (e.g., analytics server 110a). As described herein, in some embodiments, communications module 170e may be implemented in a distributed manner, such that portions of communications module 170e are implemented within one or more elements and sensors shown in FIG. 1B. In some embodiments, communications module 170e may delay communication of sensor data. For example, when ego 140 does not have network connectivity, communications module 170e may store sensor data in temporary data storage and transmit the sensor data when ego 140 is identified as having adequate network connectivity.

[0070] Speed ​​sensor 170d may be implemented as an electronic pitot tube, a metering gear or wheel, a water speed sensor, a wind speed sensor, a wind speed sensor (e.g., direction and magnitude), and / or other device capable of measuring or determining the linear velocity of ego 140 (e.g., within the surrounding medium and / or aligned with the longitudinal axis of ego 140) and providing such measurement as a sensor signal that can be communicated to various devices.

[0071] Gyroscope / accelerometer 170f may be implemented as an electronic sextant, a semiconductor device, an integrated chip, an accelerometer sensor, or other system or device capable of measuring angular velocity / acceleration and / or linear acceleration (e.g., direction and magnitude) of ego 140 and providing such measurements as sensor signals that can be communicated to other devices, such as analytics server 110a. Gyroscope / accelerometer 170f may be positioned and / or adapted to make such measurements relative to a particular coordinate frame of ego 140. In various embodiments, gyroscope / accelerometer 170f may be mounted in a common housing and / or module with other elements shown in FIG. 1B to ensure a common frame of reference or known transformations between frames of reference.

[0072] Global Navigation Satellite System (GNSS) 170h may be implemented as a global positioning satellite receiver and / or another device capable of determining the absolute and / or relative position of ego 140 based on, for example, radio signals received from space-born and / or terrestrial sources and providing such measurements as sensor signals that can be communicated to various devices. In some embodiments, GNSS 170h may be adapted to determine the velocity, speed, and / or yaw rate of ego 140 (e.g., using a time series of position measurements), such as the absolute velocity and / or yaw component of the angular velocity of ego 140.

[0073] Temperature sensor 170i may be implemented as a thermistor, an electrical sensor, an electrical thermometer, and / or other device capable of measuring a temperature associated with ego 140 and providing such measurement as a sensor signal. Temperature sensor 170i may be configured to measure an environmental temperature associated with ego 140, such as a cockpit or dash temperature, which may be used to estimate the temperature of one or more elements of ego 140.

[0074] Humidity sensor 170j may be implemented as a relative humidity sensor, an electrical sensor, an electrical relative humidity sensor, and / or another device capable of measuring the relative humidity associated with ego 140 and providing such measurement as a sensor signal.

[0075] Steering sensor 170g may be adapted to physically adjust the orientation of ego 140 according to one or more control signals provided by a logic device, such as controller 170c, and / or user input. Steering sensor 170g may include one or more actuators and control surfaces of ego 140 (e.g., rudders or other types of steering or trim mechanisms) and may be adapted to physically adjust the control surfaces to various positive and / or negative steering angles / positions. Steering sensor 170g may also be adapted to sense the current steering angles / positions of such steering mechanisms and provide such measurements.

[0076] Propulsion system 170k may be implemented as a propeller, turbine, or other thrust-based propulsion system, a mechanical wheeled and / or tracked propulsion system, a wind / sail-based propulsion system, and / or other type of propulsion system that may be used to provide motive power to ego 140. Propulsion system 170k may also monitor the direction of motive power and / or thrust of ego 140 relative to a coordinate frame of reference of ego 140. In some embodiments, propulsion system 170k may be coupled to and / or integrated with steering sensor 170g.

[0077] Occupant restraint sensor 170l may monitor the seat belt detection and lock / unlock assembly, as well as other occupant restraint subsystems. Occupant restraint sensor 170l may include various environmental and / or status sensors, actuators, and / or other devices that facilitate operation of safety mechanisms associated with operation of ego 140. For example, occupant restraint sensor 170l may be configured to receive movement and / or status data from other sensors shown in FIG. 1B. Occupant restraint sensor 170l may determine whether a safety mechanism (e.g., a seat belt) is engaged.

[0078] Camera 170m may refer to one or more cameras integrated into ego 140, as depicted in FIG. 1C, or may include multiple cameras integrated into (or retrofitted to) ego 140. Camera 170m may be an inward-facing or outward-facing camera of ego 140. For example, as depicted in FIG. 1C, ego 140 may include one or more inward-facing cameras that can monitor and collect video of occupants of ego 140. Ego 140 may include eight outward-facing cameras. For example, ego 140 may include front camera 170m-1, forward-looking side camera 170m-2, forward-looking side camera 170m-3, rearward-looking side camera 170m-4 on each front fender, camera 170m-5 on each side (e.g., integrated into the B-pillar), and rear camera 170m-6.

[0079] 1B, radar 170n and ultrasonic sensor 170p may be configured to monitor the distance of ego 140 to other objects, such as other vehicles or immovable objects (e.g., trees or garage doors). Ego 140 may also include an automatic driving or steering system 170o configured to autonomously navigate ego 140 using data collected via various sensors (e.g., radar 170n, speed sensor 170d, and / or ultrasonic sensor 170p).

[0080] Thus, automated driving or steering system 170o may analyze various data collected by one or more sensors described herein to identify driving data. For example, automated driving or steering system 170o may calculate the risk of a forward collision based on ego 140's speed and its distance to another vehicle on the road. Autonomous driving or steering system 170o may also determine whether the driver is touching the steering wheel. Autonomous driving or steering system 170o may transmit the analyzed data to various features discussed herein, such as an analytics server.

[0081] Airbag deployment sensor 170q may predict or detect a crash and cause one or more airbags to deploy or inflate. Airbag deployment sensor 170q may transmit data regarding the deployment of the airbags, including data associated with the event that caused the deployment.

[0082] 1A , administrator computing device 120 may represent a computing device operated by a system administrator. Administrator computing device 120 may be configured to display data retrieved or generated by analytics server 110a (e.g., various analytics metrics and risk scores), allowing the system administrator to monitor various models utilized by analytics server 110a, review feedback, and / or facilitate training of AI models 110c maintained by analytics server 110a.

[0083] Ego 140 may be any device configured to navigate various routes, such as vehicle 140a or robot 140b. As discussed with respect to FIGS. 1B-1C, ego 140 may include various telemetry sensors. Ego 140 may also include ego computing device 141. Specifically, each ego may have its own ego computing device 141. For example, truck 140c may have ego computing device 141c. For simplicity, ego computing devices are collectively referred to as ego computing device 141. Ego computing device 141 may control the presentation of content on ego 140's infotainment system, process commands related to the infotainment system, aggregate sensor data, manage communication of data to electronic data sources, receive updates, and / or send messages. In one configuration, ego computing device 141 communicates with an electronic control unit. In another configuration, ego computing device 141 is an electronic control unit. Ego computing device 141 may include a processor and non-transitory machine-readable storage media capable of performing the various tasks and processes described herein. For example, AI model 110c described herein may be stored and executed (or directly accessed) by ego computing device 141. Non-limiting examples of ego computing device 141 may include a vehicle multimedia and / or display system.

[0084] In one example of how AI model 110c can be trained, analytics server 110a may collect data from simulated egos 140 to train AI model 110c. Before running AI model 110c to generate / predict a training dataset, analytics server 110a may generate training data. Training allows AI model 110c to capture data from one or more simulated cameras of one or more simulated egos 140 (without having to receive radar data). The operations described in this example may be performed by any number of computing devices operating in the distributed computing system described in FIGS. 1A and 1B.

[0085] To train the AI ​​model 110c, the analytics server 110a may first use one or more egos 140 to drive a particular simulated route within the virtual environment. During the drive, the egos 140 may use one or more of their simulated sensors (including one or more simulated cameras) to generate navigation session data. For example, one or more of the simulated egos 140 equipped with various simulated sensors may navigate a specified route within the virtual environment. As the one or more egos 140 traverse their terrain, the ego's simulated sensors may capture continuous (or periodic) data of their surroundings. The simulated sensors may capture visual information about the one or more egos 140's surroundings.

[0086] Analytics server 110a may generate a training data set using data collected from ego 140 (e.g., simulated camera feeds received from ego 140). The training data set may represent video from one or more simulated cameras corresponding to viewpoints from simulated ego 140 within one or more ego 140 surroundings.

[0087] In operation, as one or more egos 140 navigate, their sensors collect data, as depicted in data stream 172, and transmit the data to analytics server 110a.

[0088] In some embodiments, one or more egos 140 may include one or more high-resolution cameras that capture a continuous stream of visual data from around the one or more egos 140 as the one or more egos 140 navigate through a route. Analytics server 110a may then use the camera feeds to generate a second dataset, with visual elements / representations of different voxels around the one or more egos 140 included in the second dataset.

[0089] In operation, as one or more egos 140 navigate, their cameras collect data and transmit the data to analytics server 110a, as depicted in data stream 172. For example, ego computing device 141 may use data stream 172 to transmit image data to analytics server 110a.

[0090] 2 depicts an exemplary method for simulating viewpoint capture from a rendered environment with ground truth heuristics according to the present disclosure. The system or vehicle of any of FIGS. 1A-1C may perform method 200.

[0091] At 210, method 200 can retrieve a camera feed of an ego object navigating within the physical environment. For example, the camera feed can correspond to a series of images or video captured from a virtual environment from a viewpoint within the virtual environment. For example, the ego object can correspond to a simulated ego object driving within the virtual environment.

[0092] At 220, method 200 can generate a 3D model including a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics, where the one or more first environmental metrics indicate boundaries of the one or more physical paths. For example, the boundaries can correspond to one or more points, vectors, planes, or volumes that define edges and surface topology of one or more roadway surfaces. For example, the boundaries can be obtained or generated from detection of the surface topology of the physical environment via one or more sensors. For example, the boundaries can define a road and curb mesh.

[0093] At 230, method 200 can generate one or more geometric 2D objects on the first surface according to one or more second environmental metrics, where the second environmental metrics represent one or more physical paths. For example, the 2D objects can correspond to one or more points, vectors, planes, textures, images, or patterns that define two-dimensional objects on one or more roadway surfaces. For example, the 2D objects can represent lane markings, directional markings, or any combination thereof. For example, the 2D objects can be obtained or generated from detecting surface images of the physical environment via one or more sensors. For example, the 2D objects can define lane paint decals and directional road markings.

[0094] At 240, method 200 can identify one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicative of a camera of a physical object configured to move along one or more physical paths. For example, the camera of the physical object can serve as a simulated viewpoint in the virtual environment and capture a rendered video object. The viewpoints can include a forward view, a right-front view, a right-rear view, a left-front view, and a left-rear view. Each of the viewpoints can correspond to a 2D image or 2D video captured by a simulated ego vehicle traveling through the virtual environment.

[0095] At 250, method 200 can render one or more 2D images, each corresponding to a respective one of the one or more viewpoints, from one or more corresponding portions of the 3D model. For example, method 200 can render a 2D video by capturing images over time from simulated viewpoints.

[0096] At 260, method 200 can train an artificial intelligence model according to the camera feed and one or more simulated environment 2D images. For example, AI model 110c can be used for the training scenario. For example, the system can acquire one or more models generated from measurements of the boundaries of various roads, medians, crosswalks, sidewalks, bike lanes, or any other feature of the outdoor built environment. The system can add textures and objects to the roadway and surrounding space to generate a visualization of the intersection as it appears in the real world. The system can modify, remove, or add road signs, road surfaces, road markings, weather conditions, road obstructions, traffic markers, or any combination thereof to create multiple variations of the intersection while maintaining the measured characteristics of the intersection as they appear in the real world. When varying the environment, the system can change the time of day, weather, or other conditions. In this way, the system can provide technical improvements for providing multiple variations of real-world locations to provide training data that far exceeds the amount and type that can be obtained by manual processes in the real world, at least in part because many of the environments that can be generated by this technical solution do not and cannot exist in the real world.

[0097] 3 depicts an exemplary system architecture according to this disclosure. As shown by way of example in FIG. 3, system architecture 300 can include at least a ground truth model 310 and a geospatial model 320, which variously depict, but are not limited to, road and curb mesh 330, lane paint decals 332, islands 334, scenario permutations 340, directional road markings 342, traffic lights and stop signs 350, buildings 360, and yards and areas outside of drivable area 362. For example, AI model 110c can have or be configured according to system architecture 300.

[0098] The ground truth model 310 may include data structures that represent physical aspects of a real-world physical environment. For example, the ground truth model 310 may correspond to spatial or geographic identifiers in a coordinate space. For example, the ground truth model 310 may correspond to one or more collections of one or more spatial elements, including points, vectors, planes, volumes, textures, images, patterns, or any combination thereof. The spatial elements may be defined or determined, for example, according to a coordinate system that represents the physical environment or a coordinate system that can be transformed to represent the physical environment. For example, the spatial elements may correspond to one or more of latitude, longitude, or altitude, or may be defined relative to other spatial elements. The ground truth model 310 may correspond to a group of models, each representing a different aspect of the physical environment. The ground truth model 310 may include a road boundary model 312, a road line model 314, a median edge model 316, and a lane graph model 318. For example, the ground truth model 310 may be derived from measurements of the real-world physical environment.

[0099] The road boundary model 312 may represent the structure of one or more outer edges of one or more roadways in the physical environment. For example, the road boundary model 312 may correspond to one or more points, vectors, planes, or volumes that define the edges and surface topology of one or more roadway surfaces. For example, the road boundary model 312 may be obtained or generated from detecting the surface topology of the physical environment via one or more sensors. For example, the road boundary model may define a road and curb mesh 330. The road line model 314 may represent the structure of one or more markings on one or more roadways in the physical environment. For example, the road line model 314 may correspond to one or more points, vectors, planes, textures, images, or patterns that define two-dimensional objects on one or more roadway surfaces. For example, the road line model 314 may represent lane markings, directional markings, or any combination thereof. For example, the road line model 314 may be obtained or generated from detecting surface images of the physical environment via one or more sensors. For example, road line model 314 may define lane paint decals 332 and directional road markings 342 .

[0100] Median edge model 316 may represent the structure of one or more interior edges of one or more roadways in a physical environment. For example, road boundary model 312 or median edge model 316 may correspond to one or more points, vectors, planes, or volumes that define the edges and surface topology of one or more roadway surfaces that are at least partially within one or more of road boundary models 312. For example, median edge model 316 may be obtained or generated from detection of the surface topology of the physical environment via one or more sensors. For example, median edge model 316 may define islands 334 that correspond to impassable areas surrounded by the roadways.

[0101] Lane graph model 318 may represent a pattern of movement of one or more movable objects within a roadway in a physical environment. For example, lane graph model 318 may correspond to one or more points, vectors, planes, or volumes that define a path along one or more roadway surfaces according to one or more indications of road line model 314. For example, road boundary model 312 may be obtained or generated from detection of the surface topology of the physical environment via one or more sensors.

[0102] Geospatial model 320 may include data structures that describe physical aspects of a real-world physical environment that differ from ground truth model 310. For example, geospatial model 320 may correspond, at least in part, to ground truth model 310 in one or more of structure and behavior. Geospatial model 320 may be obtained from sensors external to or different from the sensors used to detect ground truth model 310. For example, sensors on one or more roadway-traversing vehicles may detect data to generate ground truth model 310, and corresponding data from satellite sensors or a geographic information system (GIS) may generate geospatial model 320. Geospatial model 320 may include a map model 322, an environment model 324, and a scenario permutation model 326.

[0103] Map model 322 may represent the structure of one or more objects along one or more roadways in the physical environment. For example, road boundary model 312 may correspond to one or more points, vectors, planes, or volumes that define the shape and position of one or more objects distinct from the roadway surface. For example, map model 322 may be obtained or generated from detection of images of the physical environment above or around the roadway via one or more sensors. For example, map model 322 may define one or more of traffic light and stop sign data 350.

[0104] The environment model 324 may represent the structure of one or more objects along one or more roadways in the physical environment. For example, the road boundary model 312 may correspond to one or more points, vectors, planes, or volumes that define the shape and position of one or more objects distinct from the roadway surface. For example, the environment model 324 may be obtained or generated from detecting images of the physical environment above or around the roadway via one or more sensors.

[0105] Scenario permutation model 326 may include one or more instructions for modifying one or more of ground truth model 310 or geospatial model 320 to correspond to a particular 3D model of the physical environment. For example, scenario permutation model 326 may include one or more instructions for populating or transforming one or more environmental models 324 into one or more of scenario permutations 340 according to one or more environmental heuristics. For example, environmental heuristics may correspond to weather, biomes, or levels of urban density. For example, scenario permutation model 326 may modify or replace one or more of buildings 360 and yards and areas outside drivable area 362 according to environmental heuristics.

[0106] 4 depicts an exemplary ground truth visualization according to this disclosure. As shown by way of example in FIG. 4, ground truth model 400 may include at least road boundary model 410, road line model 420, median edge model 430, and lane graph model 440. Ground truth model 400 may correspond to, but is not limited to, a 3D model including at least one of ground truth model 310 or geospatial model 320.

[0107] Road boundary model 410 may correspond to a rendering or instantiation of one of road boundary models 312 representing road 330. For example, road boundary model 410 may include contours representing one or more intersecting roadways. Road line model 420 may correspond to a rendering or instantiation of one of road line models 314 representing one or more of lane paint decals 332. For example, road line model 420 may include contours representing one or more markings on the corresponding surfaces of one or more intersecting roadways. Median edge model 430 may correspond to a rendering or instantiation of a median edge model for one of median edge models 316 representing one or more of islands 334. For example, median edge model 430 may include contours representing one or more islands that are at least partially within one or more intersecting roadways of road boundary model 410. Lane graph model 440 may correspond to a rendering or instantiation of one of lane graph models 318, which illustrates one or more travel paths of a corresponding vehicle, pedestrian, or any combination thereof. For example, lane graph model 440 may include paths for one or more vehicles along one or more intersecting roadways. For example, lane graph model 440 may include paths for one or more pedestrians crossing one or more intersecting roadways.

[0108] FIG. 5 depicts an exemplary road visualization according to this disclosure. As shown by way of example in FIG. 5, road model 500 may include at least road topology model 510. Road topology model 510 may correspond to a rendering or instantiation of the surface topology of the road surface according to a road boundary model of road boundary model 312 that represents curb mesh 330. For example, road topology model 510 may include a mesh surface representing the surface elevation of one or more intersecting roadways at one or more corresponding points along the roadway. Road boundary model 520 may correspond, at least in part, in one or more of structure and behavior to road boundary model 410.

[0109] FIG. 6 depicts an exemplary road surface visualization according to this disclosure. As shown by way of example in FIG. 6 , a road surface model 600 may include at least a textured road topology model 610. The textured road topology model 610 may correspond to the road topology model 510 having a rendered road surface that represents the road surface at a physical location. For example, the textured road topology model 610 may include a road surface having a texture that corresponds to or represents asphalt. For example, the textured road topology model 610 may include one or more surface features or may have one or more surface features present thereon. The textured road topology model 610 may include a topology feature 620 and an environmental feature 630. The topology feature 620 may correspond to a portion of the textured road topology model 610 having a topology that represents a road feature having a shape that differs from the plane corresponding to the road topology model 510 surrounding the road feature. For example, the road feature may correspond to, but is not limited to, a pothole or a speed bump. The environmental feature 630 may correspond to a portion of the textured road topology model 610 having environmental characteristics that differ from the road characteristics corresponding to the road topology model 510 surrounding the road feature. For example, but not limited to, the environmental characteristic may correspond to a puddle having a first reflectance that is greater than a second reflectance of the asphalt of the textured road topology model 610. The road boundary model 640 may correspond, at least in part, in one or more of structure and behavior to the road boundary model 410.

[0110] FIG. 7A depicts an exemplary road line visualization according to this disclosure. As shown by way of example in FIG. 7A , road line model 700A may include at least a textured road topology model 710A. Textured road topology model 710A may correspond, at least in part, in one or more of structure and behavior, to textured road topology model 610. Road line model 700A may include road line model 712 and textured road topology model 610, and may correspond to a pre-rendered 3D model state of one or more of lane paint decals 332 corresponding to road line model 712. Road line model 712 may correspond, at least in part, in one or more of structure and behavior, to road line model 420.

[0111] 7B depicts an exemplary road line visualization according to this disclosure. As shown by way of example in FIG. 7B, road line model 700B may include at least a line-mapped road surface 710B. Line-mapped road surface 710B may correspond, at least in part, in one or more of structure and behavior to textured road topology model 710A and may include one or more of lane paint decals 332 corresponding to road line model 420. Thus, road line model 700B may correspond to a rendered 3D model state of one or more of lane paint decals 332 corresponding to road line model 420.

[0112] FIG. 8A depicts an exemplary exterior visualization according to this disclosure. As shown by way of example in FIG. 8A , the exterior model 800A may include at least a line-mapped road surface 802, a median edge model 804, a median region 810A, and a block region 820A. The median region 810A may correspond to a portion of the physical environment corresponding to one or more islands 334. The median region 810A may be at least partially surrounded by at least a portion of the road line model 420. The block region 820A may correspond to a portion of the physical environment corresponding to one or more outsides of the drivable area 362. The block region 820A may be at least partially outside at least a portion of the road line model 420. For example, the block region 820A may correspond to a portion of the physical environment representing a city block outside a public road, including, but not limited to, a roadway. The line-mapped road surface 802 may correspond, at least in part, to the line-mapped road surface 710B in one or more of its structure and operation. Median edge model 804 may correspond, at least in part, in one or more of structure and behavior to median edge model 430 .

[0113] 8B depicts an example populated exterior visualization according to this disclosure. As shown by way of example in FIG. 8B, a populated exterior model 800B can include at least a populated median region 810B and a populated block region 820B.

[0114] The populated median area 810B may correspond, at least in part, in one or more of structure and operation, to the median area 810A and may include one or more objects at least partially disposed within one or more islands 334 corresponding to the median area 810A and the populated median area 810B. For example, the populated median area 810B may correspond to one or more islands 334 having one or more tree objects thereon and including a grass surface having a color and texture different from the color and texture of the road surface 710B. For example, the tree objects may include dynamic components that can interact with the 3D environment according to one or more of the scenario permutations 340. For example, the tree objects may include branch or leaf objects covering at least a portion of the populated median area 810B or the road surface 710B. The populated block area 820B may correspond, at least in part, in one or more of structure and operation, to the block area 820A and may include one or more objects at least partially disposed within the populated block area 820B. For example, populated block region 820B may correspond to one or more areas outside drivable area 330 that have one or more building or tree objects thereon and that include a sidewalk surface that has a color and texture that differs from the color and texture of road surface 710B. For example, populated median region 810B and one or more objects in populated block region 820B may be modified or replaced according to one or more of scenario permutations 340.

[0115] FIG. 9A depicts an exemplary traffic object model according to this disclosure. As shown by way of example in FIG. 9A , traffic object model 900A may include at least a line-mapped road surface 902 and a map data model 910. Map data model 910 may correspond, at least in part, to map model 322 in one or more of its structure and behavior. For example, map data model 910 may correspond to one of map models 332 for road surface 710B. Map data model 910 may include one or more indicators corresponding to one or more traffic objects corresponding to specific physical locations. For example, map data model 910 may include position data for one or more traffic lights or stop signs 350. Line-mapped road surface 902 may correspond, at least in part, to line-mapped road surface 802 in one or more of its structure and behavior.

[0116] FIG. 9B illustrates an exemplary populated traffic object model according to this disclosure. As shown by way of example in FIG. 9B, the populated traffic object model 900B may include at least a dynamic traffic object 920 and a static traffic object 930. The dynamic traffic object 920 may correspond to an object having multiple states that may be rendered in a 3D model corresponding to a physical environment. For example, the dynamic traffic object 920 may correspond to a traffic light that may be rendered to show multiple states according to a three-light signal in vehicular traffic. For example, the dynamic traffic object 920 may correspond to a traffic light that may be rendered to show multiple states according to a multi-state lighted sign in pedestrian or cyclist traffic. The static traffic object 930 may correspond to an object having multiple states that may be rendered in a 3D model corresponding to a physical environment. For example, the static traffic object 930 may correspond to a stop sign that may be rendered in a single state according to a stop sign texture. For example, the static traffic object 930 may correspond to a road sign that may be rendered in a single state according to a road sign texture and a location indicated by the map data model 910.

[0117] FIG. 10 depicts an exemplary populated traffic environment according to this disclosure. As shown by way of example in FIG. 10 , the populated traffic environment 1000 may include at least a line-mapped road surface 1002, a lane graph model 1004, roadway traffic objects 1010, and intersecting traffic objects 1020. The roadway traffic objects 1010 may correspond to 3D objects that can move along one or more paths in the lane graph model 440 corresponding to the road surface 1002. For example, the roadway traffic objects 1010 may correspond to vehicles, including, but not limited to, cars, bicycles, motorcycles, trucks, or any combination thereof. The intersecting traffic objects 1020 may correspond to 3D objects that can move along one or more paths in the lane graph model 440 corresponding to the road surface 1002. For example, the roadway traffic objects 1010 may correspond to vehicles, including, but not limited to, cars, bicycles, motorcycles, trucks, or any combination thereof. Line mapped road surface 1002 may correspond, at least in part, in one or more of structure and behavior, to line mapped road surface 710B. Lane graph model 1004 may correspond, at least in part, in one or more of structure and behavior, to lane graph model 1004.

[0118] 11A depicts an exemplary model with a modified environmental scenario according to this disclosure. As shown by way of example in FIG. 11A , the model with the modified environmental scenario 1100A may include at least a line-mapped road surface 1102, environmental features 1104, a modified weather environment 1110A, and a locale environment 1120A. The line-mapped road surface 1102 may correspond, at least in part, to the line-mapped road surface 710B in one or more of its structure and behavior. The environmental features 1004 may correspond, at least in part, to the environmental features 630 in one or more of its structure and behavior.

[0119] The modified weather environment 1110A can correspond, at least in part, in one or more of structure and operation to the populated traffic environment 1000 and can include one or more visual characteristics corresponding to weather conditions that differ from the real-world physical environment captured by one or more sensors. For example, the modified weather environment 1110A can include modifying the road surface 710B to include additional puddle objects, modifying tree objects to include additional leaf or branch objects on the road surface 710B, or can include one or more objects or filters corresponding to reduced visibility distance in the populated traffic environment 1000. Thus, the modified weather environment 1110A can at least provide a technical improvement that eliminates or minimizes additional sensor capture of the physical environment in multiple weather conditions while maintaining model accuracy with respect to the corresponding physical environment in the real world according to the ground truth model 310.

[0120] The locale environment 1120A can correspond, at least in part, in one or more of structure and operation, to the populated traffic environment 1000 and can include one or more visual characteristics corresponding to a locale associated with a real-world physical environment. For example, the locale environment 1120A can be linked to an urban environment and to a plurality of objects each linked to the urban environment. The plurality of objects can include, for example, models of buildings 360 of a predetermined size, height, or footprint. Thus, the locale environment 1120A can correspond to a locale or biome corresponding to a real-world physical environment captured by one or more sensors.

[0121] 11B depicts an exemplary model with a modified environment scenario according to this disclosure. As shown by way of example in FIG. 11B, the model with a modified environment scenario 1100B can include at least a weather environment 1110B and a modified locale environment 1120B.

[0122] Weather environment 1110B can correspond, at least in part, in one or more of structure and operation to populated traffic environment 1000 and can include one or more visual characteristics corresponding to weather conditions in a real-world physical environment captured by one or more sensors. For example, weather environment 1110B can include puddle objects, leaf or branch objects on road surface 710B, or road surface 710B without objects or filters corresponding to reduced visibility distances in populated traffic environment 1000.

[0123] The modified locale environment 1120B can correspond, at least in part, in one or more of its structure and operation to the populated traffic environment 1000 and can include one or more visual characteristics corresponding to a locale different from the real-world physical environment. For example, the modified locale environment 1120B can be linked to a rural, rural, or tropical environment and can be linked to a plurality of objects each linked to the corresponding environment. The plurality of objects can include, for example, a model of a building 360 of a predetermined size, height, or footprint. Thus, the modified locale environment 1120B can correspond to a locale or biome different from the real-world physical environment captured by one or more sensors. Thus, the modified locale environment 1120B can provide at least a technical improvement to create additional locations with distinct physical characteristics that do not exist in the real world, while maintaining the accuracy of the model relative to the corresponding real-world physical environment according to the ground truth model 310.

[0124] 12 depicts an exemplary rendered video object according to this disclosure. As shown by way of example in FIG. 12, rendered video object 1200 may include at least forward-viewing video object 1210, right forward-viewing video object 1220, right rear-viewing video object 1230, left forward-viewing video object 1240, and left rear-viewing video object 1250. Each of video objects 1210, 1220, 1230, 1240, and 1250 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 traveling through one or more of environments 900B, 1000, 1100A, or 1100B. Forward-viewing video object 1210 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 from a first simulated viewpoint that is oriented toward and facing away from the front of simulated ego vehicle 140. Right forward looking video object 1220 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 from a second simulated viewpoint that is oriented toward and facing away from the right front of simulated ego vehicle 140. Right rear looking video object 1230 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 from a third simulated viewpoint that is oriented toward and facing away from the right rear of simulated ego vehicle 140. Left forward looking video object 1240 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 from a fourth simulated viewpoint that is oriented toward and facing away from the left front of simulated ego vehicle 140. Left rear view video object 1250 may correspond to a 2D image or 2D video captured by simulated ego vehicle 140 from a fifth simulated viewpoint that is directed toward and faces away from the left rear of simulated ego vehicle 140.

[0125] FIG. 13 depicts an exemplary segmented geographic model according to the present disclosure. As shown by way of example in FIG. 13 , the segmented geographic model 1300 may include at least a geographic region model 1310. The geographic region model 1310 may correspond to a geographic region representing a physical area. For example, the geographic region model 1310 may correspond to the physical region of San Francisco, California. The geographic region model 1310 may be linked to, define, or include one or more ground truth models 310 and one or more geospatial models 320 corresponding to multiple physical locations in the geographic region, for example. The geographic region model 1310 may include region segments 1320. Each of the region segments 1320 may correspond to a respective portion of the geographic region model 1310. For example, the size of each of the region segments 1320 may be determined according to the amount or type of computing resources required to render a 3D model based on the ground truth model 310 and one or more geospatial models 320 associated with each of the region segments 1320. For example, one or more of the region segments 1320 may be sized to include a physical area that includes the ground truth model 310 and one or more geospatial models 320 that may be allocated within the computational limits of each processor or processor core of a multiprocessor system. The multiprocessor system may correspond to any of the computing environments discussed herein.

[0126] 14 depicts an exemplary segmentation architecture according to this disclosure. As shown by way of example in FIG. 14 , the segmentation architecture 1400 may include at least a tile creator 1410, a tile extractor 1420, a tile loader 1430, a rendering engine 1440, and one or more rendered video objects 1450, and may obtain one or more of a ground truth model 1402 and a geospatial model 1404. The ground truth model 1402 may correspond, at least in part, in one or more of structure and operation, to the ground truth model 310. The geospatial model 1404 may correspond, at least in part, in one or more of structure and operation, to the geospatial model 320.

[0127] The tile creator 1410 can determine the size of one or more tiles according to at least one of the ground truth model 310 and the geospatial model 320. For example, the tile creator 1410 can identify the amount of computing resources required to render one or more portions of the geographic region model 1310 according to the ground truth model 310 and the geospatial model 320 associated with each of those portions. The tile creator 1410 can then generate region segments 1320 according to the determined sizes according to the computational limitations of each processor or processor core in the multiprocessor system.

[0128] The tile extractor 1420 can identify models corresponding to each of the region segments 1320. For example, the tile extractor 1420 can identify portions of the ground truth model 310 and the geospatial model 320 with coordinates that lie within the respective boundaries of each region segment 1320. The tile extractor 1420 can include a model geometry 1422 and a model instance 1424. The model geometry 1422 can correspond to portions of the ground truth model 310 and the geospatial model 320 that represent a real-world physical location. For example, the model geometry 1422 can correspond to one or more of models 410, 420, 430, and 440. The model instance 1424 can correspond to portions of the ground truth model 310 and the geospatial model 320 that represent an object at a real-world physical location. For example, the model instance 1424 can correspond to one or more of models 350 and 360.

[0129] The tile loader 1430 can assign portions of the ground truth model 310 and the geospatial model 320 that represent real-world physical locations to corresponding processors or processor cores. For example, the tile loader 1430 can identify a processor core that has computational limits that correspond to or exceed the computational requirements for a corresponding one of these portions. The rendering engine 1440 can convert one or more of the region segments 1320 into one or more corresponding 3D models. The rendering engine can, for example, generate the region segments 1320 according to the environments 400-100B, as discussed herein.

[0130] In one example of operation, embodiments herein may be used for training scenarios. For example, the system may obtain one or more models generated from measurements of the boundaries of various roads, medians, crosswalks, sidewalks, bike lanes, or any other feature of the outdoor built environment. The system may obtain information generated from measurements from driving through the roadway (e.g., a vehicle equipped with a camera, radar, lidar, or any combination) or by externally detecting aspects of the roadway or surrounding environment (e.g., by satellite imagery or a geospatial database of the area). The system may apply the measurements to generate a model of a given intersection in a given city block. The system may add texture and objects to the roadway and surrounding space to generate a visualization of the intersection as it appears in the real world. The system may include a traffic pattern path that controls the movement of vehicles, pedestrians, and other simulated roadway objects through the intersection. The system may modify, remove, or add road signs, road surfaces, road markings, weather conditions, road obstructions, traffic markers, or any combination thereof to create numerous variations of the intersection while maintaining the measured characteristics of the intersection as they appear in the real world. In this way, the system can provide technical improvements to provide a large number of variations of real-world locations to provide training data that far exceeds the amount and type that can be obtained by manual processes in the real world, at least for the reason that many of the environments that can be generated by this technical solution do not and cannot exist in the real world. For example, an intersection with two-way and one-way traffic patterns on the same roadway cannot exist in the real world. However, this technical solution can create training data that matches both of these traffic modes.

[0131] Although several exemplary implementations have been described herein, the foregoing are presented by way of example, not limitation. In particular, while many of the examples presented herein involve particular combinations of method operations or system elements, those operations and those elements may be combined in other ways to achieve the same purpose. Operations, elements, and features discussed in connection with one implementation are not intended to be excluded from similar roles in other implementations.

[0132] The phraseology and terminology used herein are for purposes of description and should not be considered limiting. The use of "including," "comprising," "having," "containing," "involving," "characterized by," "characterized in that," and variations thereof herein are meant to exclusively encompass the subsequently listed items, equivalents thereof, and additional items, as well as alternative implementations consisting of the subsequently listed items. In one implementation, the systems and methods described herein consist of one, each combination of more than one, or all of the described elements, operations, or components.

[0133] References to "or" may be construed as inclusive, such that any term listed using "or" may refer to either one, more than one, or all of the listed terms. Reference to at least one of a conjunctive list of terms may be construed as an inclusive OR, referring to either one, more than one, or all of the listed terms. For example, a reference to "at least one of 'A' and 'B'" may include 'A' alone, 'B' alone, and both 'A' and 'B'. Such references used in conjunction with "comprising" or other open terminology may include additional items. References to "is" or "are" may be construed as not limiting the implementation or operations referred to in connection with that term. The terms "is" or "are" or any tenses or derivatives thereof are interchangeable and synonymous with "can be" when used herein, unless otherwise stated herein.

[0134] The directional indicators shown herein are exemplary directions for facilitating understanding of the examples described herein and are not limited to the directional indicators shown herein. Any directional indicators shown herein can be modified to the opposite direction or to include both the depicted direction and the opposite direction to the depicted direction, unless otherwise stated herein. Although the figures depict operations in a particular order, such operations do not have to be performed in the particular order shown, or in sequential order, and not all depicted operations need be performed. Operations described herein can be performed in different orders. Where a reference sign follows a technical feature in a drawing, detailed description, or any claim, the reference sign is included to enhance the clarity of the drawing, detailed description, and claims. Therefore, the reference signs, or their absence, do not have a limiting effect on the scope of any claim element.

[0135] Accordingly, the scope of the systems and methods described herein is indicated by the appended claims, rather than the foregoing description, and the claims include equivalents thereto.

Claims

1. a non-transitory memory and one or more processors, the processors Search the camera feed of the ego object navigating within the physical environment, generating a three-dimensional (3D) model including a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics based on the camera feed, the one or more first environmental metrics indicating boundaries of the one or more physical paths; generating one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, the second environmental metrics indicative of the one or more physical paths; identifying one or more viewpoints oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicative of cameras of physical objects configured to move along the one or more physical paths; rendering one or more simulated environment 2D images from one or more corresponding portions of the 3D model of the physical environment, each corresponding to one of the viewpoints; training, by the processor, an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images; The system is configured as follows:

2. The processor:

2. The system of claim 1, configured to modify a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicating a condition of the physical path.

3. The processor: The system of claim 1 , configured to modify a topology of the portion of the first surface according to one or more third environmental metrics.

4. The processor:

2. The system of claim 1, configured to modify an opacity of at least a portion of one of the geometric 2D objects according to one or more third environmental metrics, the geometric 2D object being located on the portion of the first surface.

5. The processor:

2. The system of claim 1, configured to generate, according to a localization heuristic indicative of a type of physical object in the physical environment, one or more 3D objects that satisfy the localization heuristic at one or more corresponding locations in a second surface of the 3D model excluding the first surface.

6. The system of claim 5 , wherein the type of the physical object corresponds to at least one of a geography type, a climate type, or an architecture type.

7. The processor:

2. The system of claim 1, configured to generate one or more 3D objects at one or more corresponding locations in the 3D model that satisfy environmental heuristics indicative of atmospheric conditions indicative of weather in the physical environment according to the environmental heuristics.

8. The processor:

2. The system of claim 1, configured to segment an area model into a plurality of area segments according to a block heuristic that indicates an amount of computational resources, the area model representing a physical area that includes the physical environment, each of the area segments corresponding to a respective portion of the area model, and the 3D model corresponding to one of the area segments.

9. The processor: executing a first subset of the region segments with a first computational resource; The system of claim 8 , configured to execute, by a second computational resource, a second subset of the region segments concurrently with execution by the first computational resource.

10. searching, with a processor, a camera feed of the ego object navigating within a physical environment; generating, by the processor, a three-dimensional (3D) model including a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics based on the camera feed, the one or more first environmental metrics indicating boundaries of the one or more physical paths; generating, by the processor, one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, the second environmental metrics indicative of the one or more physical paths; identifying, by the processor, one or more viewpoints that are oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicative of cameras of physical objects configured to move along the one or more physical paths; rendering, by the processor, one or more simulated environment 2D images from one or more corresponding portions of the 3D model of the physical environment, each corresponding to one of the viewpoints; training, by the processor, an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images; A method comprising:

11. 11. The method of claim 10, further comprising modifying a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicating a condition of the physical path.

12. The method of claim 10 , further comprising modifying a topology of the portion of the first surface according to one or more third environmental metrics.

13. 11. The method of claim 10, further comprising modifying an opacity of at least a portion of one of the geometric 2D objects, the geometric 2D object being positioned on the portion of the first surface, according to one or more third environmental metrics.

14. 11. The method of claim 10, further comprising generating, according to a localization heuristic indicative of a type of physical object in the physical environment, one or more 3D objects that satisfy the localization heuristic at one or more corresponding locations in a second surface of the 3D model excluding the first surface.

15. The method of claim 14 , wherein the type of the physical object corresponds to at least one of a geography type, a climate type, or an architecture type.

16. 11. The method of claim 10, further comprising generating, according to an environmental heuristic indicative of atmospheric conditions indicative of weather in the physical environment, one or more 3D objects that satisfy the environmental heuristic at one or more corresponding locations in the 3D model.

17. 11. The method of claim 10, further comprising: segmenting an area model into a plurality of area segments according to a block heuristic that indicates an amount of computational resources, wherein the area model represents a physical area that includes the physical environment, each of the area segments corresponds to a respective portion of the area model, and the 3D model corresponds to one of the area segments.

18. executing, with a first computational resource, a first subset of the region segments; executing, by a second computational resource, a second subset of the region segments concurrently with execution by the first computational resource; 20. The method of claim 17, further comprising:

19. a processor searching a camera feed of the ego object navigating within the physical environment; generating, by the processor, a three-dimensional (3D) model including a first surface corresponding to one or more physical paths through the physical environment according to one or more first environmental metrics based on the camera feed, the one or more first environmental metrics indicating boundaries of the one or more physical paths; generating, by the processor, one or more geometric two-dimensional (2D) objects on the first surface according to one or more second environmental metrics, the second environmental metrics indicative of the one or more physical paths; identifying, by the processor, one or more viewpoints that are oriented to capture corresponding portions of the 3D model according to one or more viewpoint metrics indicative of cameras of physical objects configured to move along the one or more physical paths; rendering, by the processor, one or more simulated environment 2D images from one or more corresponding portions of the 3D model of the physical environment, each corresponding to one of the viewpoints; training, by the processor, an artificial intelligence model according to the camera feed and the one or more simulated environment 2D images; A non-transitory computer-readable medium comprising one or more instructions stored thereon and executable by a processor.

20. 20. The computer-readable medium of claim 19, further comprising one or more instructions executable by the processor to modify a portion of the first surface corresponding to a portion of one of the physical paths according to one or more third environmental metrics, the one or more third environmental metrics indicative of a condition of the physical path.

Images