Map creation and localization for autonomous driving applications

Abstract

An end-to-end system for data generation, creating maps using the generated data, and localizing to the created maps is disclosed. A map stream - or a stream of sensor data, perception outputs from a deep neural network (DNN), and / or relative trajectory data - can be generated corresponding to any number of drives of any number of vehicles and uploaded to the cloud. The map stream can be used to generate map data (and eventually a high definition (HD) map that is fused) representing the data generated over multiple drives. When localizing to the fused HD map, individual localization results can be generated based on a comparison of real-time data from a sensor modality to map data corresponding to the same sensor modality. This process can be repeated for any number of sensor modalities, and the results can be fused together to determine a final fused localization result.

Description

BACKGROUND

[0001] Mapping and localization are important processes for autonomous driving functionality. High definition (HD) maps, sensor perception, or a combination thereof are often used to localize a vehicle with respect to the HD map in order to make planning and control decisions. Typically, a regular HD map is generated using a survey vehicle equipped with advanced, high-precision sensors. However, these sensors are too expensive to implement on a consumer-level vehicle. In a typical deployment, a survey vehicle is able to generate a HD map suitable for localization after a single drive. Unfortunately, due to the scarcity of survey vehicles with such high-cost sensors, the availability of HD maps for any particular region can be concentrated around metropolitan or hub areas where the survey vehicles operate. For more remote areas, less-traveled areas, and areas a large distance from these active hubs, data for generating a HD map can have been collected from as few as a single drive (i.e., if data is completely available). Thus, in cases where a single drive results in lower quality or less suitable sensor data for this purpose (e.g., due to occlusions, dynamic objects, adverse weather effects, debris, construction artifacts, temporary hardware malfunctions, and / or other issues that can compromise the quality of collected sensor data), the HD map generated from the sensor data can not be safe or reliable for localization.

[0002] To remedy these quality issues, another survey vehicle can be needed to perform another drive at the location of the compromised quality. However, the identification, deployment, sensor data generation, and map update process can take a long period of time due to the system data collection and map creation process, such that the HD map is not available until the update is made. This problem is exacerbated in cases where road conditions or layouts change frequently or drastically over time (e.g., due to construction), as there can be no mechanism for identifying these changes, and even if identified, there is no way to generate updated data without deploying another survey vehicle.

[0003] Further, because consumer vehicles can not be equipped with the same high-quality, high-cost sensors, localization to an HD map (even when available) cannot be performed using many sensor modalities (e.g., cameras, LiDAR, RADAR, etc.) because the quality and type of data can not align with that used to generate the HD map. Thus, localization relies only on Global Navigation Satellite System (GNSS) data, which, even for the most expensive and precise sensor models, still achieves accuracy only within a few meters and / or has variable accuracy in certain situations. Inaccuracies of a few meters can place a vehicle in a different lane than the current lane of travel, or on a side of the road other than the current lane of travel. As such, generating an HD map using conventional solutions can result in an inaccurate map that, when synthesized due to inaccurate localization, presents a significant obstacle to achieving safe and reliable highly autonomous vehicle (e.g., Level 3, Level 4, and Level 5) autonomous vehicles. SUMMARY

[0004] Embodiments of the present disclosure relate to methods for map creation and localization for autonomous driving applications. Specifically, embodiments of the present disclosure include an end-to-end system for data generation, map creation using the generated data, and localization to the created map that can be used with generic consumer-level sensors in commercial vehicles. For example, during the data generation process, data collection vehicles and / or consumer vehicles employing consumer-quality sensors can be used to generate sensor data. The resulting data can correspond to a map stream - which can include a stream of sensor data, perception outputs from a deep neural network (DNN), and / or relative trajectory (e.g., rotation and translation) data - corresponding to any number of drives of any number of vehicles. As such, in contrast to the system data collection efforts of conventional systems, the present system can use crowd-sourced data generation using many vehicles and many drives. To reduce bandwidth and memory requirements of the system, data from the map stream can be minimized (e.g., by filtering out dynamic objects, performing LiDAR plane slicing or LiDAR point reduction, converting perception or camera-based outputs to 3D location information, performing activity only for certain data types, etc.) and / or compressed (e.g., using delta compression techniques). As a result of generating map stream data using consumer-level sensors, the sensor data - once converted to map form for localization - can be used directly for localization, rather than relying only on GNSS data. Further, because relative trajectory information corresponding to each drive is tracked, this information can be used to generate individual road segments (e.g., 25 meter, 50 meter, etc. sized road segments) that are localizable, allowing for localization accuracy in the centimeter range.

[0005] During map creation, map streams can be used to generate map data - and ultimately a fused HD map - that represents data generated by multiple drives. Moreover, as new map streams are generated, these additional drives can be merged, combined, or integrated with existing map stream data and used to further increase the robustness of the HD map. For example, each map stream can be converted into a respective map, and any number of drive segments from any number of maps (or respective map streams) can be used to generate a fused HD map representation for a particular drive segment. Drive segments can be geometrically registered with respect to one another to determine pose links representing rotation and translation between poses (or frames) of the drive pairs. Frame graphs representing the pose links can be divided into road segments - e.g., road segments for relative localization - and the poses corresponding to each road segment can undergo optimization. The resulting poses within each segment can be used to fuse individual sensor data and / or perception outputs for generating a final fused HD map. As a result, and since the map data corresponds to consumer quality sensors, sensor data and / or perception results (e.g., landmark positions) from the HD map can be directly used for localization (e.g., by comparing current real-time sensor data and / or perception to corresponding map information) in addition to using GNSS data in embodiments.

[0006] For example, when localizing to a fused HD map, individual localization results can be generated based on a comparison of sensor data and / or perception outputs from a sensor modality to map data corresponding to the same sensor modality. For example, a cost space can be sampled at each frame using data corresponding to a sensor modality, a plurality of individual cost spaces can be used to generate an aggregate cost space, and a filter (e.g., using a Kalman filter) can be used to ultimately determine a localization result for a particular sensor modality. This process can be repeated for any number of sensor modalities (e.g., LiDAR, RADAR, camera, etc.), and the results can be fused together to determine a final fused localization result for a current frame. The fused localization result can then be propagated to a next frame and used to determine a fused localization for the next frame, and so on. Since the HD map includes individual road segments for localization, and each road segment has a respective global position, global localization results can also be achieved as the vehicle localizes to a local or relative coordinate system corresponding to a road segment. BRIEF DESCRIPTION OF DRAWINGS

[0007] The present systems and methods for map creation and localization for autonomous driving applications are described in detail below with reference to the accompanying drawings, wherein:

[0008] FIG. 1 depicts a data flow diagram of a process of a map creation and localization system in accordance with some embodiments of the present disclosure;

[0009] FIG. 2A data flow diagram depicting a process for map stream generation according to some embodiments of the disclosure;

[0010] FIG. 3 A flow diagram illustrating a method for map stream generation according to some embodiments of the disclosure;

[0011] FIG. 4 A data flow diagram depicting a process for map creation according to some embodiments of the disclosure;

[0012] FIGS. 5A-5B A data flow diagram depicting a process for converting a map stream into a map for registration according to some embodiments of the disclosure;

[0013] FIG. 5C A schematic diagram depicting determination of segments and pairwise connections between segments for registration according to some embodiments of the disclosure;

[0014] FIG. 5D A table depicting the number of pairwise connections for registration of different numbers of segments using different methods according to some embodiments of the disclosure;

[0015] FIGS. 5E-5F An example visualization of registering two base layers of a picture segment using a front-facing camera and a rear-facing camera according to some embodiments of the disclosure;

[0016] FIG. 6A An example frame graph generated after registration of segments of multiple drives according to some embodiments of the disclosure;

[0017] FIG. 6B A road segment generation using a frame graph according to some embodiments of the disclosure; FIG. 6A

[0018] A segment graph corresponding to determined road segments of a frame graph according to some embodiments of the disclosure; FIG. 6C FIG. 6B An example pose link error calculation for poses of road segments according to some embodiments of the disclosure;

[0019] FIG. 6D An example pose graph and corresponding optimization algorithm according to some embodiments of the disclosure;

[0020] FIGS. 6E-6G A finalized road segment determined from a pose graph according to some embodiments of the disclosure;

[0021] FIG. 6H FIG. 6G A finalized road segment determined from a pose graph according to some embodiments of the disclosure;

[0022] ​​FIG. 6I depicts a transformation between origins of road segments of a segment graph according to some embodiments of the disclosure;

[0023] FIG. 7 is a flowchart illustrating a method for map creation according to some embodiments of the disclosure;

[0024] FIG. 8A depicts a dataflow diagram of a localization process according to some embodiments of the disclosure;

[0025] FIG. 8B depicts relationships between a vehicle origin, a road segment origin, and a global origin for localization according to some embodiments of the disclosure;

[0026] FIG. 9A depicts a sample cost space corresponding to a frame generated during localization according to some embodiments of the disclosure;

[0027] FIG. 9B depicts an aggregate cost space corresponding to sample cost spaces of multiple frames generated during localization according to some embodiments of the disclosure;

[0028] FIG. 9C depicts a filtered aggregate cost space generated during localization according to some embodiments of the disclosure;

[0029] FIGS. 10A-10C depicts an example visualization of sampling poses of a sample cost space for camera-based localization according to some embodiments of the disclosure;

[0030] FIGS. 11A-11B depicts an example visualization of sampling poses of a sample cost space for RADAR-based localization according to some embodiments of the disclosure;

[0031] FIGS. 12A-12D depicts an example visualization of sampling poses of a sample cost space for LiDAR-based localization according to some embodiments of the disclosure;

[0032] FIG. 13 depicts an example of fusing multiple localization results to generate a final localization result according to some embodiments of the disclosure;

[0033] FIG. 14 is a flowchart illustrating a method for localization according to some embodiments of the disclosure;

[0034] FIG. 15A is an illustration of an example autonomous vehicle according to some embodiments of the disclosure;

[0035] FIG. 15Bis a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 15A

[0036] FIG. 15C is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 15A

[0037] FIG. 15D is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure FIG. 15A

[0038] FIG. 16 is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure

[0039] FIG. 17 is a block diagram of an example system architecture of an example autonomous vehicle in accordance with some embodiments of the present disclosure DETAILED DESCRIPTION

[0040] Systems and methods related to map creation and localization for autonomous driving applications are disclosed. While the present disclosure can be described with respect to an example autonomous vehicle 1500 (or, in this document, referred to as“vehicle 1500” or“ego vehicle 1500,” described herein with respect to FIGS. 15A-15D A block diagram of an example system architecture of an example autonomous vehicle is described. While the present disclosure can be described with respect to an example autonomous vehicle 1500 (or, in this document, referred to as“vehicle 1500” or“ego vehicle 1500,” described herein with respect to

[0041] Reference is made to FIG. 1 , FIG. 1 ​​​A data flow diagram depicting a process 100 of a map creation and localization system in accordance with some embodiments of the present disclosure is described. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be wholly omitted or consolidated. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities can be carried out by hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in memory.

[0042] In some embodiments, the vehicle 1500 can include similar components, features, and / or functionality of the vehicle 1500 described herein with respect to FIGS. 15A-15D In addition, in embodiments, the map creation 106 can be performed in one or more data centers, and can be performed using similar components, features, and / or functionality as described herein with respect to the example computing device 1600 and / or the example data center 1700. In some embodiments, the entire end-to-end process 100 can be performed within a single vehicle 1500. While only a single vehicle 1500 is shown in FIG. 1 FIG. 1 is not intended to be limiting. For example, any number of vehicles 1500 can be used to generate the sensor data 102 for the map stream generation 104, and any number of (different) vehicles 1500 can be used to generate the sensor data 102 for the localization 110. In addition, for each vehicle 1500 used in the map stream generation 104, the map creation 106, and / or the localization 110 processes, the vehicle make, model, year, and / or type can be the same, similar, and / or different, in addition to the sensor configuration and / or other vehicle attributes.

[0043] ​The process 100 can include operations for map stream generation 104, map creation 106, and localization 110. For example, the process 100 can be executed as part of an end-to-end system that relies on map streams generated using sensor data 102 from any of a plurality of vehicles 1500 on the road, map creation 106 using map stream data received from the vehicles 1500, and localization 110 to maps generated using the map creation 106 process (e.g., high definition (HD) maps). In some non-limiting embodiments, the maps represented by the map data 108 can include maps for or generated by data collected from different sensor modalities or individual sensors for individual sensor modalities. For example, the map data 108 can represent a first map (or map layer) corresponding to image-based localization, a second map (or map layer) corresponding to LiDAR-based localization, a third map (or map layer) corresponding to RADAR-based localization, and so on. In some embodiments, image-based localization can be performed using, for example, a first map (or map layer) corresponding to a forward-facing camera and a second map (which can or can not correspond to the same map layer) corresponding to a rear-facing camera. Thus, depending on the sensor configuration of the vehicle 1500 receiving the map data 108 and performing localization 110 thereto, only the necessary portions of the map data 108 are transmitted to the vehicle 1500. As a non-limiting example, where a first vehicle 1500 includes one or more cameras and one or more RADAR sensors but not one or more LiDAR sensors, the camera map layer and the RADAR map layer can be transmitted to the vehicle 1500 and the LiDAR map layer can not be transmitted to the vehicle 1500. Thus, memory usage on the vehicle 1500 is reduced and bandwidth is conserved as the LiDAR map layer can not be transmitted and / or stored on the vehicle 1500.

[0044] Further, as the vehicles 1500 navigate the environment using the map data 108 for localization 110, a sanity check 112 can be performed to ensure that the one or more maps are up-to-date or accurate given changing road conditions, road structures, buildings, and so on. As such, when a portion or segment of a map is determined to be below a desired quality - e.g., due to difficulties in localizing to the map - the map stream generation 104 can be performed and used to update the one or more maps corresponding to the particular portion of the segment via the map creation 106. Thus, the end-to-end system can not only be used to generate maps for localization 110, but also ensure that the maps remain up-to-date over time for accurate localization. Each of the map stream generation 104, map creation 106, and localization 116 processes are described in greater detail herein.

[0045] Map Stream Generation

[0046] To generate map streams, any number of vehicles 1500 (e.g., consumer vehicles, data collection vehicles, combinations thereof) can perform any number of drives. For example, each vehicle 1500 can drive through various road segments from locations around a town, city, state, country, continent, and / or world, and can generate sensor data 102 during the drives using any number of sensors (e.g., LiDAR sensors 1564, RADAR sensors 1560, cameras 1568, 1570, 1572, 1574, 1598, etc., inertial measurement unit (IMU) sensors 1566, ultrasonic sensors 1562, microphones 1596, speed sensors 1544, steering sensors 1540, global navigation satellite system (GNSS) sensors 1558, etc.). Each individual vehicle 1500 can generate sensor data 102 that can be used for map stream generation 104 corresponding to the particular drive of the vehicle 1500 using the sensor data 102. Map streams generated to correspond to different drives from a single vehicle 1500 and drives from any number of other vehicles 1500 can be used for map creation 106, as described in greater detail herein.

[0047] Because multiple map streams are used to generate map data 108 for any particular road segment, individual map streams from each drive need not be as high precision or high fidelity as in conventional systems. For example, conventional systems use survey vehicles equipped with sensor types that are very expensive and thus undesirable for installation in consumer vehicles (e.g., because the cost of the vehicle would increase substantially). However, the sensors on these survey vehicles can generate sensor data that can be reliable enough even after a single drive. However, disadvantageously, in cases where a particular drive includes a large amount of dynamic or transient factors (such as, but not limited to, traffic, construction artifacts, debris, occlusions, adverse weather effects, transient hardware faults, or other sources of sensor data quality issues), a single drive can not produce data suitable for generating an accurate map for localization. Further, as road conditions change, and because of the small number of available survey vehicles, maps can not be updated quickly - e.g., a map is not updated until another survey vehicle traverses the same route. In contrast, with the system of the present disclosure, by utilizing consumer vehicles with lower cost mass-market sensors, any number of map streams from any number of drives can be used to generate map data 108 more quickly and more frequently. Thus, individual map streams from drives that have occlusions or other quality issues can be relied on to a lesser extent, and map streams from higher quality sensor data can be relied on more heavily. Further, as road structures, layouts, conditions, surroundings, and / or other information change, normal checks 112 can be performed to update map data 108 more quickly (e.g., in real-time or substantially real-time). The result of this process is a more crowd-sourced approach to map stream generation, rather than the system data collection efforts of conventional approaches.

[0048] Referring now to FIG. 2 , FIG. 2 A data flow diagram of a process 104 for map stream generation according to some embodiments of the present disclosure is depicted. For example, the process 104 can correspond to generating a map stream from a single drive by a vehicle 1500. This process 104 can be repeated by any number of vehicles 1500 on any number of drives. As described herein, sensor data 102 can correspond to sensor data 102 from any number of different sensor modalities and / or any number of sensors of a single modality. For example, sensor data 102 can correspond to sensor data 102 described herein with respect to FIG. 1, and / or any other sensor data 102 described herein. The process 104 can include a number of steps, which are described in more detail herein. FIGS. 15A-15Dany of the sensor types described with respect to vehicle 1500, such as one or more GNSS sensors 1558 (e.g., one or more global positioning system sensors), one or more RADAR sensors 1560, one or more ultrasonic sensors 1562, one or more LiDAR sensors 1564, ultrasonic sensors, one or more IMU sensors 1566 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more microphones 1576, one or more stereo cameras 1568, one or more wide-angle cameras 1570 (e.g., fisheye cameras), one or more infrared cameras 1572, one or more surround cameras 1574 (e.g., 360-degree cameras), one or more long-range and / or mid-range cameras 1598, a speed sensor 1544 (e.g., to measure a speed of vehicle 1500), and / or other sensor types. In some embodiments, sensor data 102 can be included directly in map stream 210 - e.g., with or without compression using data compressor 208. For example, for LiDAR data and / or RADAR data, detections represented by sensor data 102 can be in a three-dimensional (3D) coordinate space (e.g., world space), and LiDAR points and / or RADAR points (or detections) can be used directly to generate LiDAR layers and / or RADAR layers, respectively. In some embodiments, sensor data 102 can be converted from a two-dimensional (2D) coordinate space (e.g., image space) to a 3D coordinate space, e.g., using data converter 206, and then included in map stream 210 (e.g., after compression using data compressor 208, in embodiments).

[0049] In some embodiments, LiDAR slicing can be performed on LiDAR data (e.g., on point clouds generated using raw LiDAR data) to slice the LiDAR data into different height ranges. The LiDAR data can be sliced into any number of height ranges. In some embodiments, the LiDAR height ranges can be defined relative to an origin or equipment of the vehicle 1500. As non-limiting examples, the LiDAR data can be sliced into an above ground slice (e.g., from 5 meters to 300 meters relative to the origin of the vehicle 1500), a giraffe plane slice (e.g., from 2.5 meters to 5 meters relative to the origin of the vehicle 1500), and / or a ground plane slice (e.g., from -2.5 meters to 0.5 meters relative to the origin of the vehicle 1500). Where LiDAR slicing is performed, the different slices can be stored as separate LiDAR layers in the map stream 210 and / or can be used to generate separate LiDAR map layers for localization 110 during map creation 106. In some embodiments, data corresponding to certain slices can be filtered out or removed so that less data is encoded to the map stream 210 and less data is transmitted to the cloud for map creation 106. For example, the above ground slice can not be as valuable as the giraffe plane slice or the ground plane slice because detections far from the ground plane can not be as useful, accurate, and / or precise enough for localization. In this example, the above ground slice (e.g., from 5 meters to 300 meters) can be filtered out.

[0050] Sensor data 102 can include, for example, data generated by one or more IMU sensors 1566, GNSS sensors 1558, one or more speed sensors 1544, camera sensors, LiDAR sensors 1564, and / or other sensor types, which can be used to track an absolute position of vehicle 1500 and / or a local or relative position of vehicle 1500. For example, at each frame of sensor data 102 generated from any number of sensor modalities, a global position for the frame of sensor data 102 can be recorded - e.g., using GNSS sensors 1558. This global position (e.g., in a WGS84 reference frame) can be used to place vehicle 1500 generally within a global coordinate system. However, when using an HD map for autonomous driving operations, a global scale of localization accuracy is not as valuable as a localization accuracy on a given road segment. For example, when driving on Highway 101 in San Clara, California, the position of vehicle 1500 relative to Interstate 95 in Boston is not as critical as the position of the vehicle 100 meters, 1 mile, etc. ahead on Highway 101. Moreover, GNSS sensors (even of the highest quality) can not be accurate to more than five or ten meters, and thus, global localization can be less accurate and / or can have variable accuracy affected by satellite relative positioning. This can still be the case - e.g., global localization can be off by five or ten meters - but relative localization to a current road segment can be accurate to within five or ten centimeters. When autonomous driving, to ensure safety, localization to a relative local layout or road segment thereof provides greater accuracy and precision than a global-only approach. Thus, as described in greater detail herein, when performing localization, systems of the present disclosure can use GNSS coordinates to determine which road segment or segments vehicle 1500 is currently driving on, and then can use a local or relative coordinate system of the determined road segment or segments to more precisely localize vehicle 1500 (e.g., in embodiments, without requiring impractical high cost GNSS sensor types to be implemented on consumer vehicles). Thus, once vehicle 1500 is localized to a given road segment, GNSS coordinates can not be needed for accurate and precise localization, as vehicle 1500 can localize itself from road segment to road segment as vehicle 1500 drives. In some non-limiting embodiments, as described herein, each road segment can be 25 meters, 50 meters, 80 meters, 100 meters, and / or another distance.

[0051] To generate data for map stream 210 that can be used to generate an HD map that can be used to generate accurate and precise local or relative positioning, sensor data 102 can be used to track movement (e.g., rotation and translation) of vehicle 1500 at each frame or time step using IMU sensors 1566, one or more GNSS sensors 1558, one or more speed sensors 1544, one or more wheel sensors (e.g., counting wheel ticks of vehicle 1500), one or more perception sensors (e.g., cameras, LiDAR, RADAR, etc.), and / or other sensor types. The trajectory or ego motion of vehicle 1500 can be used to generate a trajectory layer of map stream 210. For a non-limiting example, the movement of one or more perception sensors can be tracked to determine the corresponding movement of vehicle 1500 - e.g., referred to as a visual odometry. One or more IMU sensors 1566 can be used to track the rotation or pose of vehicle 1500, and one or more speed sensors 1544 and / or one or more wheel sensors can be used to track the distance traveled by vehicle 1500. As such, at a first frame, a first pose (e.g., angles along x, y, and z axes) and a first position (e.g., (x, y, z)) of an origin of the rig or vehicle 1500 can be determined using sensor data 102. At a second frame, a second pose and a second position (e.g., relative to the first position) of the origin of the rig or vehicle 1500 can be determined using sensor data, etc. Thus, a trajectory can be generated with points (corresponding to frames), where each point can encode information corresponding to the relative position of vehicle 1500 relative to a previous point. Moreover, sensor data 102 and / or outputs 204 of one or more DNNs 202 captured at each of these points or frames can be associated with these points or frames. As such, when creating an HD map (e.g., map data 108) of the environment, sensor data 102 and / or outputs 204 of one or more DNNs 202 can have a known position relative to the origin of vehicle 1500 or the rig, and since the origin of vehicle 1500 or the rig can have a corresponding position on a global coordinate system, sensor data 102 and / or outputs 204 of one or more DNNs 202 can also have a position on the global coordinate system. FIG. 1

[0052] ​By using the relative motion of the vehicle 1500 from frame to frame, accuracy can be maintained even in tunnels, urban and / or other environments where GNSS signals can be weak or lost. However, even with the use of relative motion and added vectors (e.g., representing translation and rotation between frames) for each frame or time step, the relative position can drift after some time or distance traveled. Thus, at predefined intervals, when drift is detected, and / or based on some other criteria, the relative motion can be reset or recalibrated. For example, there can be anchor points in the global coordinate system that can have known positions, and the anchor points can be used to recalibrate or reset the relative motion at a frame.

[0053] In some embodiments, the sensor data 102 is applied to one or more deep neural networks (DNNs) 202 trained to compute various different outputs 204. Prior to application or input to the one or more DNNs 202, the sensor data 102 can undergo pre-processing, such as converting, cropping, upsampling, downsampling, scaling, rotating, and / or otherwise modifying the sensor data 102. For example, where the sensor data 102 corresponds to camera image data, the image data can be cropped, reduced, scaled, flipped, rotated, and / or otherwise adjusted to a suitable input format for the respective one or more DNNs 202. In some embodiments, the sensor data 102 can include image data representing one or more images, image data representing a video (e.g., a snapshot of a video), and / or sensor data representing a representation of a sensor’s sensory field (e.g., a depth map for a LIDAR sensor, a value map for an ultrasonic sensor, etc.). For example, any type of image data format can be used, such as but not limited to, compressed images (e.g., Joint Photographic Experts Group (JPEG) or luminance / chrominance (YUV) formats), compressed images (e.g., frames derived from a compressed video format (e.g., H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC))), raw images (e.g., derived from a red clear blue (RCCB), red clear clear (RCCC), or other type of imaging sensor), and / or other formats. Moreover, in some examples, the sensor data 102 can be used without any pre-processing (e.g., in a raw or captured format), while in other examples, the sensor data 102 can undergo pre-processing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data pre-processor (not shown)).

[0054] In the case where the sensor data 102 corresponds to LiDAR data, for example, the raw LiDAR data can be accumulated, self-motion compensated, and / or otherwise adjusted, and / or the raw LiDAR data can be converted to another representation, such as a 3D point cloud representation (e.g., from an overhead view, sensor perspective, etc.), a 2D projected image representation (e.g., a LiDAR range image), and / or another representation. Similarly, for RADAR and / or other sensor modalities, the sensor data 102 can be converted to a suitable representation for input to the respective one or more DNNs 202. In some embodiments, the one or more DNNs 202 can process two or more different sensor data inputs from any number of sensor modalities to generate the output 204. As such, as used herein, the sensor data 102 can refer to unprocessed sensor data, pre-processed sensor data, or a combination thereof.

[0055] While examples are described herein with respect to the use of one or more DNNs 202, this is not intended to be limiting. For example, and without limitation, the one or more DNNs 202 can include one or more machine learning models of any type, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), Naive Bayes, k-nearest neighbors algorithm (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptrons, long / short term memory / LSTMs, Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid machines, etc.), interest region detection algorithms, computer vision algorithms, and / or other types of algorithms or machine learning models.

[0056] As an example, the DNNs 202 can process the sensor data 102 to generate detections of lane markings, road boundaries, signs, poles, trees, static objects, vehicles and / or other dynamic objects, wait conditions, intersections, distances, depths, sizes of objects, etc. The detections can correspond to locations (e.g., in 2D image space, in 3D space, etc.), geometric shapes, poses, semantic information, and / or other information about the detections, for example. As such, for lane lines, the locations of the lane lines and / or the types of the lane lines (e.g., dashed, solid, yellow, white, crosswalk, bike lane, etc.) can be detected by one or more DNNs 202 processing the sensor data 102. As for signs, the locations or other wait condition information of the signs and / or their types (e.g., yield, stop, crosswalk, traffic light, yield light, construction, speed limit, exit, etc.) can be detected using one or more DNNs 202. For detected vehicles, motorcyclists, and / or other dynamic actors or road users, the locations and / or types of the dynamic actors can be identified and / or tracked, and / or can be used to determine wait conditions in a scene (e.g., where a vehicle behaves in a certain way with respect to an intersection, like by coming to a stop, the intersection or wait condition corresponding thereto can be detected as an intersection with a stop sign or traffic light).

[0057] In embodiments, the outputs 204 of the one or more DNNs 202 can undergo post-processing, such as by converting raw outputs into useful outputs— e.g., where the raw outputs correspond to points or pixels corresponding to a confidence of each point (e.g., in LiDAR, RADAR, etc.) or pixel (e.g., for camera images) of a particular object type, post-processing can be performed to determine each of the points or pixels that correspond to a single instance of the object type. This post-processing can include temporal filtering, weighting, outlier removal (e.g., removing pixels or points that are determined to be outliers), upsizing (e.g., the outputs can be predicted at a lower resolution than the input sensor data instances, and the outputs can be upsized back to the input resolution), downsizing, curve fitting, and / or other post-processing techniques. In implementations, after post-processing, the outputs 204 can be in a 2D coordinate space (e.g., image space, LiDAR range image space, etc.) and / or can be in a 3D coordinate system. In embodiments where the outputs are in a 2D coordinate space and / or in a 3D coordinate space other than 3D world space, the data converter 206 can convert the outputs 204 to 3D world space.

[0058] In some non-limiting examples, one or more DNNs 202 and / or outputs 204 can be similar to the DNNs and / or outputs described in U.S. Non-Provisional Patent Application No. 16 / 286,329, filed February 26, 2019, U.S. Non-Provisional Patent Application No. 16 / 355,328, filed March 15, 2019, U.S. Non-Provisional Patent Application No. 16 / 356,439, filed March 18, 2019, U.S. Non-Provisional Patent Application No. 16 / 385,921, filed April 16, 2019, U.S. Non-Provisional Patent Application No. 16 / 728,595, filed December 27, 2019, U.S. Non-Provisional Patent Application No. 16 / 728,598, filed December 27, 2019, U.S. Non-Provisional Patent Application No. 16 / 813,306, filed March 9, 2020, U.S. Non-Provisional Patent Application No. 16 / 848,102, filed April 14, 2020, U.S. Non-Provisional Patent Application No. 16 / 814,351, filed March 10, 2020, U.S. Non-Provisional Patent Application No. 16 / 911,007, filed June 24, 2020, and / or U.S. Non-Provisional Patent Application No. 16 / 514,230, filed July 17, 2019, the entire contents of each of which are incorporated herein by reference.

[0059] In embodiments, data converter 206 can convert all outputs 204 and / or sensor data 102 to a 3D world space coordinate system that is originated at the equipment of vehicle 1500 (e.g., (0, 0, 0)). The origin of vehicle 1500 can be the most forward or rearward point on vehicle 1500 along the wheel axis of vehicle 1500, and / or at any location on or relative to vehicle 1500. In some non-limiting embodiments, the origin can correspond to the center of the rear axle of vehicle 1500. For example, at a given frame or time step, sensor data 102 can be generated. As a non-limiting example, a first subset of sensor data 102 can be generated in 3D world space relative to the origin, and can be used directly (e.g., after compression by data compressor 208) to generate map stream 210. A second subset of sensor data 102 can be generated in 3D world space, but not relative to the origin of vehicle 1500. Accordingly, data converter 206 can convert sensor data 102 (e.g., using intrinsic and / or extrinsic parameters of the respective one or more sensors) such that the 3D world space position of sensor data 102 is relative to the origin of vehicle 1500. A third subset of sensor data 102 can be generated in 2D space. Data converter 206 can convert this sensor data 102, e.g., using intrinsic and / or extrinsic parameters of the respective one or more sensors, such that the 2D space position of sensor data 102 is in 3D space and relative to the origin of vehicle 1500.

[0060] In embodiments implementing one or more DNNs 202, outputs 204 (e.g., before or after post-processing) can be generated in 2D space and / or 3D space (relative to an origin or not). Similar to the description herein regarding converting positions from sensor data 102 directly to 3D world space relative to the origin of vehicle 1500, 2D and / or 3D outputs 204 (not relative to an origin) can be converted by data converter 206 to 3D space relative to the origin of vehicle 1500. As such, and as described herein, because the origin of vehicle 1500 has a known relative position relative to the current road segment of the road, or within the sequence of map stream frames, and the current road segment has a relative position in a global coordinate system (e.g., WGS84 reference frame), the positions of sensor data 102 and / or outputs 204 from one or more DNNs 202 can also have a relative position relative to the current road segment and the global coordinate system.

[0061] With respect to detected road boundaries and lane lines (e.g., detected using one or more DNNs 202 processing sensor data 102), a landmark filter can be executed to stitch and / or smooth the detected road boundary lines and / or lane lines. For example, 3D positions of lane lines and / or road boundaries can include gaps in the detection, can include noise, and / or can not be as accurate, precise, or artifact-free as optimal or desired. Accordingly, landmark filtering can be performed to stitch together the detections within a frame and / or across frames to generate virtual continuous lane dividers and road boundary lines. These continuous lane dividers and / or road boundary lines can be similar to a lane graph used to define a plurality of lanes, positions of lanes, and / or positions of road boundaries on a travel surface. In some embodiments, smoothing can be performed on the generated continuous lines to more accurately reflect known geometric information of lane lines and road boundaries. For example, in cases where detection of lane lines for a frame is skewed relative to previous and / or subsequent detections, the skewed portions of the lane lines can be smoothed to more precisely conform to known patterns of lane lines. As such, in addition to or as an alternative to encoding each detection into the map stream 210, the encoded information in the map stream 210 can correspond to these continuous lane lines and / or road boundaries.

[0062] In some embodiments, sensor data 102 and outputs 204 can be generated at all times and for every frame, and all data can be transmitted to a map creation cloud or server as a map stream 210. However, in some embodiments, sensor data 102 and / or outputs 204 can not be generated at every frame, all data can not be transmitted in map stream 210, or a combination thereof. For example, map stream campaigns can be implemented that identify and direct what types and amounts of data to collect, where to collect the data, how often to collect the data, and / or other information. These campaigns can allow certain types and / or at certain locations to be generated purposefully or selectively in order to fill in gaps, provide additional data to improve accuracy, update the map when changes to the road are detected (e.g., via normal checks 112), and / or for other reasons. As an example, a map stream campaign can be executed that identifies a vehicle 1500 at a particular location, and instructs the vehicle 1500 to generate (or prioritize generation of) certain data types - e.g., LiDAR data and RADAR data starting at the location and over a certain distance - in order to reduce computation (e.g., by the vehicle 1500 when generating map stream 210 and during map creation 106 when processing map stream data by the vehicle 1500) and bandwidth (e.g., for transmitting map stream 210 to the cloud). In such an example, some amount of driving through one or more particular sections of the road can have encountered a lot of occlusions, or the vehicle 1500 performing the driving is not equipped with certain sensor modalities. As such, the map stream campaign can instruct the vehicle 1500 to collect data corresponding to data that was previously occluded and / or to generate data for missing modalities. As another example, a map stream campaign can be generated to more accurately identify lane marks, signs, traffic lights, and / or other information, and instructions to the vehicle 1500 can be to generate sensor data 102 and / or outputs 204 that can be used to generate or update an HD map with this information. As such, one or more DNNs 202 that compute information about lane marks, signs, traffic lights, etc. can be executed using respective sensor data types, and outputs 204 (e.g., after post-processing, data conversion, compression, etc.) can be transmitted to the cloud via map stream 210 for map creation.

[0063] In some embodiments, the map stream activity can be part of a map sanity check 112 - e.g., after an HD map is generated and used for localization. For example, in the event that an inconsistency is detected between current sensor data or DNN detections and the HD map represented by map data 108, the sanity checker can trigger one or more vehicles 1500 to generate and / or upload new map stream data for that location. For example, in some embodiments, vehicles 1500 can generate data for map stream 210 and not upload map stream 210, while in other embodiments, vehicles 1500 can only generate and upload map stream data when triggered. As such, in the event that localization to a map results in poor planning and / or control operations, map stream 210 can be uploaded to the cloud to update the map information through the map creation process. Thus, the map can not be constantly updated, but rather only updated when localization errors, planning errors, and / or control errors are detected.

[0064] In some examples, the system can minimize the frequency at which trajectory points or frames are generated and / or included in map stream 210. For example, instead of including every frame in map stream 210, a distance threshold, a time threshold, or a combination thereof can be used to determine which frames to include in map stream 210. As such, if a vehicle 1500 has traveled a certain distance (e.g., half a meter, one meter, two meters, five meters, ten meters, etc.) and / or has passed a certain amount of time (e.g., half a second, one second, two seconds, etc.), a trajectory point or frame can be included in the map stream. This distance or time threshold can be used based on which is met first or which is met last. For example, a first frame can be included in map stream 210, then a distance threshold can be met and a second frame at the distance threshold can be included in map stream 210. Once the second frame is included, the distance and time thresholds can be reset, and then a time threshold can be met and a third frame can be included in map stream 210, and so on. Thus, less redundant data can be included in map stream 210.

[0065] For example, when the vehicle 1500 is stationary at a traffic light for 30 seconds, the time threshold is 1 second, and the frame rate is 30 frames per second (fps), instead of including 900 (e.g., 30*30) frames in the map stream 210, only 30 frames can be included in the map stream 210. In some embodiments, once the vehicle 1500 is idle for the threshold amount of time (e.g., two seconds, four seconds, etc.), the frame generation and / or frame inclusion in the map stream 210 can be paused (e.g., until movement is detected). As another non-limiting example, when the vehicle 1500 is traveling at a speed of one meter per second (or 2.24 miles per hour), the distance threshold is two meters, and the frame rate is 30 fps, instead of including 60 (e.g., 30*2) frames in the map stream 210 during a two meter distance, only a single frame can be included in the map stream 210. Thus, the amount of data transmitted from the vehicle 1500 to the cloud for map creation 106 is reduced without impacting the accuracy of the map creation process, e.g., because at least some of the data can be repetitive or only incrementally different, and thus unnecessary for accurate map creation.

[0066] In addition to or instead of sending less data in the map stream 210 (e.g., minimizing the amount of data), in embodiments data can be compressed - e.g., to reduce bandwidth and to reduce run-time. In some embodiments, extrapolation and / or interpolation can be used to determine points or frames such that fewer points or frames need to be sent in the map stream 210 (e.g., rotation / translation information for points or frames) and extrapolation and / or interpolation can be used to generate additional frames or points. For example, because rotation and translation information can be data intensive - e.g., a full 64 bits can be needed to fully encode (x, y, z) position information and x-axis, y-axis, and z-axis rotation information - less data needs to be sent because there are fewer points along the trajectory for which rotation and / or translation information needs to be encoded. As such, the history of the trajectory can be used to extrapolate future points in the trajectory. As another example, interpolation can be used to generate additional frames between frames. As an example, in the case where the trajectory corresponds to a vehicle 1500 traveling in a straight line at a substantially constant speed, then the first and last frames of the sequence can be included in the map stream 210 and the frames in between can be interpolated from the first and last frames of the sequence. However, in the case of a speed change, more frames can need to be encoded in the map stream 210 in order to more accurately linearly interpolate at a particular point in time where the vehicle 1500 is located. In such an example, the system can still not use all of the frames, but can use more frames than in the straight line driving constant speed example. In some embodiments, cubic interpolation or a cubic polynomial interpolation can be used to encode a derivative, which can be used to determine a rate of change of speed and thus to determine or interpolate other points along the trajectory without having to directly encode them into the map stream 210. As such, instead of encoding rotation, translation, position, and / or pose at each frame, encoded interpolation and / or extrapolation information can instead be used to recreate additional frames. In some embodiments, the frequency of trajectory data generation can be adaptively updated to achieve a maximum error in accuracy of the pose used in the map creation 106, within which the pose is interpolated between two other frames or time steps, as compared to the correct pose known at the time of data reduction.

[0067] As another example, delta compression can be used to encode a difference or delta between a current position (e.g., pose and position) of the vehicle 1500 and a previous position of the vehicle 1500. For example, 64 bits can be used to encode each of the latitude, longitude, and altitude or x, y, and z coordinates of the vehicle 1500 at a frame, for a total of 192 bits. However, as the vehicle 1500 moves, these values can not change much from frame to frame. Thus, instead of using a full 64 bits for each of the latitude, longitude, and altitude at each frame, the encoded values for a frame can instead correspond to a difference from a previous frame (which can be encoded in fewer bits). For example, in embodiments, 12 bits or fewer can be used to encode the difference, thereby producing a large memory and bandwidth savings for the system. Delta compression encoding can be used for local layout relative coordinates and / or global coordinates.

[0068] In addition to or instead of compressing position, pose, rotation, and / or translation information, sensor data 102 and / or output 204 can be compressed for inclusion in map stream 210. For example, for position, pose, size, and / or other information about lanes, lane markings, lane dividers, signs, traffic lights, wait conditions, static and / or dynamic objects, and / or other output 204, data can be delta encoded, extrapolated, interpolated, and / or otherwise compressed in order to reduce the amount of data included in map stream 210. For sensor data 102 such as LiDAR data, RADAR data, ultrasound data, etc., points represented by the data can be voxelized such that repeated points are removed, and volumes are represented by the data. This can allow for lower density points to be encoded in map stream 210 while still including sufficient information for accurate and detailed map creation 106. RADAR data, for example, can use octree encoding. Further, quantization of RADAR points and / or points from other sensor modalities can be performed to minimize the number of bits used to encode RADAR information. For example, since RADAR points can have x and y positions, the RADAR points can be encoded using fewer bits using line dividers, markers, and / or other information’s geometry in addition to the x and y positions.

[0069] In some embodiments, a buffer or other data structure can be used to encode map stream data into serialized structured data. Thus, instead of having a single field of a data structure describing some amount of information, bit packing into an array of bytes can be performed, for example, such that data in the buffer includes only numbers and not field names in order to provide bandwidth and / or storage savings compared to systems that include field names in the data. As a result, an integer can be used to identify each field to define the schema as associating data types with field names. For example, an interface description language can be used to describe the structure of the data, and a program can be used to generate source code from the interface description language for generating or parsing a byte stream representing the structured data. As such, a compiler can receive the file and produce an application programming interface (API) (e.g., a cAPI, a python API, etc.) that indicates how a system consumes information from a pin file encoded in this particular way. Additional compression can be achieved by chunking the data into smaller (e.g., 10,000 byte) chunks, converting the data into an array of bytes, and then transmitting or uploading to the cloud for map creation 106.

[0070] In some embodiments, dynamic obstacle removal can be performed using LiDAR data, RADAR data, and / or data from other sensor types to remove or filter out sensor data 102 corresponding to dynamic objects (e.g., vehicles, animals, pedestrians, etc.). For example, different frames of sensor data 102 can be compared (e.g., after motion compensation) to determine points that are inconsistent across frames. In such examples, for instance, where a bird can fly through the sensing field of a LiDAR sensor, points corresponding to the detected bird at one or more frames can not be present in one or more previous or subsequent frames. Accordingly, these points corresponding to the detected bird can be filtered out or removed such that they are not used during the generation of a final LiDAR layer of the HD map during map creation 106.

[0071] As such, sensor data 102 can be processed by the system to produce outputs corresponding to image data, LiDAR data, RADAR data, and / or trajectory data, and one or more of these outputs can undergo post-processing (e.g., perception outputs can undergo fusion to produce fused perception outputs, LiDAR data can undergo dynamic obstacle filtering to produce filtered LiDAR data, etc.). The produced data can be aggregated, merged, edited (e.g., trajectory completion, interpolation, extrapolation, etc.), filtered (e.g., landmark filtering for creating continuous lane lines and / or road boundary lines), and / or otherwise processed to generate a map stream 210 representing this data generated from any number of different sensors and / or sensor modalities.

[0072] Referring now to FIG. 3 , each block of the method 300 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 300 can also be embodied as computer-usable instructions stored on computer storage media. The method 300 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with other hosted services), or a plug-in to another product, to name a few. FIG. 2 The method 300 is described by way of example with reference to the processes 104 of

[0073] FIG. 3 is a flowchart illustrating a method 300 for map stream generation in accordance with some embodiments of the present disclosure. At block B302, the method 300 includes generating sensor data using sensors of a vehicle. For example, sensor data 102 can be produced.

[0074] At block B304, the method 300 includes applying at least a first subset of the sensor data to the one or more DNNs. For example, the sensor data 102 can be applied to the one or more DNNs 202.

[0075] At block B306, the method 300 includes computing an output using the one or more DNNs. For example, the one or more DNNs 202 can compute one or more outputs 204, and in one or more embodiments, the outputs can include, but are not limited to, lane divider information, road boundary information, static object information, dynamic object information, wait condition information, intersection information, sign, pole, or traffic light information, and / or other information corresponding to objects (static and / or dynamic) in the environment of the vehicle 1500.

[0076] At block B308, the method 300 includes converting at least a first subset of the output to a 3D coordinate system with a vehicle origin to generate a converted output. For example, the data converter 206 can convert the one or more outputs 204 to a 3D coordinate system with the origin of the vehicle 1500 as the origin.

[0077] At block B310, the method 300 includes converting at least a second subset of the sensor data to a 3D coordinate system to generate converted sensor data. For example, at least some of the sensor data 102 can be used directly in the map stream 210, but can not be generated in a 3D coordinate system relative to the origin of the vehicle 1500. Thus, the sensor data 102 can be converted to a 3D coordinate space with the vehicle 1500 as the origin.

[0078] At block B312, the method 300 includes compressing and / or minimizing the sensor data, the converted sensor data, the converted output, and / or the second subset of the output to generate compressed data. For example, the sensor data 102 and / or the outputs (e.g., without conversion or after conversion in the 3D coordinate space relative to the vehicle 1500) can be compressed and / or minimized. The compression and / or minimization can be performed using any known techniques, including but not limited to those described herein.

[0079] At block B314, the method 300 includes encoding the compressed data, the sensor data, the converted sensor data, the converted output, and / or the second subset of the output to generate a map stream. For example, the sensor data 102 (with or without conversion) and / or the outputs 204 (with or without conversion) (e.g., after compression by the data compressor 208) can be encoded to generate the map stream 210. The map stream 210 can then be transmitted to the cloud for map creation 106.

[0080] It can be repeated for any number of vehicles, 1500, for any number of drivers or their segments. FIG. 3 The process described. Information from each map stream 210 can then be used for map creation 106.

[0081] Map creation

[0082] refer to FIG. 4 , FIG. 4 A data flow diagram depicting a map creation process 106 according to some embodiments of the present disclosure is provided. In some embodiments, one or more data centers (such as...) may be used. FIG. 17 Example data center 1700) computing devices (e.g., similar to FIG. 16 Example computing device 1600) executes process 106 in the cloud. In some embodiments, process 106 may be executed using one or more virtual machines, one or more discrete computing devices (e.g., servers), or a combination thereof. For example, virtual graphics processing units (GPUs), virtual central processing units (CPUs), and / or other virtual components may be used to execute process 106. In some embodiments, one or more process steps described with respect to process 106 may be executed in parallel using one or more parallel processing units. For example, the registration 402 of fragment pairs (e.g., cost space sampling, aggregation, etc.) may be executed in parallel (e.g., the first pair may be registered in parallel with another pair). Additionally, within a single registration, the pose sampled for updating the cost of points in the cost space may be executed in parallel with one or more other poses. Furthermore, because map data corresponding to layers of an individual map can be stored as textures on the GPU, texture lookup can be performed to quickly determine the cost values ​​of the cost space—resulting in a reduction in the runtime of each cost space analysis.

[0083] The map creation process 106 can include receiving map streams 210 from one or more vehicles 1500 corresponding to any number of drives. As described herein, each map stream 210 can include various layers of data generated using a variety of different methods, such as by tracking ego motion (e.g., relative and global), sensor data 102 generation and processing, perception using one or more DNNs 202, etc. Each layer of a map stream 210 can correspond to a series of frames corresponding to sensor events recorded at a variable frame rate. For example, a map stream 210 layer can correspond to one or more camera layers, one or more LiDAR layers (e.g., a layer for each different slice), one or more RADAR layers, one or more trajectory (or ego motion) layers, and / or other layers. A camera layer can contain information obtained by performing perception (e.g., via a DNN 202) on a stream of 2D camera images and converting the (2D and / or 3D) detections or outputs 204 of the DNN 202 into 3D landmarks and paths (e.g., by combining lane marks to define lane line and road boundary locations). A LiDAR and / or RADAR layer (or other sensor modality layer) can correspond to point cloud information collected using a LiDAR sensor 1564 and / or a RADAR sensor 1560, respectively. As described herein, during the map stream generation process 104, pre-processing can be performed on the LiDAR data and / or RADAR data, such as data reduction, ego motion compensation, and / or dynamic obstacle removal. A trajectory layer can contain information corresponding to the absolute position of the origin or equipment of the vehicle 1500 and relative ego motion over some timeframe.

[0084] Referring to FIG. 5A For each map stream 210(1)-210(N), the map creation process 106 can include converting the map stream 210 to a map 504(1)-504(N) via a conversion 502(1)-502(N) (e.g., converting the map stream to a DriveWorks map format), where N corresponds to the number of map streams 210 used for a particular registration process. For example, referring to FIG. 5BFor individual map stream 210(1), conversion 502(1) can include base conversion 506, RADAR conversion 508, LiDAR height slicing 510, LiDAR conversion 512, RADAR map image creation 514, LiDAR map image creation 516, and / or LiDAR voxelization using LiDAR voxelizer 518. Base conversion 506 can correspond to landmarks as determined using perception via one or more DNNs 202 (e.g., lane lines, road boundary lines, signs, poles, trees, other vertical structures or objects, sidewalks, etc.). For example, 3D landmark positions can be converted into a map format using base conversion 506 to generate a base layer 520 (or “camera layer” or “perception layer”) of map 504(1). In addition to landmark positions, base layer 520 can also represent a trajectory or path (e.g., global or relative) of vehicle 1500 that generated map stream 210(1). When generating base layer 520, a 1:1 map between aggregated input frames of map stream 210(1) and output base layer 520 map road segments can be maintained.

[0085] RADAR data from map stream 210(1) - e.g., when received or accessed in raw format - can be converted via RADAR conversion 508 into a RADAR point cloud layer 522. In some embodiments, RADAR point clouds from RADAR point cloud layer 522 can be used to generate a RADAR map image layer 524 via RADAR map image creation 514. For example, RADAR point clouds can be converted from one or more different perspectives (e.g., top-down, sensor perspective, etc.) into one or more images of the RADAR point cloud. For example, a virtual camera with an overhead field of view can be used to project the RADAR point cloud into a frame of the virtual camera to generate a RADAR map image of RADAR map image layer 524.

[0086] LiDAR data from map stream 210(1) - e.g., as received or accessed in raw format - can be converted to LiDAR point cloud layers 526 via LiDAR conversion 512. In some embodiments, as described herein, LiDAR data can be generated in slices (e.g., ground slices (e.g., from 5 meters to 300 meters from an origin of vehicle 1500), giraffe plane slices (e.g., from 2.5 meters to 5 meters from an origin of vehicle 1500), ground plane slices (e.g., from -2.5 meters to 0.5 meters from an origin of vehicle 1500), etc. In addition to or instead of LiDAR height slicing in map stream generation process 104, LiDAR height slicing 510 can be performed during conversion 502(1) to determine separate slices of LiDAR data for use in generating one or more LiDAR point cloud layers 526. For example, LiDAR data corresponding to a particular slice (e.g., a giraffe slice) can be used to generate a LiDAR point cloud layer 526 of map 504(1). In some embodiments, LiDAR point clouds from LiDAR point cloud layers 526 can be used to generate LiDAR map image layers 528 via LiDAR map image creation 516. For example, a LiDAR point cloud (e.g., corresponding to a particular slice) can be converted into one or more images of the LiDAR point cloud from one or more different perspectives (e.g., top-down, sensor perspective, etc.). For example, a virtual camera with an overhead view can be used to project a LiDAR point cloud into a frame of the virtual camera to generate a LiDAR map image for a LiDAR map image layer 528. LiDAR map image layers 528 can include LiDAR map images encoded with height values determined from the point cloud (e.g., top-down depth maps), LiDAR map images encoded with intensity values determined from the point cloud, and / or other LiDAR map image types. In some embodiments, LiDAR point cloud layers 526 can be used to generate LiDAR voxel map layers 530 of map 504(1), e.g., by LiDAR voxelizer 518. LiDAR voxel map layers 530 can represent a voxelized representation of the LiDAR point cloud. Voxel map layers 530 can be used to edit LiDAR data in individual maps 504 and / or fuse LiDAR data in HD maps (as described in greater detail herein), e.g., to filter out dynamic objects.

[0087] Referring again to FIG. 5AThe process described with respect to map stream 210(1) can be performed for each map stream 210(1)-210(N) to generate maps 504(1)-504(N). Maps 504(1)-504(N) or a subset thereof can then be used for registration 402. As such, once maps 504 have been generated for each map stream 210 to be used in the current registration process, registration 402 can be performed to generate an aggregated map corresponding to multiple map streams 210. The aggregated map can include aggregated layers— e.g., aggregated camera or base layers, aggregated LiDAR layers, aggregated RADAR layers, etc. To determine which map streams 210 (and thus maps 504) should be registered together, the locations of map streams 210 or segments thereof can be determined. For example, referring to FIG. 5C , a first map stream segment 540 (e.g., representing a trajectory of vehicle 1500 that generated map stream 210(1)) can correspond to first map stream 210(1), a second map stream segment 542 can correspond to second map stream 210(2), and a third map stream segment 544 can correspond to third map stream 210(3). To determine that these map streams 210(1)-210(3) correspond to similar locations or road segments, location or trajectory information— e.g., GNSS data— can be used. For example, GNSS coordinates from map streams 210 and / or maps 504 can be used to determine whether map streams 210 or segments thereof are close enough in space for a long enough distance to be registered to one another. Once located using GNSS coordinates, relative coordinates from map streams 210(1)-210(3) can be used to determine how close map stream segments are (e.g., how close trajectories of vehicle 1500 are). This process can yield a final list of map stream segments to be registered. For example, GNSS coordinates and / or relative coordinates from map streams 210 and / or maps 504 can be used to determine that map stream segments 540, 542, and 544 are on the same road— or at least have portions that overlap between demarcations 550A and 550B.

[0088] In some examples, individual map streams 210 or drives— e.g., map stream 210(1) corresponding to map stream segment 540— can include loops. For example, during map stream generation process 104, vehicle 1500 can have traveled the same road segment two or more times at different times. Accordingly, map stream segment 540 can be split into two separate segments 552A and 552B via deduplication, and treated as individual map stream segments for registration 402. The deduplication process can be performed as a pre-processing step that identifies when vehicle 1500 (or map streams 210 corresponding thereto) is traveling in a loop or a circle. Once a loop is identified, a drive or map stream 210 can be split into two or more segments to ensure that each physical location is represented only once.

[0089] A layout of all map stream segments that are determined to be within a certain spatial threshold of each other can be generated. For example, FIG. 5C An example visualization of the layout of map stream segments 540, 542, and 544 can be represented. In FIG. 5C In the example of registration 402, the overlapping portions 552A-552D between demarcations 550A and 550B can be used, for example. This layout can then be used to determine which pairs of overlapping portions 552A-552D to register together. In some embodiments, a minimum spanning tree algorithm can be used to determine the minimum number of portions 552 to register together to have a connection between each of the portions 552 (e.g., if portion 552A and portion 552B are registered together, and portion 552B and 552C are registered together, then portion 552A and 552C have a connection via portion 552B). The minimum spanning tree algorithm can include randomly selecting a pair of portions 552, and then randomly selecting another pair of portions until the minimum spanning tree is complete. In some embodiments, in addition to the minimum spanning tree algorithm, for robustness and accuracy, an error margin can be used to include additional pairs.

[0090] As an example, with respect to FIG. 5D Table 560 illustrates different numbers of connections or pairs between map streams or map stream segments using various different techniques. The number of segments M can correspond to the total number of map streams or map stream segments available for registration 402 for a particular segment (e.g., portions 552A-552D of FIG. 5C If registration is performed for every possible connection between pairs of segments, then all connections can correspond to the number of pairs to be registered. As an example, to calculate all connections, the following equation (1) can be used:

[0091] All connections = M * (M - 1) / 2 (1)

[0092] As another example, to calculate minimum spanning tree connections, the following equation (2) can be used:

[0093] Minimum spanning tree connections = M - 1 (2)

[0094] As a further example, to calculate safe margin connections—e.g., the minimum spanning tree connections plus a safe margin— the following equation (3) can be used:

[0095] Safe margin connections = MIN [(minimum spanning tree connections * 2), all connections] (3)

[0096] Accordingly, also with reference to FIG. 5CUsing equation (3) for the portions 552A-552D, connections 554A-554F can be determined. For example, because there are four portions or segments to be registered, the safety margin will result in six connections between the four portions or segments of the map stream section between boundaries 550A and 550B.

[0097] In some embodiments, to determine which segments to use and / or to determine which segments to use more frequently (e.g., when some portions are registered more than once), the quality of the segments can be analyzed. For example, the quality can correspond to the number of sensor modalities represented in the map stream segment. As such, in a case where one segment includes a camera, a trajectory, RADAR, and LiDAR, while another segment includes only a camera, a trajectory, and RADAR, the segment with the LiDAR can be weighted or favored such that this segment is more likely to be included in more registrations 402. Further, in some embodiments, when determining segments to be registered together, a geometric distance can be used as a criterion. For example, two segments that are geometrically closer can be selected on a geometric distance segment, e.g., because closer positioned segments can correspond to the same lane of travel, as opposed to a farther segment that can correspond to a different lane of travel or an opposite side of a road.

[0098] Referring again to FIG. 4 and FIG. 5A , each pair of segments can then be registered to each other to generate a pose link between them, which can be used for pose optimization 404. The registrations 402 can be performed to determine a number of different locations at which the drive or map stream 210 overlap, e.g., at boundaries 550A-550F. FIG. 5Ccorresponding to the multi-polar drive or map stream 210 between the geometric relationships of the multi-polar drive or map stream 210 in the portions 552A-552D. The output of the registration process can include relative pose links between pairs of frames or segments in different map streams (e.g., a rotation and a translation between a pose of a first map stream frame or segment and a pose of a second map stream frame or segment). In addition to the relative pose, the pose link can further represent a covariant representing a confidence in the corresponding pose link. The relative pose links are then used to align the maps 504 corresponding to each of the map streams 210 so that landmarks and other features - e.g., point clouds, LiDAR image maps, RADAR image maps, etc. - are aligned in the final aggregated HD map. Thus, the registration process is performed by positioning one map 504 or portion thereof to another map 504 or portion thereof. The registration 402 can be performed for each sensor modality and / or for each map layer corresponding to different sensor modalities. For example, camera-based registration, LiDAR-based registration, RADAR-based registration, and / or other registrations can be performed separately. The results can include an aggregated base layer of the HD map, an aggregated LiDAR point cloud layer of the HD map, an aggregated RADAR point cloud layer of the HD map, and an aggregated LiDAR map image layer of the HD map, etc.

[0099] As described in greater detail herein, the positioning process used to position one segment to another segment for registration can be performed similarly to the positioning process 110 used in live perception. For example, the cost space can be sampled for a single frame or pose pair, an aggregated cost space can be accumulated over frames or poses, and a Kalman filter can be used on the aggregated cost space to position one segment relative to another. As such, the poses of the trajectories from the first segment can be known and the poses of the trajectories from the second segment can be sampled relative to the first segment using the positioning in order to determine the relative poses between the two once alignment between landmarks is achieved. This process can be repeated for each of the poses of the segments so that pose links between the poses are generated.

[0100] For example, with respect to the base layer or perception layer of map 504, during registration 402, 3D world space locations of landmarks can be projected into the virtual field of view of a virtual camera to generate an image corresponding to 2D image space locations of landmarks within the virtual image. In some embodiments, the 3D world space landmark locations can be projected into a field of view or more than one virtual camera to generate images from different perspectives. For example, a forward-facing virtual camera and a rear-facing virtual camera can be used to generate two separate images for localization in the registration process. Using two or more images can increase the robustness and accuracy of the registration process. To register to an image, virtual image space locations of landmarks from two segments undergoing registration 402 can be compared to each other (e.g., using camera-based localization techniques) to sample a cost space representing locations consistent between segments. For example, detections from a first segment can be converted into a distance function image (e.g., as described herein with respect to localization 110), and detections from a second segment can be compared to the distance function image to generate a cost space. In addition to comparing geometry, semantic information can also be compared (e.g., simultaneously or in separate projections) to compute costs. For example, where semantic information (e.g., lane line type, pole, sign type, etc.) does not match one or more particular points, the cost of the one or more particular points can be set to a maximum cost. The geometric and semantic costs can then be used together to determine a final cost for each pose. This process can be performed on any number of frames to determine an aggregated cost space that is more finely tuned than any individual cost space, and a Kalman filter (or other filter type) can be performed on the aggregated cost space to determine a pose and pose link between the two segments.

[0101] Referring to FIG. 5E and 5F , FIG. 5E and 5F An example of camera or base layer based registration is shown, according to some embodiments of the present disclosure. For example, FIG. 5E Visualization 570A can correspond to a forward-facing virtual camera registration, and FIG. 5FThe visualization 570B can correspond to a rear-facing virtual camera. The 3D landmark locations from the base layer of the first map 504 can be projected into the 2D image space of the front-facing virtual camera and the rear-facing virtual camera. The 2D projections can then be converted into distance functions - shown as stippled patterns in visualizations 570A and 570B - where the centerline of each stippled segment 572A-572K can correspond to a zero cost, and as the dots move away from the centerline of the segment, the cost increases, until a white region can correspond to a maximum cost. The dashed segments 572 can correspond to distance function equivalents of landmarks. For example, dashed segments 572A-572C and 572J-572K can correspond to lane lines and / or road boundaries, and dashed segments 572D-572I can correspond to poles, signs, and / or other static objects or structures.

[0102] The distance functions can then be compared to the 3D landmark information from the 2D projections of the base layer of the second map 504. For example, the solid black segment 574 (e.g., including lines and / or points) can represent the 2D projection of the 3D landmarks of the second map 504 at the sampled pose 576. As such, this comparison of the 2D projection to the distance function projection of the first map 504 can correspond to a single location on the cost space that corresponds to the pose 576, and any number of other poses can also be sampled to fill out the cost space. Thus, for example, each point along the solid black segment 574A can have a relatively low cost, as many of the points match up with the points along the dashed segment 572A. Conversely, each point along the solid black segment 574J can have a high cost, as many of the points do not match up with the points along the dashed segment 572J. Ultimately, for the pose 576, the cost corresponding to each of the points of the solid black segments 574A-574K can be computed, e.g., using an average, and the point in the cost space that corresponds to the pose 576 can be updated to reflect the computed cost. This process can be repeated for any number of poses (e.g., every possible pose) until the most likely relative pose of the base layer of the second map 504 relative to the base layer of the first map 504 is determined for a particular time step or frame. The process can then be repeated for each time step or frame of the pair of segments being registered together (e.g., with reference to FIG. 5C The process can be repeated for each respective time step or frame from portions 552A and 552B, for each respective time step or frame from portions 552C and 552D, etc.

[0103] As another example, with respect to the LiDAR point cloud layers of map 504, during registration 402, the LiDAR point cloud from the first segment can be converted to a distance function point cloud and compared to the LiDAR point cloud of the second segment to generate a cost space. For example, the cost can be sampled at every possible pose of the second segment relative to the distance function image corresponding to the first segment. This process can similarly be performed with respect to LiDAR intensity map images, LiDAR height map images, and / or other LiDAR image types. Similarly, for RADAR, registration 402 can be performed in this manner.

[0104] The result of registration 402 between any pair of segments is a pose link defining the rotation, translation, and / or covariance between the poses of the segments. This registration process is performed for each pair of segments determined from the connections (e.g., safety margin connections 554A-554F) of the graph 550. FIG. 5C Using the output of the registration process, a pose link or pose graph of the poses representing the relative positions of the different drives or segments based on time stamp can be generated.

[0105] Referring again to FIG. 4 and 5AAfter the registration process 402 is completed and pose links have been generated between frames or poses of the registration pairs, pose optimization 404 can be performed to smooth the pose links, segment groupings of poses from various different drives into segments corresponding to road segments for relative localization in the final aggregated HD map. The pose optimization 404 can be performed to obtain a more desirable or optimal geometric alignment of sub-maps or layers based on the input absolute poses from the map stream 210, the relative pose links generated during registration 402, and the relative trajectory pose links generated using self-motion for each individual drive. Prior to pose optimization, a pose graph can be generated to represent the absolute and relative poses. In some embodiments, a filtering process can be performed to remove poses with low confidence - e.g., using covariances as determined during the registration process 402. The (filtered) frames or poses can be aggregated into road segments that can be different from the segments of the individual input maps 504, e.g., because the same area need only be represented by a single road segment rather than multiple overlapping road segments. During the pose optimization process 400, each output road segment can be associated with an absolute pose or origin. Given an initial pose graph layout, an optimization can be performed to minimize the pose error of the absolute poses relative to the observed relative poses and input absolute poses, while taking into account the uncertainty information associated with each observation (e.g., maximum likelihood optimization). The optimization can be initialized by way of a random sample consensus (RANSAC) process to find a subset of the maximum consistent pose links measured via cyclic consistency checks across loops within the pose graph. Once optimized, each road segment can be compared to coupled road segments to determine the relative transforms between road segments, such that during localization 110, the cost function calculations can be transferred from road segment to road segment using the transforms, such that an accurate cumulative cost space can be computed.

[0106] For example, FIGS. 6A-6I An example pose optimization process 404 corresponding to four drives or segments from the same and / or different maps 504 is shown. In some embodiments, the pose optimization process 404 can be performed separately for different sensor data modalities or layers of the maps 504. For example, any number of base layers of the maps 504 can be registered together and subsequently pose optimized, any number of LiDAR point cloud layers of the maps 504 can be registered together and subsequently pose optimized independently of the pose optimization of the base layers, and so on. In other embodiments, the results of the registration process for different layers of the maps 504 can be used to generate aggregated poses and pose links, and the pose registration process can be performed collectively.

[0107] Referring to FIG. 6ARegistration segments from four different drives 602A-602D are shown in frame graph 600A. Frame graph 600A is an example of a portion of a larger frame graph, where the larger frame graph can correspond to any portion or all of a road structure or layout. The four drives 602A-602D can have been registered together to generate poses 604 or pose links 608 (e.g., pose links 608A-608C) between the positions of the different drives 602. Each pose 604 or position of a single drive 602 can be represented by a link 606. For example, a link 606 can represent a translation, rotation, and / or covariance from one pose 604 to the next pose 604. As such, link 606A(6) can represent a rotation, translation, and / or covariance between the poses 604 of drive 602A connected by link 606A(5). Pose links 608A can correspond to the output of registration process 402, and can encode the translation, rotation, and / or covariance between the poses 604 of the different drives 602. For example, pose link 608A can encode a six degree of freedom transformation between the poses — e.g., a translation (e.g., a difference in (x, y, z) position), a rotation (e.g., a difference in x, y, and z axis angles), and / or a covariance (e.g., a confidence corresponding to the values of the pose link) between pose 604A(l) of drive 602A and pose 604B(2) of drive 602B.

[0108] FIG. 6BRoad segment generation is shown whereby pose sets from any number of drives 602 can be combined into a single road segment. The resulting road segment can be included in a final aggregated HD map for local or relative positioning. Further, the resulting road segment can have an origin, and the origin can have a relative position in a global coordinate system, such as the WGS84 coordinate system. Frame map 600B, which can correspond to frame map 600A but with road segment identification, shows how the frame map can be separated into different road segments 610A-610D. For example, poses with horizontal striped fill can correspond to a first road segment 610A, poses with vertical striped fill can correspond to a second road segment 610B, poses with dashed line fill can correspond to a third road segment 610C, and poses with shaded fill can correspond to a fourth road segment 610D. To determine which poses 604 or frames are to be included in each road segment, a first random pose 604 - such as pose 604A(6) - can be selected. Once a random pose 604 is selected, links 606 and / or pose links 608 can be iterated from pose 604 to pose 604, starting at the randomly selected pose 604 until a maximum distance corresponding to the road segment 610 is reached (e.g., 25 meters, 50 meters, etc.), or a pose 604 or frame is determined to have already been encoded to road segment 610. Once road segment 610 is fully encoded, another random, unencoded pose 604 can be selected and the process can be repeated. This can be repeated until each of the poses 604 are encoded to or included in road segment 610. Due to the random nature of this process, some very small road segments 610 can be produced. As such, a post-processing algorithm can be executed to analyze road segment size and merge road segments below a certain size threshold into neighboring road segments.

[0109] FIG. 6C A segment map 620 of road segments 610A-610D after the road segment encoding or generation process is shown. For example, each block labeled with 610A to 610D can correspond to a collapsed representation of all poses 604 corresponding to the respective road segment 610. Once the road segments 610 are determined, an origin or seed location for each road segment can be determined. The origin or seed location can correspond to the center of the road segment 610 (e.g., the origin can be (25m, 25m) in the case of a road segment that is 50 meters by 50 meters long). In some embodiments, the origin can correspond to the average or median position of each of the poses 604 within the road segment 610. For example, the (x, y, z) coordinates corresponding to each pose 604 can be averaged, and the result can be selected as the origin. In other embodiments, a different method can be used to select the origin.

[0110] FIG. 6DPose link error processing is shown. For example, it can not be guaranteed that the pose link transformations between poses of a frame graph are consistent with each other. Thus, in the case where three poses 604 are taken (as in FIG. 6D the road segment 610 with the lowest pose link error 612 can be determined. In one or more embodiments, the layout of poses determined during pose optimization 404 can not be globally consistent, but can be optimized such that the layout is consistent within the same road segment 610 and adjacent road segments 610. For example, pose links 614A and 614B should be approximately the same, and pose link 614C should be approximately equal to the combination of pose links 614A and 614B. However, in practice, due to some error (e.g., positioning error during registration 402), there can be some delta or pose link error 612 that manifests between the poses 604. Thus, pose optimization 404 can be used to move or shift the poses 604 or frames such that the best fit is achieved within the corresponding road segment 610. With respect to pose links 614A and 614B, the poses 604 can be shifted to distribute the error between all of the pose links 614, rather than having the pose link error 612 primarily manifest in pose link 614C.

[0111] FIG. 6E A pose graph 630 is shown that can represent FIG. 6B frame graph 600B, but the pose links connecting road segments 610A-610D to the external road segment 610 are omitted. FIG. 6F A RANSAC operation performed on the pose graph 630 is shown. This process can be performed based on the dependence that the links 606 between poses are more accurate than the pose links 608. For example, a minimum set or minimum spanning tree of pose links 608 in the pose graph 630 that keeps all nodes or poses 604 connected can be sampled. Once the minimum set is selected, the remaining portion of the pose links 608 can be sampled to see how many pose link errors 612 are detected. This can correspond to one iteration, and at the next iteration, another random minimum set of pose links 608 can be selected, and then all of the remaining portion of the pose links 608 can be sampled for pose link errors 612, and so on for a number of iterations (e.g., 100, 1000, 2000, etc.). Once the number of iterations is complete, the layout with the most consistency can be used. For example, with respect to FIG. 6F the cost of error can be computed for each of the pose links 608 other than the minimum sample— e.g., according to the following equation (4):

[0112]

[0113] where u = log(Err), u e se3, Err e SE3, Err is the difference between the layout of the current evaluated road segment 610, the pose link transform, and the transform between poses connected by the pose link, and s i is the standard deviation of the i-th component of the pose link transform (e.g., transform confidence). As such, the cost can express the pose link error 612 in terms of the number of standard deviations. The RANSAC sampling can be repeated over a number of iterations, and at each iteration, the number of layouts whose pose links fit within N standard deviations can be counted. The pose links that satisfy this condition and are included in the count can be referred to as inliers.

[0114] Reference is made to FIG. 6G , FIG. 6G An updated pose graph 650 after the RANSAC processing is complete is shown - e.g., the pose graph 650 shows the pose arrangement with the most inliers. FIG. 6G The pose graph 650 can undergo an optimization process, such as a non-linear optimization process (e.g., a bundle adjustment process), which aims to minimize the sum of the squared costs of the inliers, e.g., using the computed cost function described herein with respect to FIG. 6F After bundle adjustment, the pose graph 650 can be fixed, and the road segments 610A-D can be fixed, such that the poses 604 can be in their final fixed positions for the road segment 610 - e.g., as shown in the example road segment 610C in FIG. 6H As described herein, the final road segment 610 can have a relative origin (e.g., for positioning relative to a local or relative coordinate system). As such, and since the relative origin has a position in the global coordinate system, then positioning to the local coordinate system can also be positioned to the global coordinate system. For example, with respect to FIG. 6H , the road segment 610C can have an origin 652C (as shown). Although not shown, the other road segments 610A, 610B, and 610D can also have respective origins 652.

[0115] Reference is made to FIG. 6I Each road segment 610 can have a relative transform computed for each directly adjacent road segment 610. In the illustration of FIG. 6I , the road segment 610C can have a relative transform determined between the road segment 610C and the road segment 610A (e.g., transform, T610C 610A ), a relative transform determined between the road segment 610C and the road segment 610B (e.g., transform, T610C 610B ), and a relative transform determined between the road segment 610C and the road segment 610D (e.g., transform, T610C 610D). Relative transforms can be used during localization 110 - e.g., more specifically when generating the aggregate cost function and / or when generating a local map layout for localization - because the aggregate cost space can include one or more cost spaces that arise when localizing with respect to, e.g., road segment 610A and can include cost spaces that arise when localizing with respect to, e.g., road segment 610C. Thus, when entering road segment 610C from road segment 610A, the cost space can be updated or transformed - e.g., using transform T610C 610A - to update the cost space so that each aggregate cost space references the same origin (e.g., origin 652C of road segment 610C). Relative transforms can represent six degrees of freedom, e.g., rotations (e.g., differences between x, y, and z axis rotation angles) and translations (e.g., differences between (x, y, z) coordinates).

[0116] Referring again to FIG. 4 After registration 402 and pose optimization 404, the updated pose graph (e.g., pose graph 650) can be fixed and, as a result, the relative poses of the plurality of drives 602 can be fixed. As such, the map 504 from each drive 602 (whose output is now currently aligned) (e.g., as a result of registration 402 and pose optimization 404) can be used to fuse (e.g., via fusion 406) the map layers of the individual maps 504 together to form an aggregate HD map. The aggregate HD map can include the same individual layers as the individual maps 504, but the aggregate HD map can include aggregate layers - e.g., an aggregate base layer, an aggregate LiDAR point cloud layer, an aggregate RADAR point cloud layer, and an aggregate RADAR map image layer, etc. The fusion process 604 can be used to improve the data quality of the map data relative to the data present in any single drive - e.g., the fused map can be more accurate or reliable than the individual maps 504 from the individual drive map streams 210. For example, inconsistencies between the maps 504 can be removed, the precision of the map content can be improved, and a more complete representation of the world can be achieved - e.g., by combining the geographic ranges of two different map layers or sub-maps, or by combining observations about wait conditions across multiple drives through the same intersection. Once the final (e.g., more optimized or desired) geometric layout of the poses 604 of the different drives 602 is determined by the pose graph (e.g., pose graph 650), the information from the plurality of maps 504 and / or map streams 210 corresponding thereto can be readily converted into the same output coordinate system or pose associated with each road segment 610.

[0117] As an example, with respect to 3D landmark locations represented in the base layer, 3D landmark locations from multiple maps 504 can be fused together to generate a final representation of each lane line, each road boundary, each sign, each pole, etc. Similarly, for LiDAR intensity maps, LiDAR height maps, LiDAR range function images, RADAR range function images, and / or other layers of maps 504, individual map layers can be fused to generate an aggregated map layer. Thus, where a first map layer includes data that matches up with data of another map layer within some threshold similarity, the matching data can be used (e.g., averaged) to generate a final representation. For example, with respect to the base layer, where data corresponds to a sign, a first representation of the sign in a first map 504 can be compared to a second representation of the sign in a second map 504. Where the first and second signs are within a threshold distance of each other and / or have the same semantic class, a final (e.g., averaged) representation of the sign can be included in the aggregated HD map. This same process can be performed for lane dividers, road boundaries, wait conditions, and / or other map information. Conversely, where data from a map layer of a map 504 does not match up with data from other map layers of other maps, that data can be filtered out. For example, where a first map 504 includes data for a lane divider at a location and semantic class, and one or more other maps 504 do not share that information, the lane divider from the first map 504 can be removed or filtered out from consideration for the aggregated HD map.

[0118] As described herein, the fusion process 406 can be different for different map layers (or different features represented therein) of the maps 504. For example, for lane map fusion (e.g., fusion of lane dividers, road boundaries, etc.), individually observed lanes, paths, and / or trajectories from multiple base map layers of multiple maps 504 can be fused into a single lane map in the aggregated HD map. The lane boundaries and / or lane dividers of the individual maps 504 can be fused not only for a lane map (e.g., dividers that are boundaries of lanes that the vehicle 1500 can travel), but also for camera-based localization as 3D landmarks or 2D landmarks generated from 3D landmarks (e.g., as a set of stable semantic landmarks). In addition to lane boundaries or dividers, other road marks can also be collected from the map stream 210 and included in the maps 504 for the fusion process 604. For example, stop lines, road text, patchwork areas, and / or other markings can be fused together to serve as semantic landmarks for localization and / or for updating wait condition information. With respect to wait conditions in the base layers, wait conditions from multiple maps 504 can be fused to generate a final representation of the wait conditions. Poles, signs, and / or other static objects can also be fused from multiple base layers of the maps 504 to generate an aggregated representation thereof. The poles, signs, and / or other (vertical) static objects can be used for localization.

[0119] In some embodiments, lane dividers, lane centers (e.g., tracks), and / or road boundaries can not be clearly identified in the base layer from perception. In such embodiments, trajectory information from the base layer and / or map stream 210 can be used to infer tracks and / or lane dividers. For example, after bundle adjustment, pose graph 650 can be viewed from a top view to determine trajectory patterns. Trajectories determined to be in the same lane can be used to generate lane dividers and / or tracks for a particular lane. However, to determine that two or more trajectories are from the same travel lane, heuristics can be used. For example, to determine to cluster two trajectories together as belonging to the same lane, the current poses or frames can be compared. In cases where the current frames or poses of the two trajectories appear to match the same travel lane (e.g., based on some distance heuristic), poses some distance (e.g., 25 meters) before the current pose and some distance (e.g., 25 meters) after the current pose of the two trajectories can also be analyzed. In cases where the current pose, the previous pose, and the forward pose all indicate the same travel lane, the current poses can be clustered together to determine lane dividers, lane tracks, and / or road boundary locations. Thus, when one trajectory corresponding to a vehicle 1500 changes lanes while another vehicle 1500 remains stable in the lane, the combination of the two trajectories will result in an inaccurate representation of the actual lane dividers, lane tracks, and / or road boundaries. In some embodiments, rather than, or in addition to, analyzing the distance between the poses of the trajectories, the angle formed between the trajectories can be analyzed. For example, in cases where the angle difference is greater than some threshold (e.g., 15 degrees, 40 degrees, etc.), the two trajectories can be considered not to be from the same lane and can not be clustered together.

[0120] With respect to LiDAR and / or RADAR map layers, fusion of LiDAR and RADAR points from multiple maps 504 can ensure that the fused or aggregated map contains a more complete point cloud than any individual map 504 (e.g., due to occlusions, dynamic objects, etc.). The fusion process 604 can also reduce redundancy in the point cloud coverage by removing redundancies to reduce the amount of data needed to store and / or transmit the point cloud information of the aggregated HD map. For example, where point cloud points of one or more drives do not align with point cloud points from another drive, points from the non-matching drive can be removed - e.g., via dynamic object removal. The resulting aggregated map layers for LiDAR can include one or more point cloud layers of aggregated slices (e.g., ground plane slices, giraffe plane slices, etc.), a LiDAR height map image layer storing an average height for each pixel, and / or a ground reflectance or intensity map image layer storing an average intensity value for each pixel. For RADAR, the resulting aggregated map layers can include a RADAR cross-section (RCS) map image layer and / or a RADAR point cloud layer (which can or can not be sliced). In embodiments, in-memory representation of all LiDAR and RADAR map image layers can use floating point representation. For the elevation model, ground reflectance model, and / or RCS, pixels with no valid data can be encoded by a non-numeric (NaN) data type. With respect to the ground reflectance and height map image layers, the map image generator can search for peak density in the height distribution of each point. Only these points can be considered ground detections, and noise measurements can be filtered out (e.g., measurements corresponding to obstacles). In some embodiments, a median filter can be applied to fill in missing measurements, which would otherwise be gaps.

[0121] With respect to LiDAR, a voxel-based fusion strategy can be implemented. For example, in cases where a voxel is observed by only a few of many drives, the voxel can be determined to be a noisy detection. In implementations, this LiDAR fusion can be performed for each road segment 610. In non-limiting embodiments, LiDAR fusion can be performed according to the following procedure: (1) find a tight 3D bounding box of LiDAR points; (2) given a preset voxel resolution and a 3D point, (x, y, z), compute the index of the 3D point; (3) use a sparse representation of 3D volume instead of a dense representation for memory usage reduction; (4) save the ID of the drive or map stream 210 that observed the voxel and the average color in the voxel data; and (5) after updating the voxel volume with all points in the drive, threshold the number of drives in each voxel so that only voxels with a number of drives greater than a threshold are kept (e.g., the threshold can be set to half of the total drives 602 in the road segment 610). In addition to removing noisy 3D points, LiDAR fusion can generate and save point clouds in giraffe planes and ground plane slices. The 3D points in the respective planes or slices can then be used to generate different kinds of LiDAR map images for LiDAR localization. For example, a giraffe plane can include points in the z = [al, bl] range, and ground plane points can be in the z = [a2, b2] range, where al, bl, a2, b2 can be preset parameters. Thus, points are selected based on their z coordinate (height) in their frame coordinate system. Note that the filtering process is done based on the frame coordinates, not based on the road segment coordinates, so points can not be filtered based on their z coordinates after transforming the points to the road segment coordinate system. To address this, it can be noted that points belong to a giraffe plane or a ground plane before transforming the points, and this information can be saved inside the data structure associated with the voxel. If a voxel passes the noise filtering process and also belongs to one of these two planes, then the voxel can be saved to the corresponding point cloud file. Similar procedures can be performed for RADAR data.

[0122] In some embodiments, such as where the fusion process 406 is performed during a normal check 112, more outdated LiDAR data, RADAR data, and / or map data can be weighted more negatively than more recent data. As such, when an inconsistency is determined between a newer map 504 and an older map 504, data from the older map 504 can be filtered out, and data from the newer map 504 can be retained. This can be a result of changing road conditions, construction, etc., and more recent or current data can be more useful for navigation of the road segment 610.

[0123] After the fusion process 406, map data 108 (e.g., representing an aggregated HD map including an aggregation layer) can have been generated. The map data 108 can then be used for localization 110, as described in greater detail herein.

[0124] Referring now to FIG. 7 The blocks of the method 700 described herein each include a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, different blocks can be implemented by a processor executing instructions stored in memory. The method 700 can also be embodied as computer-usable instructions stored on computer storage media. The method 700 can be provided by a standalone application, a service or hosted service (standalone or in combination with other hosted services), or a plug-in to another product, to name a few. FIG. 4 The method 700 is described with respect to the process 106. However, the method 700 can additionally or alternatively be performed within any one process or by any one system or any combination of systems, including but not limited to those described herein.

[0125] FIG. 7 is a flowchart illustrating a method 700 for map creation according to some embodiments of the present disclosure. At block B702, the method 700 includes receiving, from a plurality of vehicles, data representing a plurality of map streams corresponding to a plurality of drives. For example, map streams 210 can be received for the map creation 106 process, where the map streams 210 can correspond to any number of drives from any number of vehicles 1500.

[0126] At block B704, the method 700 includes converting each map stream into a respective map including a plurality of layers to generate a plurality of individual maps. For example, each received map stream 210 can undergo the conversion 502 to generate a map 504, such that a plurality of maps 504 (e.g., maps 504(1)-504(N)) are generated. Each map 504 can include a plurality of layers, such as but not limited to the layers described with respect to FIG. 5B .

[0127] At block B706, the method 700 includes geometrically registering pairs of segments from two or more individual maps to generate a frame graph representing pose links between poses of the pairs of segments. For example, segments (e.g., portions 552A-552D) of different map streams 210 and / or maps 504 can be determined, e.g., using proximity-indicative trajectory information and GNSS data, and can be registered to one another (e.g., using a minimum spanning tree plus a safety margin in embodiments) to generate a pose graph (e.g., pose graph 602) representing pose links between poses of the pairs of segments. FIG. 6AThe pose graph can include poses for different drives 602 or pose links between poses. In some embodiments, different registration processes can be performed for different map layers and / or sensor modalities - e.g., because the frame rate of different sensors can be different, the pose of the vehicle 1500 at each frame can be different for different sensor modalities and thus also for different map layers.

[0128] At block B708, the method 700 includes assigning pose groups from the frame graph to road segments to generate a pose graph including poses corresponding to the road segments. For example, the processes described herein with respect to FIGS. 6A-6B can be performed to determine poses 604 from different drives 602 corresponding to a single road segment 610. Once the road segment 610C is determined, a pose graph including poses 604 from the road segment 610C and each adjacent road segment (e.g., 610A, 610B, and 610D) can be generated to generate a pose graph 630 - e.g., similar to the larger frame graph 600A, but without omitting the external pose connections (e.g., to poses 604 outside of the road segments 610A-610D). FIG. 6B

[0129] At block B710, the method 700 includes performing one or more pose optimization algorithms on the pose graph to generate an updated pose graph. For example, RANSAC operations (e.g., RANSAC, RANSAC-aided, etc.), pose graph optimization (e.g., g2o, etc.), and / or other optimization algorithms can be performed to generate an updated pose graph 650. FIG. 6F FIG. 6G

[0130] At block B712, the method 700 includes fusing map layers from the multiple individual maps based on the updated poses from the updated pose graph to generate a fused map. For example, once the relative poses 604 of each of the frames of the map layers are known, the data from the map layers can be fused to generate an aggregated or fused HD map layer. For example, 3D landmark locations from two or more base layers of the respective maps 504 can be fused according to an average position of their aligned poses. As such, where a pole in a first base layer of a first map 504 has a position (xl, yl, zl) and a pole in a second base layer of another map 504 has a position (x2, y2, z2), the pole can have a single final position in the fused base layer of the HD map as ((xl+x2) / 2, (yl+y2) / 2, (zl+z2) / 2). Furthermore, because the origin 652 of the road segment 610 to which the poles correspond can be known, the final pole position can be determined relative to the origin 652 of the road segment. This process can be repeated for all 3D landmarks in the base layer, and can be repeated for each other layer type (e.g., RADAR layer, LiDAR layer, etc.) of the maps 504.​​​

[0131] At block B714, the method 700 includes transmitting data representing the fused map to one or more vehicles for use in performing one or more operations. For example, map data 108 representing the final fused HD map can be transmitted to one or more vehicles 1500 for use in localization, path planning, control decisions, and / or other operations. For example, once localized to the fused HD map, information from the planning and control layers of the autonomous driving stack - such as lane graphs, wait conditions, static obstacles, etc. - can be obtained and provided to the planning and control subsystems of the autonomous vehicle 1500. In some embodiments, selected layers of the fused HD map can be transmitted to respective vehicles 1500 based on the configuration of the vehicle 1500. For example, where the vehicle 1500 does not have a LiDAR sensor, the LiDAR layer of the fused map can not be transmitted to the vehicle 1500 for storage and / or use. As a result of this customization approach, bandwidth requirements for transmitting the map data 108 and / or storage requirements of the vehicle 1500 for storing the map data 108 can be reduced.

[0132] localization

[0133] reference FIG. 8A , FIG. 8A A dataflow diagram depicting a localization process 110 according to some embodiments of the present disclosure is depicted. In some embodiments, the process 110 can be performed using a vehicle 1500. In some embodiments, one or more processes described with respect to the process 110 can be performed in parallel using one or more parallel processing units. For example, localization using different map layers can be performed in parallel with one or more other map layers. Within a single map layer, cost space sampling 802, cost space aggregation 804, and / or filtering 806 can be performed in parallel. For example, different poses can be sampled in parallel during cost space sampling 802 to more efficiently generate a cost space for a current frame or time step. Further, because map data corresponding to a map layer can be stored as a texture on a GPU, texture lookups can be performed to quickly determine cost values for a cost space - resulting in reduced runtimes for each cost space analysis. Further, as described herein, the localization process 110 - e.g., cost space sampling 802, cost space aggregation 804, and / or filtering 806 - can be performed during registration 402 in the map creation process 106. For example, the localization process 110 can be used to geometrically register poses from multiple pairs of segments, and cost space sampling 802, cost space aggregation 804, and / or filtering 806 can be used to align map layers from one segment to map layers from another segment.

[0134] The goal of the localization process 110 can be to locate the origin 820 of vehicle 1500 relative to the local origin 652 of the road segment 610 of the fused HD map represented by map data 108. For example, the localization process 110 can be used to determine the origin 820 of vehicle 1500 relative to the fused localization 1308 (as described herein) at specific time steps or frames. FIGS. 9A-9C (A more detailed description is needed) The corresponding ellipsoid. The origin 820 of vehicle 1500 may correspond to a reference point or origin of vehicle 1500, such as the center of the rear axle of vehicle 1500. Vehicle 1500 may be located relative to the origin 652 of road segment 610, and road segment 610 may have a corresponding position 824 in the global coordinate system. Thus, once vehicle 1500 is located to road segment 610 of the HD map, vehicle 1500 can also be globally located. In some embodiments, ellipsoid 910 may be determined for each individual sensor modality (e.g., LiDAR positioning, camera positioning, RADAR positioning, etc.), and the output of each positioning technique may be fused via positioning fusion 810. A final origin position—e.g., the origin 820 of vehicle 1500—may be generated and used as the positioning result of vehicle 1500 at the current frame or time step.

[0135] At the beginning of a drive, a current road segment 610 of the vehicle 1500 can be determined. In some embodiments, the current road segment 610 can be known from the last drive - for example, when the vehicle 1500 is turned off, the last known road segment that the vehicle 1500 was located on can be stored. In other embodiments, the current road segment 610 can be determined. To determine the current road segment 610, GNSS data can be used to globally locate the vehicle 1500 and then determine one or more road segments 610 that correspond to the global location result. In cases where the result returns two or more road segments 610, the road segment 610 with an origin 652 that is closest to the origin of the vehicle 1500 can be determined to be the current road segment 610. Once the current road segment 610 is determined, the road segment 610 can be determined to be a seed road segment for a breadth-first search. The breadth-first search can be performed to generate a local layout of road segments 610 that are adjacent to the current road segment 610 at a first level, then generate a local layout of road segments 610 that are adjacent to the road segments 610 at a second level from the first level, and so on. Understanding the road segments 610 that are adjacent to the current road segment 610 can be useful for the localization process 110 because when the vehicle 1500 moves from one road segment 610 to another road segment 610, the relative transformation between the road segments 610 can be used to update the sampled cost space generated for the previous road segment 610, which is used in the aggregated cost space for localization. Once the vehicle 1500 moves from the seed road segment to an adjacent road segment, another breadth-first search can be performed on the new road segment to generate an updated local layout, and this process can be repeated as the vehicle 1500 traverses the map from road segment to road segment. Further, as described herein, when the vehicle 1500 moves from one road segment 610 to another road segment 610, the previously calculated cost space (e.g., a certain number of previous cost spaces in a buffer, such as 50, 100, etc.) can be updated to reflect the same cost space, but relative to the origin 652 of the new road segment 610. Thus, the calculated cost space can be carried through the road segments to generate the aggregated cost space via the cost space aggregation 804.

[0136] The localization process 110 can localize the vehicle 1500 at each time step or frame using the sensor data 102 (e.g., real-time sensor data 102 generated by the vehicle 1500), the map data 108, and / or the output 204. For example, the sensor data 102, the output 204, and the map data 108 can be used to perform cost space sampling 802. The cost space sampling 802 can be different for different sensor modalities corresponding to different map layers. For example, the cost space sampling 802 can be performed separately for a LiDAR map layer (e.g., a LiDAR point cloud layer, a LiDAR map image layer, and / or a LiDAR voxel map layer), a RADAR map layer (e.g., a RADAR point cloud layer and / or a RADAR map image layer), and / or for a base layer (e.g., for landmark or camera-based map layers). Within each sensor modality, cost sampling can be performed using one or more different techniques, and the costs over the different techniques can be weighted to produce a final cost for the sampled poses. This process can be repeated for each pose in the cost space to produce a final cost space for the frame during localization. For example, a LiDAR intensity cost, a LiDAR height cost, and a LiDAR (slice) point cloud cost (e.g., using a distance function) can be computed, then averaged or otherwise weighted, and used for the final cost for a pose or point in the cost space. Similarly, for camera-based or landmark-based cost space sampling, a semantic cost can be computed, and a geometric cost (e.g., using a distance function) can be computed, then averaged or otherwise weighted, and used for the final cost for a pose or point in the cost space. As another example, a RADAR point cloud cost (e.g., using a distance function) can be computed and used for the final cost for a pose or point in the cost space. As such, the cost space sampling 802 can be performed to sample the cost for each different pose within the cost space. The cost space can correspond to a portion of the map that can include only a portion of the current road segment 610, the entire current road segment 610, some region of the current road segment 610 and one or more adjoining road segments 610, and / or some other region of the entire fused HD map. As such, the size of the cost space can be a programmable parameter of the system.

[0137] For any individual sensor modality, the result of the cost space sampling 802 can be a cost space representing the geometric match or likelihood that the vehicle 1500 can be localized relative to each particular pose. For example, a point in the cost space can have a corresponding relative position relative to the current road segment 610, and the cost space can indicate the likelihood or likelihood that the vehicle 1500 is currently in each particular pose (e.g., an (x, y, z) position relative to the origin of the road segment 610 and an axis angle about each of the x, y, and z axes).

[0138] Reference FIG. 9A, the cost space 902 can represent a cost space for a sensor modality - e.g., according to FIGS. 10A-10C a camera-based cost space generated. For example, the cost space 902 can represent a likelihood that the vehicle 1500 is currently at each of a plurality of poses (e.g., represented by points of the cost space 902) in a current frame. While represented in 2D in FIG. 9A , the cost space 902 can correspond to a 3D cost space (e.g., with (x, y, z) locations and / or axis angles for each of x, y, and z axes). As such, for each of the plurality of poses, the sensor data 102 (e.g., before or after pre-processing) and / or the output 204 (e.g., detection of landmark locations in a 2D image space and / or a 3D image space) can be compared to the map data 108. In cases where a pose does not match well with the map data 108, the cost can be higher, and a point in the cost space corresponding to the pose can be represented, e.g., in red, or represented relative to FIG. 9A , by a non-dashed or white portion. In cases where a pose matches well with the map data 108, the cost can be lower, and a point in the cost space corresponding to the pose can be represented, e.g., in green, or represented relative to FIG. 9A , by a dashed point. For example, referring to FIG. 9A , in cases where the cost space 902 corresponds to the visualization 1002 of FIG. 10A , the dashed portion 908 can correspond to low costs for poses along the diagonal, where the symbols 1010 can match well with predictions or outputs 204 of the DNN 202. For example, at a pose in the lower left of the dashed portion of the cost space, a prediction of a symbol can align well with the symbol 1010 from the map data 108, and similarly, at a pose in the upper right of the dashed portion, a prediction of a symbol from the respective pose can also align well with the symbol 1010. As such, these points can be represented locally with low cost. However, due to noise and a large number of low cost poses, a single cost space 902 can not be accurate for localization - e.g., the vehicle 1500 can not be located at each of the poses represented by the dashed portion 908. As such, an aggregated cost space 904 can be generated via cost space aggregation 804, as described herein.

[0139] Cost space sampling 802 can be performed separately for different sensor modalities, as described herein. For example, with respect to FIGS. 10A-10D, a camera or landmark-based cost space (e.g., corresponding to a base layer of a fused HD map) can be generated using a geometric cost and / or semantic cost analysis at each pose of the cost space. For example, at a given time step or frame, the output 204 - e.g., landmark locations of lane dividers, road boundaries, signs, poles, etc. - can be computed with respect to one or more images, such as the images represented in the visualization 1002. 3D landmark information from the map data 108 can be projected into 2D image space to correspond to a current prediction of the location of 3D landmarks in 2D image space with respect to the current pose of the vehicle 1500 sampled in the cost space. For example, the symbol 1010 can correspond to a 2D projection from the map data, and the symbol 1012 can correspond to a current prediction or output 204 from the one or more DNNs 202. Similarly, the lane divider 1014 can correspond to a 2D projection from the map data 108, and the lane divider 1016 can correspond to a current prediction or output 204 from the one or more DNNs 202. To compute a cost for the current pose (e.g., represented by the pose indicator 1018), the current output 204 from the DNNs 202 can be converted into a distance function corresponding to the predicted geometry represented in the visualization 1004 (e.g., where the prediction is divided into a plurality of points, and each point has a zero cost at its center, and the cost moves outward from the center until a maximum cost is reached, as represented by the white area of the visualization 1004), and the current output 204 can be separately converted into a semantic label for the prediction represented in the visualization 1006. Additionally, a 2D projection of the 3D landmarks can be projected into image space, and each point from the 2D projection can be compared to a portion of the distance function representation that the projected point lands on to determine an associated cost. As such, the dashed portion can correspond to a distance function representation of the current prediction of the one or more DNNs 202, and the dark solid line or points can represent the 2D projection from the map data 108. A cost can be computed for each point of the 2D projection, and an average cost can be determined using the relative costs from each point. For example, the cost at point 1020A can be high or maximum, and the cost at point 1020B can be low - e.g., because the cost at point 1020B is aligned with the center of the distance function representation of the lane divider. This cost can correspond to a geometric cost for the current pose of the current frame. Similarly, as represented in the visualization 1006, the cost at point 1022A can be high or maximum, and the cost at point 1022B can be low - e.g., because the cost at point 1022B is aligned with the center of the distance function representation of the lane divider. This cost can correspond to a semantic cost for the current pose of the current frame. FIG. 10CAs shown, the semantic label corresponding to the 2D projected point from the map data 108 can be compared to the projected semantic information. As such, in the event that the points do not match semantically, the cost can be set to a maximum, while in the event that the points match, the cost can be set to a minimum. These values can be averaged or otherwise weighted to determine a final semantic cost. The final semantic cost and the final geometric cost can be weighted to determine a final total cost for updating the cost space (e.g., cost space 902). For example, for each point, the semantic cost can have to be lower than the corresponding geometric cost with a vote. As such, in the event that the semantic information does not match, the cost for that particular point can be set to a maximum. In the event that the semantic information matches, for the semantic cost, the cost can be set to a minimum value or 0, and the final cost for that point can represent the geometric cost. Ultimately, the points in the cost space corresponding to the current pose can be updated to reflect the final costs for all of the 2D projected points.

[0140] As another example, with respect to FIGS. 11A-11B A RADAR-based cost space (e.g., a RADAR layer corresponding to the fused HD map) can be generated using a distance function corresponding to the map data 108 (e.g., a top-down projection of a RADAR point cloud, where each point is converted to a distance function representation). For example, at a given time step or frame, the map data 108 corresponding to a RADAR point cloud (as represented in visualization 1102) can be converted to a distance function (as represented in visualization 1104), for example, where each RADAR point can have a zero cost at its center, with the cost increasing to a maximum cost as the distance from the center increases. For example, with respect to visualization 1104, the white portions of the visualization 1102 can correspond to the maximum cost. To compute the cost for the current pose (e.g., represented by pose indicator 1106), the RADAR data from the sensor data 102 can be converted to a RADAR point cloud and compared to the distance function representation of the RADAR point cloud (e.g., as represented in visualization 1104). The hollow circles in visualizations 1102 and 1104 can correspond to the current RADAR point cloud prediction for the vehicle 1500. As such, for each current RADAR point, the cost can be determined by comparing each current RADAR point to the distance function RADAR value that the current RADAR point corresponds to or lands on. As such, current RADAR point 1108A can have a maximum cost, while current RADAR point 1108B can have a low cost - for example, because point 1108B lands close to the center of the point from the RADAR point cloud in the map data 108. Ultimately, an average or other weighting of each of the costs from the current RADAR points can be computed, and the final cost value can be used to update the cost space for the currently sampled pose.

[0141] As another example, with respect toFIGS. 12A-12D A LiDAR-based cost space (e.g., corresponding to a LiDAR layer of the fused HD map) can be generated using a distance function on (sliced) LiDAR point cloud, LiDAR intensity map, and / or LiDAR elevation map. For example, for a given pose as indicated by pose indicator 1210, current or real-time LiDAR data (e.g., corresponding to sensor data 102) can be generated and converted to values for comparison to an intensity map generated from map data 108 (e.g., as shown in visualization 1202), to values for comparison to an elevation map generated from map data 108 (e.g., as shown in visualization 1204), and a LiDAR point cloud of map data 108 (e.g., as shown in visualization 1206) can be converted to its distance function representation (e.g., as shown in visualization 1208) for comparison to a current LiDAR point cloud corresponding to sensor data 102. In embodiments, the LiDAR point cloud can correspond to a slice of the LiDAR point cloud, and one or more separate slices can be converted to distance function representations and used to compute a cost. The costs from the elevation comparison, intensity comparison, and distance function comparison can be averaged or otherwise weighted to determine a final cost on the LiDAR-based cost map corresponding to the current pose.

[0142] For example, with respect to FIG. 12A A LiDAR layer of the fused HD map represented by map data 108 can include a LiDAR intensity (or reflectivity) image (e.g., a top-down projection of intensity values from fused LiDAR data). For example, a painted surface (such as a lane marking) can have a higher reflectivity, and this reflected intensity can be captured and used to compare map data 108 to current LiDAR sensor data. Current LiDAR sensor data from vehicle 1500 can be converted to a LiDAR intensity representation 1212A and compared to a LiDAR intensity image from map data 108 at a current pose. For points of the current LiDAR intensity representation 1212A that have similar or matching intensity values to points from map data 108, the cost can be lower, and in the case of non-matching intensity values, the cost can be higher. For example, a zero difference in intensity of a point can correspond to a zero cost, a threshold difference and above can correspond to a maximum cost, and between the zero difference and the threshold difference, the cost can increase from zero cost to maximum cost. The cost of each point of the current LiDAR intensity representation 1212A can be averaged or otherwise weighted with each other point to determine a cost of the LiDAR intensity comparison.

[0143] As another example, with respect to FIG. 12BThe LiDAR layer of the fused HD map represented by map data 108 can include a LiDAR height image (e.g., a top-down projection of height values resulting in a top-down depth map). Current LiDAR sensor data from vehicle 1500 can be converted into a LiDAR height representation 1212B and compared to a LiDAR height image generated from map data 108 at the current pose. The cost can be lower for points of the current LiDAR height representation 1212B that have similar or matching height values to points from map data 108, and higher where the height values do not match. For example, zero difference in height of a point can correspond to zero cost, threshold difference and above can correspond to a maximum cost, and between zero difference and threshold difference, the cost can increase from zero cost to maximum cost. The cost of each point of the current LiDAR height representation 1212B can be averaged or otherwise weighted with each other point to determine the cost of the LiDAR height comparison.

[0144] In some embodiments, because the height values from map data 108 can be determined relative to the origin 652 of the current road segment 610, and the height values from the current LiDAR height representation 1212B can correspond to the origin 820 or reference point of vehicle 1500, a transformation can be performed to compare the values from LiDAR height representation 1212B to the LiDAR height image from map data 108. For example, where a point from LiDAR height representation 1212B has a height value of 1.0 meters (e.g., 1.0 meters high from the origin of vehicle 1500), the point from map data 108 corresponding to the point from representation 1212B has a value of 1.5 meters, and the height difference between the origin 820 of vehicle 1500 and the origin 652 of road segment 610 is 0.5 meters (e.g., the origin 652 of the road segment is 0.5 meters higher than the origin 820 of vehicle 1500), the actual difference between the point from map data 108 and representation 1212B can be 0.0 meters (e.g., 1.5 meters - 0.5 meters = 1 meter as the final value of the point from map data 108 relative to the origin 820 of vehicle 1500). According to this embodiment, the transformation between the values of map data 108 or representation 1212B can correspond to a transformation from the road segment origin 652 to the vehicle origin 820, from the vehicle origin 820 to the road segment origin 652, or a combination thereof.

[0145] As another example, with respect to FIGS. 12C-12DThe LiDAR layer of the fused HD map represented by map data 108 can include a sliced LiDAR point cloud (e.g., corresponding to a ground plane slice, a giraffe plane slice, another defined slice such as a 1 meter thick slice extending 2 meters to 3 meters from the ground plane, etc.). In some embodiments, the point cloud can not be sliced and can instead represent the entire point cloud. The sliced LiDAR point cloud (e.g., as shown in visualization 1206) can be converted to its distance function representation (e.g., as shown in visualization 1208). For example, each point from the LiDAR point cloud can be converted such that the center of the point has a zero cost and the cost increases as one gets farther away from the center of the point up to some maximum cost (e.g., as represented by the white area of visualization 1208). To compute the cost for the current pose (e.g., represented by pose indicator 1210), the LiDAR data from sensor data 102 can be converted to a LiDAR point cloud (or its corresponding slice) and compared to the distance function representation of the LiDAR point cloud (e.g., as represented in visualization 1208). The hollow circles in visualizations 1206 and 1208 can correspond to the current LiDAR point cloud prediction of vehicle 1500. Thus, for each current LiDAR point, the cost can be determined by comparing each current LiDAR point to the distance function LiDAR value that the current LiDAR point corresponds to or lands on. Thus, for example, because point 1214B lands close to the center of the points from the LiDAR point cloud in map data 108, current LiDAR point 1214A can have a maximum cost and current LiDAR point 1214B can have a low cost. Ultimately, an average or other weighting of each cost from the current LiDAR points can be computed and the final cost value can be used to update the cost space for the current sampled pose in addition to the cost values from the height and intensity comparisons.

[0146] In some embodiments, at least one of LiDAR-based cost space sampling 802, cost space aggregation 804, and / or filtering 806 can be performed on a GPU (e.g., a discrete GPU, a virtual GPU, etc.) and / or using one or more parallel processing units. For example, for camera-based cost spaces (e.g., with respect to FIGS. 10A-10CThe described), the detection information and map data 108 projection can be stored as textures in memory on the GPU or in memory accessible to the GPU, and the comparison can correspond to a texture lookup performed using the GPU. Similarly, with respect to LiDAR and / or RADAR, the comparison can correspond to a texture lookup. Further, in some embodiments, parallel processing can be used to perform two or more cost spaces in parallel - for example, a first cost space corresponding to LiDAR and a second cost space corresponding to RADAR can be generated in parallel using different GPU and / or parallel processing unit resources. For example, individual localizations 808 can be computed in parallel in order to reduce the runtime of the system for fusing localizations. As a result, these processes can be performed more efficiently than if performed individually on a CPU.

[0147] Referring again to FIG. 8A Cost space aggregation 804 can be performed after cost space sampling 802 is performed for a single frame or time step and for any number of sensor modalities. Cost space aggregation 804 can be performed individually for each sensor modality - for example, LiDAR-based cost space aggregation, RADAR-based cost space aggregation, camera-based cost space aggregation, etc. For example, cost space aggregation 804 can aggregate cost spaces computed for any number of frames (e.g., 25 frames, 80 frames, 100 frames, 300 frames, etc.). To aggregate cost spaces, each previous cost space that has been computed can be self-motion compensated to correspond to the current frame. For example, a rotation and / or translation of the vehicle 1500 relative to the current pose of the vehicle 1500 and a rotation and / or translation from each previous frame included in the aggregation can be determined, and the rotation and / or translation used to carry the cost space values from the previous frame. In addition to transforming previous cost spaces based on self-motion, cost spaces can also be transformed using, for example, transformations from one road segment to the next, such as relative to FIG. 6I The described, so that each cost space corresponds to an origin 652 of a current road segment 610 of the fused HD map. For example, when the vehicle 1500 is localized relative to a first road segment 610, some number of cost spaces can have been generated to be aggregated, and when the vehicle 1500 is localized relative to a second road segment 610, some other number of cost spaces can have been generated to be aggregated. Thus, cost spaces from the first or previous road segment 610 can be transformed so that the cost space values are relative to the second or current road segment 610. Once in the same frame of reference as the current frame and the current road segment 610, the cost spaces can be aggregated. Thus, and referring to FIG. 9B , self-motion of the vehicle 1500 over time can help disambiguate individual cost spaces, so that a total cost space 904 can be generated.

[0148] The aggregated cost space 904 can then undergo filtering 806 (e.g., using a Kalman filter or another filter type) to determine an ellipsoid 910 corresponding to the computed position of the vehicle 1500 relative to the current road segment 610. Similar to the description above regarding the transformation for the aggregated cost space 904, the filtered cost space 906 can also undergo a transformation to compensate for ego motion and road segment switching. The ellipsoid 910 can indicate a current position relative to the current road segment 610 of the vehicle 1500, and the process can be repeated at each new frame. The result can be individual localization based on the sensor modalities for which ellipsoids 910 are computed, and multiple ellipsoids 910 can be computed at each frame, e.g., one for each sensor modality.

[0149] Localization fusion 810 can then be performed on the individual localizations 808 to generate a final localization result. For example, with reference to FIG. 13 , the individual localizations 808 can correspond to a LiDAR-based localization 1302 (e.g., represented by the ellipsoid and origin in the visualization 1300), a RADAR-based localization 1304, a camera-based localization 1306, other sensor modality localizations (not shown), and / or a fused localization 1308. While only a single localization per sensor modality is described herein, this is not intended to be limiting. In some embodiments, there can be more than one localization result for different sensor modalities. For example, the vehicle 1500 can be localized relative to a first camera (e.g., a forward-facing camera) and can be separately localized relative to a second camera (e.g., a rear-facing camera). In this example, the individual localizations 808 can include a first camera-based localization and a second camera-based localization. In some embodiments, the fused localization 1308 can correspond to a fusion of the individual localizations 808 at the current frame and / or can correspond to one or more previous fused localization results from one or more previous frames carried forward (e.g., based on ego motion) to the current frame. As such, in embodiments, the fused localization 1308 of the current frame can consider the individual localizations 808 and previous fused localization results to advance the current localization state through the frames.

[0150] To compute the fused position 1308 for the current frame, a consensus / non-consensus analysis can be performed on the individual positions 808. For example, in some embodiments, a distance threshold can be used to determine clusters 808 of individual positions, and the cluster with the smallest intra-cluster covariance can be selected for fusion. The individual positions 808 within the selected cluster can then be averaged or otherwise weighted to determine the fused position 1308 for the current frame. In some embodiments, a filter such as a Kalman filter can be used to generate a fused position 1308 for the clustered individual positions 808 for the current frame. For example, where a Kalman filter is used, the Kalman filter can not handle outliers well, so severe outliers can have an undesired impact on the final result. As such, the clustering approach can help filter out or remove outliers, making the Kalman filter-based fusion more accurate. In some embodiments, such as where a previous fused result is carried over as an individual position 808 to the current frame, the fused result from the previous frame can drift. For example, once the current individual position 808 is sufficiently different from the fused result (e.g., where the fused result can be filtered out of a cluster), the fused result can be reinitialized for the current frame, and then the reinitialized fused result can be carried over to subsequent frames until a certain amount of drift is detected again.

[0151] In some embodiments, the fused position 1308 can be determined by decomposing factors in each of the individual positions 808. For example, instead of grouping results into clusters, each individual position 808 can be weighted based on a distance assessment. For example, a covariance can be computed for the individual positions 808, and the individual position 808 with the highest covariance (e.g., corresponding to the largest outlier) can be weighted less to determine the fused position 1308. This can be performed using a robust mean, such that outliers have an undesired impact on the fused position 1308. For example, a distance of each individual position 808 to the robust mean can be computed, and the greater the distance, the less weight the individual position 808 can have in determining the fused position 1308.

[0152] The fused position 1308 for the current frame can then be used to localize the vehicle 1500 relative to the road segments 610 of the fused HD map (represented by the map data 108) and / or relative to a global coordinate system. For example, because the road segments 610 can have known global positions, localization of the vehicle 1500 to the road segments 610 can have a corresponding global localization result. Moreover, because local or relative localization to the road segments 610 is more accurate than a standalone global or GNSS localization result, planning and control of the vehicle 1500 can be more reliable and safer than a standalone GNSS-based localization system.

[0153] Reference is now made to FIG. 14Each block of the method 1400 described herein comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 1400 can also be embodied as computer-usable instructions stored on computer storage media. The method 1400 can be provided by a standalone application, a service or hosted service, an add-in to another product, or a plug-in to another product, to name a few possibilities. In addition, a variety of implementations can be made to the process 110 described with respect to FIG. 8A The method 1400 is described with respect to the process 110. However, this method 1400 can additionally or alternatively be performed by any one system or any combination of systems within any one process, including but not limited to those described herein.

[0154] FIG. 14 is a flowchart illustrating a method 1400 for localization according to some embodiments of the present disclosure. At block B1402, the method 1400 includes generating, at each frame of a plurality of frames, one or more of sensor data or DNN outputs based on the sensor data. For example, at each frame or time step of a buffered number of frames (e.g., 25, 50, 70, 100, etc.), the sensor data 102 and / or outputs 204 can be computed. For example, in the case of LiDAR-based localization or RADAR-based localization, the sensor data 102 can be generated and / or processed at each frame or time step (e.g., to generate a height representation, an intensity representation, etc.). In the case of camera-based localization, the sensor data 102 can be applied to one or more DNNs 202 to generate outputs 204 corresponding to landmarks.

[0155] At block B1404, the method 1400 includes comparing, at each frame, the one or more of sensor data or DNN outputs to map data to generate a cost space representing probabilities that the vehicle is at each of a plurality of poses. For example, cost space sampling 802 can be performed at each frame to generate a cost space corresponding to a particular sensor data modality. The comparison between the map data 108 and the sensor data 102 and / or outputs 204 can include comparing LiDAR height information, LiDAR intensity information, LiDAR point cloud (slice) information, RADAR point cloud information, camera landmark information (e.g., comparing 2D projections of 3D landmark locations from the map data 108 to current real-time predictions of one or more DNNs 202), etc.

[0156] At block B1406, the method 1400 includes aggregating each cost space from each frame to generate an aggregated cost space. For example, the cost space aggregation 804 can be performed on some number of buffered or previous cost spaces to generate an aggregated cost space. The aggregation can include a self-motion transform of previous cost spaces generated with respect to road segments 610 other than the current road segment 610 of the vehicle 1500 and / or a road segment transform of previous cost spaces.

[0157] At block B1408, the method 1400 includes applying a Kalman filter to the aggregated cost space to compute a final position of the vehicle at a current frame of the plurality of frames. For example, the filtering 806 can be applied to the aggregated cost space to generate an ellipsoid or other representation of the estimated position of the vehicle 1500 at the current frame or time step with respect to a particular sensor modality.

[0158] In embodiments, the blocks B1402-B1408 of the process 1408 can be repeated in parallel for any number of different sensor modalities, resulting in the generation of two or more ellipsoids or position predictions of the vehicle 1500.

[0159] At block B1410, the method 1400 includes determining a fused position of the vehicle using the final position in addition to one or more other final positions of the vehicle. For example, the ellipsoids of the other representations output after filtering 806 for different sensor modalities can undergo localization fusion 810 to generate a final fused localization result. In some embodiments, as described herein, a previous fused localization result from a previous frame or time step can also be used when determining the current fused localization result.

[0160] Example autonomous vehicle

[0161] FIG. 15Ais a diagram of an example autonomous vehicle 1500 according to some embodiments of the present disclosure. The autonomous vehicle 1500 (alternatively referred to herein as “vehicle 1500”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, first responder vehicle, shuttle, electric or motorized bicycle, motorcycle, fire vehicle, police vehicle, ambulance, boat, construction vehicle, underwater vessel, drone, and / or another type of vehicle (e.g., a vehicle that is unmanned and / or that accommodates one or more passengers). Autonomous vehicles are often described in terms of levels of automation as defined by a department of the United States Department of Transportation, the National Highway Traffic Safety Administration (NHTSA), and the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). The vehicle 1500 can be capable of implementing functionality that complies with one or more of Levels 3-5 of autonomous driving. For example, depending on the embodiment, the vehicle 1500 can be capable of implementing conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0162] The vehicle 1500 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 1500 can include a propulsion system 1550 such as an internal combustion engine, a hybrid power plant, a fully electric motor, and / or another type of propulsion system. The propulsion system 1550 can be connected to a drivetrain of the vehicle 1500 that can include a transmission in order to effect propulsion of the vehicle 1500. The propulsion system 1550 can be controlled in response to receiving a signal from a throttle / accelerator 1552.

[0163] A steering system 1554, which can include a steering wheel, can be used to steer the vehicle 1500 (e.g., along a desired path or route) while the propulsion system 1550 is operating (e.g., while the vehicle is in motion). The steering system 1554 can receive a signal from a steering actuator 1556. For full automation (Level 5) functionality, the steering wheel can be optional.

[0164] A braking sensor system 1546 can be used to operate vehicle brakes in response to receiving a signal from a braking actuator 1548 and / or a braking sensor.

[0165] One or more controllers 1536 of one or more SoCs 1504 FIG. 15C ) and / or one or more GPUs can provide signals (e.g., representative of commands) to one or more components and / or systems of vehicle 1500. For example, the one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 1548, to operate a steering system 1554 via one or more steering actuators 1556, to operate a propulsion system 1550 via one or more throttle / accelerator 1552. The one or more controllers 1536 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representative of commands) to enable autonomous driving and / or to assist a human driver in driving vehicle 1500. The one or more controllers 1536 can include a first controller 1536 for autonomous driving functions, a second controller 1536 for functional safety functions, a third controller 1536 for artificial intelligence functions (e.g., computer vision), a fourth controller 1536 for infotainment functions, a fifth controller 1536 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1536 can handle two or more of the above functions, two or more controllers 1536 can handle a single function, and / or any combination thereof.

[0166] The one or more controllers 1536 can provide signals for controlling one or more components and / or systems of vehicle 1500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, global navigation satellite system sensors 1558 (e.g., global positioning system sensors), RADAR sensors 1560, ultrasonic sensors 1562, LIDAR sensors 1564, inertial measurement unit (IMU) sensors 1566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1596, stereo cameras 1568, wide-angle cameras 1570 (e.g., fisheye cameras), infrared cameras 1572, surround cameras 1574 (e.g., 360 degree cameras), long and / or medium range cameras 1598, speed sensors 1544 (e.g., to measure a speed of vehicle 1500), vibration sensors 1542, steering sensors 1540, brake sensors (e.g., as part of brake sensor system 1546), and / or other sensor types.

[0167] One or more of the controllers 1536 can receive input from the instrument cluster 1532 of the vehicle 1500 (e.g., represented by input data) and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1534, audible annunciators, speakers, and / or via other components of the vehicle 1500. These outputs can include information such as vehicle speed, velocity, time, map data (e.g., HD map 1522), location data (e.g., location of the vehicle 1500, e.g., on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and object states as perceived by the controllers 1536, and so on. For example, the HMI display 1534 can display information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, exit 34B in two miles, etc.). FIG. 15C

[0168] The vehicle 1500 also includes a network interface 1524 that can communicate over one or more networks using one or more wireless antennas 1526 and / or modems. For example, the network interface 1524 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The one or more wireless antennas 1526 can also enable communication between objects (e.g., vehicles, mobile devices, etc.) in the environment using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc. and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0169] FIG. 15B FIG. 15 illustrates an example autonomous vehicle 1500 for use with some embodiments of the present disclosure. FIG. 15A FIG. 16 illustrates example camera positions and fields of view for the example autonomous vehicle 1500 of FIG. 15. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included, and / or these cameras can be located at different positions on the vehicle 1500.

[0170] ​Camera types for the cameras can include, but are not limited to, digital cameras that can be adapted for use with components and / or systems of the vehicle 1500. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, and so on, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array can include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with a CCC, RCCB, and / or RBGC color filter array, can be used in efforts to improve light sensitivity.

[0171] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more (e.g., all) of the cameras can simultaneously record and provide image data (e.g., video).

[0172] One or more of the cameras can be installed in mounting assemblies, such as custom designed (3-D printed) assemblies, to cut off stray light and reflections from within the car that can interfere with the image data capture capabilities of the cameras (e.g., reflections from the dashboard reflected in the windshield mirror). With regard to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3-D printed such that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0173] Cameras with a field of view that includes an environmental portion in front of the vehicle 1500 (e.g., front-facing cameras) can be used for surround view to help identify a forward path and obstacles, and to assist in providing information that is critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 1536 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems, including lane departure warning (“LDW”), adaptive cruise control (“ACC”), and / or other functions such as traffic sign recognition.

[0174] A variety of cameras can be used in front-facing configurations, including monocular camera platforms such as CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 1570, which can be used to perceive objects entering the field of view from the periphery (e.g., pedestrians, traffic at intersections, or bicycles). Although FIG. 15B The middle image shows only one wide-angle camera, but any number of wide-angle cameras 1570 can be present on the vehicle 1500. Furthermore, remote cameras 1598 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which neural networks have not yet been trained. Remote cameras 1598 can also be used for object detection and classification, as well as basic object tracking.

[0175] One or more stereo cameras 1568 may also be included in a front-mounted configuration. The stereo camera 1568 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1568 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1568 may be used in addition to those described herein, or alternatively.

[0176] Cameras with a field of view including the side portion of the vehicle 1500 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 1574 (e.g., ... FIG. 15B The four surround cameras 1574 shown can be mounted on vehicle 1500. The surround cameras 1574 can include a wide-angle camera 1570, a fisheye camera, a 360-degree camera, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1574 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround-view camera.

[0177] Cameras with fields of view that include portions of the environment behind the vehicle 1500, such as rearview cameras, can be used to assist with parking, surround view, rear collision warning, and creating and updating the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long and / or mid-range cameras 1598, stereo cameras 1568, infrared cameras 1572, etc.) as described herein.

[0178] FIG. 15C FIG. 1 illustrates an example autonomous vehicle 1500 in accordance with some embodiments of the present disclosure. FIG. 15A FIG. 1 illustrates an example autonomous vehicle 1500 in accordance with some embodiments of the present disclosure.

[0179] FIG. 15C Each of the components, features, and systems of the vehicle 1500 are illustrated as being connected via a bus 1502. The bus 1502 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1500 that is used to assist in controlling various features and functions of the vehicle 1500, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0180] Although bus 1502 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 1502 is represented with a single line, this is not intended to be limiting. For example, there can be any number of buses 1502, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 1502 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 1502 can be used for collision avoidance functions, and a second bus 1502 can be used for drive control. In any example, each bus 1502 can communicate with any component of vehicle 1500, and two or more buses 1502 can communicate with the same components. In some examples, each SoC 1504, each controller 1536, and / or each computer within the vehicle can have access to the same input data (e.g., input from sensors of vehicle 1500), and can be connected to a common bus, such as a CAN bus.

[0181] Vehicle 1500 can include one or more controllers 1536, such as those described herein with respect to FIG. 15A controllers described herein. Controllers 1536 can be used for a wide variety of functions. Controllers 1536 can be coupled to any other different components and systems of vehicle 1500, and can be used for control of vehicle 1500, artificial intelligence of vehicle 1500, infotainment for vehicle 1500, and / or the like.

[0182] Vehicle 1500 can include one or more system on chips (SoCs) 1504. SoCs 1504 can include CPUs 1506, GPUs 1508, processors 1510, caches 1512, accelerators 1514, data stores 1516, and / or other components and features not illustrated. In a wide variety of platforms and systems, SoCs 1504 can be used to control vehicle 1500. For example, one or more SoCs 1504 can be used in a system (e.g., a system of vehicle 1500) in conjunction with an HD map 1522, which can obtain map refreshes and / or updates from one or more servers (e.g., one or more servers 1578) via a network interface 1524. FIG. 15D

[0183] ​CPU 1506 can include a CPU cluster or CPU complex (alternatively referred to herein as a "CCPLEX"). CPU 1506 can include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1506 can include eight cores in a coherent multi-processor configuration. In some embodiments, CPU 1506 can include four dual-core clusters, with each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). CPU 1506 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1506 can be active at any given time.

[0184] CPU 1506 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. CPU 1506 can further implement enhanced algorithms for managing power states, with specified allowed power states and desired wake-up times, and the hardware / microcode determines the best power state for the cores, clusters, and CCPLEX to enter. The processing cores can support a simplified power state entry sequence in software, with the work offloaded to microcode.

[0185] GPU 1508 can include an integrated GPU (alternatively referred to herein as an "iGPU"). GPU 1508 can be programmable and efficient for parallel workloads. In some examples, GPU 1508 can use an enhanced tensor instruction set. GPU 1508 can include one or more streaming microprocessors, where each streaming microprocessor can include an Ll cache (e.g., an Ll cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 1508 can include at least eight streaming microprocessors. GPU 1508 can use a compute application programming interface (API). In addition, GPU 1508 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0186] In the case of automotive and embedded uses, the GPU 1508 can be power-optimized for best performance. For example, the GPU 1508 can be fabricated on a fin field-effect transistor (FinFET) for this purpose. This is, however, not intended to be limiting, and the GPU 1508 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor Cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. Further, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and address computation for efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for finer-grain synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0187] The GPU 1508 can include a high bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, synchronous graphics random access memory (SGRAM) can be used, such as fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0188] The GPU 1508 can include a unified memory technology that includes an access counter to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow the GPU 1508 to access CPU 1506 page tables directly. In such examples, when the GPU 1508 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to the CPU 1506. In response, the CPU 1506 can look up the virtual-to-physical mapping for the address in its page tables and transmit the translation back to the GPU 1508. In this way, the unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1506 and the GPU 1508, simplifying GPU 1508 programming and porting applications to the GPU 1508.

[0189] In addition, GPU 1508 can include an access counter that can track how often GPU 1508 accesses memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that most frequently accesses those pages.

[0190] SoC 1504 can include any number of caches 1512, including those described herein. For example, caches 1512 can include an L3 cache that is available to both CPU 1506 and GPU 1508 (e.g., connected to both CPU 1506 and GPU 1508). Caches 1512 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0191] SoC 1504 can include an arithmetic logic unit (ALU) that can be utilized in processing to perform any of a variety of tasks or operations with respect to vehicle 1500, such as processing a DNN. In addition, SoC 1504 can include a floating point unit (FPU) (or other mathematical co-processor or digital co-processor type) for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 1506 and / or GPU 1508.

[0192] SoC 1504 can include one or more accelerators 1514 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1504 can include a hardware accelerator cluster that can include optimized hardware accelerators and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to supplement GPU 1508 and offload some of the tasks of GPU 1508 (e.g., freeing up more cycles of GPU 1508 for performing other tasks). As one example, accelerators 1514 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to accelerate easily. As used herein, the term “CNN” can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0193] The accelerators 1514 (e.g., a cluster of hardware accelerators) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPUs can be accelerators that are configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and are optimized for performing image processing functions. The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The design of the DLA can provide higher performance per mm than general purpose GPUs and far exceeds the performance of CPUs. The TPUs can perform several functions including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0194] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not by way of limitation: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and identification and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for safety and / or safety related events.

[0195] The DLA can perform any of the functions of the GPU 1508, and by using an inferencing accelerator, the designer can target the DLA or the GPU 1508 for any function, for example. For example, the designer can focus the processing and floating point operations of CNNs on the DLA and leave other functions to the GPU 1508 and / or other accelerators 1514.

[0196] The accelerators 1514 (e.g., a cluster of hardware accelerators) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example and not by way of limitation.

[0197] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of these RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include instruction caches and / or tightly coupled RAM.

[0198] The DMA can enable components of the PVA to access system memory independently of the CPU 1506. The DMA can support any number of features to provide optimization to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0199] The vector processor can be a programmable processor that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0200] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in a hardware accelerator cluster, and any number of vector processors can be included in each of those PVAs, among other things. Furthermore, the PVAs can include additional error-correcting code (ECC) memory to enhance overall system security.

[0201] The accelerator 1514 (e.g., hardware accelerator cluster) can include on-chip computer vision networks and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1514. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include on-chip computer vision networks that interconnect the PVA and the DLA to the memory (e.g., using APB).

[0202] The on-chip computer vision networks can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0203] In some examples, the SoC 1504 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison to LIDAR data for purposes of localization and / or other functionality, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0204] The accelerator 1514 (e.g., a hardware accelerator cluster) has a wide range of uses for autonomous driving. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0205] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0206] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0207] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections and not false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold as true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1566 outputs related to vehicle 1500 orientation, distance, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LIDAR sensor 1564 or RADAR sensor 1560), etc.

[0208] SoC 1504 can include one or more data stores 1516 (e.g., memory). Data stores 1516 can be on-chip memory of SoC 1504, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 1516 can be large enough in capacity to store multiple instances of a neural network. Data stores 1512 can include L2 or L3 cache 1512. References to data stores 1516 can include references to memory associated with PVA, DLA, and / or other accelerators 1514 as described herein.

[0209] SoC 1504 can include one or more processors 1510 (e.g., embedded processors). The processors 1510 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions, as well as security implementation. The boot and power management processor can be part of the SoC 1504 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 1504 thermal and temperature sensor management, and / or SoC 1504 power state management. Each temperature sensor can be implemented as a ring oscillator, whose output frequency is proportional to temperature, and the SoC 1504 can use the ring oscillator to detect the temperature of the CPU 1506, GPU 1508, and / or accelerators 1514. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 1504 in a lower power state and / or place the vehicle 1500 in a driver safe park mode (e.g., safely park the vehicle 1500).

[0210] The processors 1510 can also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces, as well as a range of extensive and flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0211] The processors 1510 can also include an always-on processor engine, which can provide the necessary hardware features to support low power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0212] The processors 1510 can also include a security cluster engine, which includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0213] The processors 1510 can also include a real-time camera engine, which can include a specialized processor subsystem for handling real-time camera management.

[0214] The processor 1510 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0215] The processor 1510 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 1570, surround camera 1574, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0216] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0217] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 1508 does not need to continuously render new surfaces. Even when the GPU 1508 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 1508 to improve performance and responsiveness.

[0218] The SoC 1504 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 1504 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0219] The SoC 1504 can also include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1504 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor 1564, RADAR sensor 1560, etc. that can be connected over Ethernet), data from the bus 1502 (e.g., speed of the vehicle 1500, steering wheel position, etc.), data from GNSS sensor 1558 (connected over Ethernet or CAN bus). The SoC 1504 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines, and which can be used to free up the CPU 1506 from routine data management tasks.

[0220] The SoC 1504 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 1504 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerators 1514, when combined with the CPU 1506, GPU 1508, and data storage 1516, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0221] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of vision data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and for practical level 3-5 autonomous vehicles.

[0222] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a cluster of hardware accelerators, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1520) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can also include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0223] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Caution: flashing lights indicate icy conditions" along with electric lights can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 1508.

[0224] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensors to recognize the presence of an authorized driver and / or owner of the vehicle 1500. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1504 provides security against theft and / or carjacking.

[0225] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphones 1596 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, the SoC 1504 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 1558. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine to slow the vehicle, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensors 1562.

[0226] The vehicle can include a CPU 1518 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1504 via a high-speed interconnect (e.g., PCIe). The CPU 1518 can include, for example, an X86 processor. The CPU 1518 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 1504, and / or monitoring the status and health of the controller 1536 and / or infotainment SoC 1530.

[0227] The vehicle 1500 can include a GPU 1520 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 1504 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 1520 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on input (e.g., sensor data) from sensors of the vehicle 1500.

[0228] The vehicle 1500 can also include a network interface 1524, which can include one or more wireless antennas 1526 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 1524 can be used to enable wireless connections over the Internet with a cloud (e.g., with a server 1578 and / or other network devices), with other vehicles, and / or with computing devices (e.g., a client device of a passenger). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 1500 with information about vehicles that are approaching the vehicle 1500 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1500). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 1500.

[0229] The network interface 1524 can include a SoC that provides modulation and demodulation functionality and enables the controller 1536 to communicate over a wireless network. The network interface 1524 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0230] The vehicle 1500 can also include a data store 1528, which can include off-chip (e.g., off-SoC 1504) storage. The data store 1528 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0231] The vehicle 1500 can also include a GNSS sensor 1558. The GNSS sensor 1558 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1558 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0232] The vehicle 1500 can also include a RADAR sensor 1560. The RADAR sensor 1560 can be used by the vehicle 1500 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 1560 can use the CAN and / or the bus 1502 (e.g., to transmit data generated by the RADAR sensor 1560) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 1560 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0233] The RADAR sensor 1560 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) achieved through two or more independent scans. The RADAR sensor 1560 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as a high-speed CAN and FlexRay interface. In examples with six antennas, the central four antennas can create a focused beam pattern designed to record the surroundings of the vehicle 1500 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1500.

[0234] As one example, a mid-range RADAR system can include a range of up to 1560 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1550 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and the blind spot next to the vehicle.

[0235] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0236] The vehicle 1500 can also include ultrasonic sensors 1562. Ultrasonic sensors 1562 that can be placed on the front, rear, and / or sides of the vehicle 1500 can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 1562 can be used, and different ultrasonic sensors 1562 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 1562 can operate at an ASIL B functional safety level.

[0237] The vehicle 1500 can include LIDAR sensors 1564. The LIDAR sensors 1564 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 1564 can be at an ASIL B functional safety level. In some examples, the vehicle 1500 can include multiple LIDAR sensors 1564 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0238] In some examples, the LIDAR sensors 1564 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 1564 can have, for example, an advertised range of approximately 1500 m, a precision of 2 cm - 3 cm, and support for a 1500 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 1564 can be used. In such examples, the LIDAR sensors 1564 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 1500. In such examples, the LIDAR sensors 1564 can provide a field of view of up to 120 degrees horizontal and 35 degrees vertical for low reflectivity objects, with a range of 200 m. Front-mounted LIDAR sensors 1564 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0239] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 1500. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than fans. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1564 can be less susceptible to motion blur, vibration, and / or jostling.

[0240] The vehicle can also include an IMU sensor 1566. In some examples, the IMU sensor 1566 can be located at the center of the rear axle of the vehicle 1500. The IMU sensor 1566 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor 1566 can include an accelerometer and a gyroscope, while in nine-axis applications, the IMU sensor 1566 can include an accelerometer, a gyroscope, and a magnetometer.

[0241] In some embodiments, the IMU sensor 1566 can be implemented as a microelectromechanical systems (MEMS) based inertial navigation system (INS) that combines a microelectromechanical systems (MEMS) inertial sensor, a high-sensitivity GPS receiver, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1566 can enable the vehicle 1500 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 1566 and correlating it. In some examples, the IMU sensor 1566 and the GNSS sensor 1558 can be combined into a single integrated unit.

[0242] The vehicle can include a microphone 1596 placed in and / or around the vehicle 1500. The microphone 1596 can be used for emergency vehicle detection and identification, among other things.

[0243] The vehicle can also include any number of camera types, including stereo cameras 1568, wide-view cameras 1570, infrared cameras 1572, surround-view cameras 1574, long and / or mid-range cameras 1598, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 1500. The types of cameras used depend on the embodiment and requirements of the vehicle 1500, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1500. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and not by way of limitation, the cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to FIG. 15A and FIG. 15B are described in more detail.

[0244] The vehicle 1500 can also include vibration sensors 1542. The vibration sensors 1542 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1542 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0245] The vehicle 1500 can include an ADAS system 1538. In some examples, the ADAS system 1538 can include a SoC. The ADAS system 1538 can include adaptive / autonomous / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0246] The ACC system can use RADAR sensors 1560, LIDAR sensors 1564, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 1500 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 1500. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0247] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link via network interface 1524 and / or wireless antenna 1526 or indirectly through a network connection, such as through the Internet. The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about the immediately preceding vehicle, such as a vehicle immediately ahead of and in the same lane as vehicle 1500, while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of the I2V and V2V information sources. Given information about vehicles ahead of vehicle 1500, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0248] An FCW system is designed to alert the driver to a hazard so that the driver can take corrective action. An FCW system uses a front-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component. An FCW system can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0249] An AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. An AEB system can use a front-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. An AEB system can include technologies such as dynamic brake support and / or crash imminent braking.

[0250] An LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when vehicle 1500 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. An LDW system can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0251] An LKA system is a variation of the LDW system. If the vehicle 1500 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 1500.

[0252] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0253] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 1500 is backing up. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0254] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 1500, in the case of conflicting results, the vehicle 1500 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1536 or the second controller 1536). For example, in some embodiments, the ADAS system 1538 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1538 can be provided to a supervisory MCU. If the outputs from the primary and secondary computers conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0255] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0256] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based at least in part on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running a neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 1504.

[0257] In other examples, the ADAS system 1538 can include a secondary computer that performs ADAS functions using traditional computer vision rules. In this way, the secondary computer can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0258] In some examples, the output of the ADAS system 1538 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 1538 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0259] The vehicle 1500 can also include an infotainment SoC 1530 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 1530 can include a combination of hardware and software that can be used to provide audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.), and / or information services (e.g., a navigation system, a park assist, a radio data system, vehicle-related information such as fuel level, total distance covered, brake fuel level, oil level, doors open / closed, air filter information, etc.) to the vehicle 1500. For example, the infotainment SoC 1530 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), an HMI display 1534, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1530 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 1538, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0260] The infotainment SoC 1530 can include GPU functionality. The infotainment SoC 1530 can communicate with other devices, systems, and / or components of the vehicle 1500 over a bus 1502 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1530 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 1536 (e.g., a primary and / or backup computer of the vehicle 1500), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 1530 can place the vehicle 1500 in a driver safe park mode as described herein.

[0261] The vehicle 1500 can also include an instrument cluster 1532 (e.g., a digital dashboard, electronic instrument cluster, digital instrument panel, etc.). The instrument cluster 1532 can include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 1532 can include a set of instruments, such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn indicators, shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, supplemental restraint system (SRS) system information, lighting controls, safety system controls, navigation information, and so forth. In some examples, information can be displayed and / or shared between the infotainment SoC 1530 and the instrument cluster 1532. In other words, the instrument cluster 1532 can be included as part of the infotainment SoC 1530, or vice versa.

[0262] FIG. 15D FIG. 15 illustrates a system diagram of communication between a cloud-based server and an example autonomous vehicle 1500 in accordance with some embodiments of the present disclosure. FIG. 15A The system 1576 can include servers 1578, a network 1590, and vehicles including the vehicle 1500. The servers 1578 can include a plurality of GPUs 1584(A)- 1584(H) (collectively referred to herein as GPUs 1584), PCIe switches 1582(A)- 1582(H) (collectively referred to herein as PCIe switches 1582), and / or CPUs 1580(A)- 1580(B) (collectively referred to herein as CPUs 1580). The GPUs 1584, CPUs 1580, and PCIe switches can be interconnected with high-speed interconnects such as, for example and without limitation, NVLink interfaces 1588 developed by NVIDIA and / or PCIe connections 1586. In some examples, the GPUs 1584 are connected via NVLink and / or NVSwitch SoC connections, and the GPUs 1584 and PCIe switches 1582 are connected via PCIe interconnects. Although eight GPUs 1584, two CPUs 1580, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1578 can include any number of GPUs 1584, CPUs 1580, and / or PCIe switches. For example, each of the servers 1578 can include eight, sixteen, thirty-two, and / or more GPUs 1584.

[0263] The server 1578 can receive image data from vehicles over the network 1590 and representing images showing unexpected or changing road conditions such as a road work that has recently started. The server 1578 can transmit neural networks 1592, updated neural networks 1592, and / or map information 1594, including information about traffic and road conditions, to vehicles over the network 1590. Updates to the map information 1594 can include updates to the HD map 1522, e.g., information about construction sites, potholes, curves, flooding, or other obstacles. In some examples, the neural networks 1592, updated neural networks 1592, and / or map information 1594 can have been produced from experience using training performed at a data center (e.g., using the server 1578 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0264] The server 1578 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to categories such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 1590), and / or the machine learning models can be used by the server 1578 to remotely monitor vehicles.

[0265] In some examples, the server 1578 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 1578 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 1584, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 1578 can include deep learning infrastructure of a data center that is powered by CPUs only.

[0266] The deep learning infrastructure of the server 1578 can be capable of fast real-time inference, and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 1500. For example, the deep learning infrastructure can receive periodic updates from the vehicle 1500, such as a sequence of images and / or objects located in the sequence of images that the vehicle 1500 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 1500, and if the results do not match and the infrastructure concludes that the AI in the vehicle 1500 is malfunctioning, the server 1578 can transmit a signal to the vehicle 1500 instructing the failsafe computer of the vehicle 1500 to take control, notify the passengers, and complete a safe parking operation.

[0267] For inference, the server 1578 can include GPUs 1584 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time response possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0268] Example Computing Device

[0269] FIG. 16 is a block diagram of an example computing device 1600 suitable for implementing some embodiments of the present disclosure. The computing device 1600 can include an interconnection system 1602 directly or indirectly coupling the following devices: memory 1604, one or more central processing units (CPUs) 1606, one or more graphics processing units (GPUs) 1608, a communication interface 1610, input / output (I / O) ports 1616, I / O components 1614, a power supply 1616, one or more presentation components 1618 (e.g., display(s)), and one or more logic units 1620. In at least one embodiment, the computing device(s) 1600 can include one or more virtual machines (VMs), and / or any component thereof can include a virtual component (e.g., a virtual hardware component). For non-limiting examples, one or more of the GPUs 1608 can include one or more vGPUs, one or more of the CPUs 1606 can include one or more vCPUs, and / or one or more of the logic units 1620 can include one or more virtual logic units. As such, the computing device(s) 1600 can include discrete components (e.g., a full GPU dedicated to the computing device 1600), virtual components (e.g., a portion of a GPU dedicated to the computing device 1600), or a combination thereof.

[0270] Although FIG. 16 various blocks of the computing device are shown as being connected by lines through the interconnect system 1602, this is not intended to be limiting and is for clarity only. For example, in some embodiments, the presentation component 1618 (such as a display device) can be considered an I / O component 1614 (e.g., if the display is a touchscreen). As another example, the CPU 1606 and / or GPU 1608 can include memory (e.g., the memory 1604 can represent a storage device in addition to the memory of the GPU 1608, CPU 1606, and / or other components). In other words, FIG. 16 the computing device of FIG. 16 is illustrative. No distinction is made between “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types in such categories because all are considered within the scope of the computing device of FIG. 16. FIG. 16

[0271] The interconnect system 1602 can represent one or more links or buses, such as an address bus, data bus, control bus, or a combination thereof. The interconnect system 1602 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1606 can be directly connected to the memory 1604. Further, the CPU 1606 can be directly connected to the GPU 1608. Where there are direct or point-to-point connections between components, the interconnect system 1602 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1600.

[0272] The memory 1604 can include any of a variety of computer-readable media. Computer-readable media can be any available media that can be accessed by computing device 1600. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0273] ​Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1604 can store computer readable instructions (e.g., representing program(s) and / or program element(s), such as an operating system). Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store the desired information and which can be accessed by computing device 1600. As used herein, computer storage media does not include a signal per se.

[0274] Computer storage media can embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the any of the above should also be included within the scope of computer readable media.

[0275] CPUs 1606 can each include one or more cores capable of handling a multitude of software threads concurrently (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.). CPUs 1606 can include any type of processors, and can include different types of processors depending on the type of computing device 1600 implemented (e.g., less powerful processors for mobile devices and more powerful processors for servers). For example, depending on the type of computing device 1600, the processors can be Advanced RISC Machines (ARM) processors implemented using Reduced Instruction Set Computing (RISC) or x86 processors implemented using Complex Instruction Set Computing (CISC). Computing device 1600 can include one or more CPUs 1606 in addition to, or instead of, one or more microprocessors or co-processors such as mathematical co-processors.

[0276] In addition to or in place of the CPU(s) 1606, the GPU(s) 1608 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1600 to perform one or more of the methods and / or processes described herein. One or more of the GPU(s) 1608 can be integrated GPUs (e.g., with one or more of the CPUs 1606) and / or one or more of the GPU(s) 1608 can be discrete GPUs. In embodiments, one or more of the GPU(s) 1608 can be a co-processor of one or more of the CPUs 1606. The GPU(s) 1608 can be used by the computing device 1600 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 1608 can be used for general-purpose computing on GPUs (GPGPU). The GPU(s) 1608 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads concurrently. The GPU(s) 1608 can generate pixel data for an output image in response to rendering commands (e.g., received from the CPU(s) 1606 via a host interface). The GPU(s) 1608 can include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). The display memory can be included as part of the memory 1604. The GPU(s) 1608 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or can connect the GPUs through a switch (e.g., using an NVSwitch). When combined together, each GPU 1608 can generate pixel data or GPGPU data for a different portion of an output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or can share memory with other GPUs.

[0277] In addition to or in place of CPU(s) 1606 and / or GPU(s) 1608, logic unit(s) 1620 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1600 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 1606, GPU(s) 1608, and / or logic unit(s) 1620 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. One or more of logic unit(s) 1620 can be part of and / or integrated in one or more of CPU(s) 1606 and / or GPU(s) 1608, and / or one or more of logic unit(s) 1620 can be discrete components or otherwise external to CPU(s) 1606 and / or GPU(s) 1608. In embodiments, one or more of logic unit(s) 1620 can be a co-processor of one or more of CPU(s) 1606 and / or GPU(s) 1608.

[0278] Examples of logic unit(s) 1620 include one or more processing cores and / or components thereof, such as tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), visual processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multi-processors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0279] Communication interface 1610 can include one or more receivers, transmitters, and / or transceivers that enable computing device 1600 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communications. Communication interface 1610 can include components and functionality to enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., through Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0280] I / O port(s) 1612 can enable the computing device 1600 to logically couple to other devices including I / O components 1614, presentation components 1618, and / or other components, some of which can be built in to (e.g., integral with) the computing device 1600. Illustrative I / O components 1614 include a microphone, keyboard, pointing devices such as a mouse, a joystick, game pad, game controller, a satellite dish, a scanner, a printer, a wireless device, etc. The I / O components 1614 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to appropriate network elements for further processing. A NUI can implement any combination of speech recognition, gesture recognition, facial recognition, biometric recognition, posture recognition, gesture recognition within a

[0281] Power supply 1616 can include a hard-wired power supply, a battery power supply, or a combination thereof. Power supply 1616 can provide power to the computing device 1600 to enable the components of the computing device 1600 to operate.

[0282] Presentation components 1618 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. Presentation components 1618 can receive data from other components (e.g., GPU 1608, CPU 1606, etc.) and output the data (e.g., as images, video, sound, etc.).

[0283] Example data center

[0284] FIG. 17 An example data center 1700 that can be used in at least one embodiment of the present disclosure is shown. The data center 1700 can include a data center infrastructure layer 1710, a framework layer 1720, a software layer 1730, and / or an application layer 1740.

[0285] As FIG. 17As shown, the data center infrastructure layer 1710 can include a resource orchestrator 1712, grouped computing resources 1714, and node computing resources (“node C.R.s”) 1716(1)-1716(N), where “N” represents any whole, positive integer. In at least one embodiment, the node C.R.s 1716(1)-1716(N) can include, but are not limited to, any number of central processing units (“CPUs” or “processors”) including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), memory devices (e.g., dynamic random access memory), storage devices (e.g., solid state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules, and / or cooling modules, and the like. In some embodiments, one or more node C.R.s from the node C.R.s 1716(1)-1716(N) can correspond to a server having one or more of the above-described computing resources. Moreover, in some embodiments, the node C.R.s 1716(1)-1716(N) can include one or more virtual components, such as a vGPU, a vCPU, etc., and / or one or more of the node C.R.s 1716(1)-1716(N) can correspond to a virtual machine (VM).

[0286] In at least one embodiment, the grouped computing resources 1714 can include individual groups of node C.R.s 1716 housed within one or more racks (not shown) or housed within many racks at different geographic locations (also not shown) within a data center. The individual groups of node C.R.s 1716 within the grouped computing resources 1714 can include grouped computing, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several node C.R.s 1716 including CPUs, GPUs, and / or other processors can be grouped within one or more racks to provide computing resources to support one or more workloads. The one or more racks can also include any quantity of power modules, cooling modules, and / or network switches in any combination.

[0287] The resource orchestrator 1722 can configure or otherwise control the one or more node C.R.s 1716(1)-1716(N) and / or the grouped computing resources 1714. In at least one embodiment, the resource orchestrator 1722 can comprise a software design infrastructure (“SDI”) management entity for the data center 1700. The resource orchestrator 1722 can comprise hardware, software, or some combination thereof.

[0288] In at least one embodiment, as FIG. 17As shown, framework layer 1720 may include a job scheduler 1732, a configuration manager 1734, a resource manager 1736, and / or a distributed file system 1738. Framework layer 1720 may include a framework for software 1732 supporting software layer 1730 and / or one or more applications 1742 supporting application layer 1740. Software 1732 or application 1742 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1720 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1738 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of resource manager. In at least one embodiment, job scheduler 1732 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1700. Configuration manager 1734 may be able to configure different layers, such as software layer 1730 and framework layer 1720 (which includes Spark and distributed file system 1738 for supporting large-scale data processing). Resource manager 1736 may be able to manage compute resources mapped to or allocated to clusters of distributed file system 1738 and job scheduler 1732 or to support clusters of distributed file system 1738 and job scheduler 1732. In at least one embodiment, clusters or groups of compute resources may include grouped compute resources 1714 in data center infrastructure layer 1710. Resource manager 1736 may coordinate with resource coordinator 1712 to manage these mapped or allocated compute resources.

[0289] In at least one embodiment, the software 1732 included in software layer 1730 may include software used in at least a portion of the nodes CRs 1716(1)-1716(N), the grouped computing resources 1714, and / or the distributed file system 1738 of framework layer 1720. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0290] In at least one embodiment, applications 1742 included in application layer 1740 can include one or more types of applications used by at least portions of node C.R.s 1716(1)-1716(N), grouped computing resources 1714, and / or distributed file system 1738 of framework layer 1720. One or more types of applications can include, but are not limited to, any number of genomics applications, cognitive computing and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0291] In at least one embodiment, any of configuration manager 1734, resource manager 1736, and resource orchestrator 1712 can implement any number and type of self-modification actions based on any quantity and type of data acquired in any technically feasible manner. Self-modification actions can free data center operators of data center 1700 from making potentially poor configuration decisions and can avoid underutilization and / or poor performance portions of data center.

[0292] According to one or more embodiments described herein, data center 1700 can include tools, services, software, or other resources to train one or more machine learning models or use one or more machine learning models to predict or infer information. For example, machine learning model(s) can be trained by computing weight parameters according to a neural network architecture using software and / or computing resources described above with respect to data center 1700. In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer or predict information using resources described above with respect to data center 1700 by using weight parameters computed through one or more training techniques such as, but not limited to, those described herein.

[0293] In at least one embodiment, data center 1700 can use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or virtual computing resources corresponding thereto) to perform training and / or inference using resources described above. Furthermore, one or more software and / or hardware resources described above can be configured as a service that allows users to train or perform inference on information, such as image recognition, speech recognition, or other artificial intelligence services.

[0294] Example network environment

[0295] Network environments suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of the computing device(s) 1600 - e.g., each device can include similar components, features, and / or functionality of the computing device(s) 1600. Further, where backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of a data center 1700, an example of which is described in more detail herein with respect to FIG. 17. FIG. 16 FIG. 17 are described in more detail.

[0296] Components of the network environment can communicate with each other via a network, which can be wired, wireless, or both. The network can include multiple networks or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the public switched telephone network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunication network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0297] Compatible network environments can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein for servers can be implemented on any number of client devices.

[0298] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework that supports one or more applications of a software layer and / or an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a free and open-source software web application framework as can be used for large-scale data processing (e.g., “big data”) using a distributed file system. ​

[0299] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functions described herein (or one or more portions thereof). Any of these different functions can be distributed across multiple locations from a central or core server (e.g., can be distributed across one or more data centers of a state, region, country, globally, etc.). The core server can designate at least a portion of the functions to an edge server if the connection to the user (e.g., client device) is relatively close to the edge server. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0300] The client device(s) can include at least some of the components, features, and functionality of the example computing device 1600 described herein with respect to FIG. 16 As examples and not by way of limitation, a client device can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance device or system, vehicle, boat, spaceship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, appliance, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0301] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0302] As used herein, the term "and / or," with respect to a listing of two or more elements, means that at least one of the listed elements is present. For example, "A, B, and / or C" can mean A alone, B alone, C alone, A and B, A and C, B and C, or A, B, and C. Further, "at least one of A or B" can mean at least one of A, at least one of B, or at least one of A and at least one of B.

[0303] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of the disclosure. Rather, the inventive subject matter contemplates that the claimed subject matter can also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" can be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

[0304] Example Clauses

[0305] In one or more embodiments, a method includes, for at least two frames of a plurality of frames corresponding to travel of a vehicle: generating sensor data using one or more sensors of the vehicle; computing, using one or more neural networks (NNs) and based at least in part on the sensor data, an output indicative of a location of a landmark; converting the location to a three-dimensional (3D) world space location relative to an origin of the vehicle; encoding the 3D world space location to generate encoded data corresponding to the frame; and transmitting the encoded data to a server to cause the server to generate a map including the landmark.

[0306] In one or more embodiments, the sensor data includes image data representing an image; the one or more outputs are computed in a two-dimensional (2D) image space; and converting the location includes converting a 2D image space location to a 3D world space location based at least in part on at least one of intrinsic camera parameters or extrinsic camera parameters corresponding to a camera that generated the image data.

[0307] In one or more embodiments, the landmark includes one or more of: a lane dividing line; a road boundary; a sign; a pole; a wait condition; a vertical structure; another road user; a static object; or a dynamic object.

[0308] In one or more embodiments, the output further represents at least one of a pose of the landmark or a geometry of the landmark.

[0309] In one or more embodiments, the output further represents semantic information corresponding to the landmark.

[0310] In one or more embodiments, the landmark comprises a lane marking, and the method further comprises: for the at least two lane markings, combining the detection of each of the at least two lane markings to generate a continuous lane marking representation.

[0311] In one or more embodiments, the method further comprises, for each frame of the at least two frames: determining, based at least in part on the sensor data, a translation and a rotation of the vehicle relative to a previous frame of the frames, wherein encoding further comprises encoding the rotation and the translation.

[0312] In one or more embodiments, the method further comprises: compressing the data representing the rotation and the translation using a delta compression algorithm to generate delta compressed data, wherein encoding the rotation and the translation further comprises encoding the delta compressed data.

[0313] In one or more embodiments, the sensor data comprises global navigation satellite system (GNSS) data, and the method further comprises, for each frame of the at least two frames: determining, based at least in part on the GNSS data, a global position of the vehicle, wherein encoding further comprises encoding the global position.

[0314] In one or more embodiments, the sensor data comprises LiDAR data, and the method further comprises: filtering out dynamic objects from the LiDAR data to generate filtered LiDAR data, wherein encoding the data further represents the filtered LiDAR data.

[0315] In one or more embodiments, the encoded data further represents the sensor data, the sensor data comprising one or more of: LiDAR data; RADAR data; ultrasonic data; ultrasonic wave data; GNSS data; image data; or inertial measurement unit (IMU) data.

[0316] In one or more embodiments, the method further comprises: receiving, from a server, data representing a request for a location of the landmark, wherein transmitting the encoded data is based at least in part on the request.

[0317] In one or more embodiments, the method further comprises: determining to generate each frame of the at least two frames based at least in part on one or more of a time threshold being met or a distance threshold being met.

[0318] In one or more embodiments, the encoded data is encoded as serialized structured data using protocol buffers.

[0319] In one or more embodiments, the server comprises at least one of: a data center server; a cloud server; or an edge server.

[0320] In one or more embodiments, a method comprises, for at least two frames of a plurality of frames corresponding to travel of a vehicle: generating one or more LiDAR data or RADAR data using one or more sensors of the vehicle; filtering out points corresponding to one or more of the LiDAR data or the RADAR data, generating filtered data; determining a pose of the vehicle relative to a previous frame; encoding the filtered data and the pose, generating encoded data; and transmitting the encoded data to a cloud server to cause the cloud server to generate at least one of a LiDAR layer of a map or a RADAR layer of the map using the pose.

[0321] In one or more embodiments, the filtering out comprises at least one of dynamic object filtering or deduplication.

[0322] In one or more embodiments, the method further comprises: generating sensor data using the one or more sensors of the vehicle; computing, using one or more neural networks (NNs) and based at least in part on the sensor data, an output indicative of a landmark location, wherein the transmitting further comprises transmitting data representative of the landmark location.

[0323] In one or more embodiments, the encoding the filtered data comprises encoding the filtered data using an octree.

[0324] In one or more embodiments, the method further comprises compressing the filtered data using quantization.

[0325] In one or more embodiments, a system comprises: one or more sensors; one or more processors; and one or more memory devices having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations comprising: at a current frame, generating sensor data using the one or more sensors; computing, using one or more neural networks (NNs) and based at least in part on the sensor data, an output indicative of a landmark location; converting the location to a three-dimensional (3D) world space location relative to an origin of the vehicle; determining a pose of the current frame relative to a previous pose of a previous frame; encoding the 3D world space location and the pose to generate encoded data corresponding to the current frame; and transmitting the encoded data.

[0326] In one or more embodiments, the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system of an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system containing one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0327] In one or more embodiments, a method includes: using one or more neural networks (NNs) and based at least in part on sensor data generated by one or more sensors of a vehicle, computing an output indicative of a position in a two-dimensional (2D) image space corresponding to a detected landmark; generating a distance function representation of the detected landmark based at least in part on the position; for at least two poses of a plurality of poses of the vehicle represented in a cost space: projecting map landmarks corresponding to a map into the 2D image space to generate projected map landmarks; comparing the projected map landmarks to the distance function representation; computing a cost based at least in part on the comparison; updating a point of the cost space corresponding to each of the at least two poses based at least in part on the cost; and localizing the vehicle to the map based at least in part on the cost space.

[0328] In one or more embodiments, the vehicle is localized to an origin of a plurality of road segments of the map, and the method further includes localizing the vehicle to a global coordinate system based at least in part on localizing the vehicle to the origin of the road segment.

[0329] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes: generating a plurality of additional cost spaces corresponding to a plurality of previous frames; generating an aggregated cost space corresponding to the cost space and the plurality of additional cost spaces, the generating the aggregated cost space including using a self-motion compensation with respect to the current frame; and applying a filter to the aggregated cost space to determine a position representation of the vehicle for the current frame, wherein the localizing is further based at least in part on the position representation.

[0330] In one or more embodiments, the filter includes a Kalman filter.

[0331] In one or more embodiments, the position representation includes an ellipsoid.

[0332] In one or more embodiments, each of the plurality of cost spaces is generated based at least in part on a respective output computed based at least in part on respective sensor data corresponding to a respective frame of the plurality of frames.

[0333] In one or more embodiments, the plurality of additional cost spaces are generated to correspond to a first road segment, and generating the aggregated cost space includes transforming the one or more additional cost spaces to correspond to a second road segment corresponding to the cost space of the current frame.

[0334] In one or more embodiments, the transforming includes a translation transform and a rotation transform.

[0335] In one or more embodiments, the localization corresponds to a localization modality, and the method further includes: localizing the vehicle using one or more additional localization modalities; and fusing the localization modality with the one or more additional localization modalities to determine a final localization result.

[0336] In one or more embodiments, the one or more additional localization modalities include a LiDAR modality, a RADAR modality, or a fusion modality.

[0337] In one or more embodiments, the localization modality corresponds to a camera modality.

[0338] In one or more embodiments, the fusing includes applying a Kalman filter to localization results from the localization modality and the additional localization modalities.

[0339] In one or more embodiments, one or more of the map landmarks or the detected landmarks correspond to at least one of: a lane dividing line, a road boundary, a sign, a pole, a vertical structure, a wait condition, a static object, or a dynamic actor.

[0340] In one or more embodiments, generating the cost space further for at least two of the plurality of poses of the vehicle represented in the cost space: projecting the map landmarks corresponding to the map into another 2D image space to generate additional projected map landmarks; comparing map semantic information of the additional projected map landmarks to detected semantic information of the detected landmarks; and computing another cost based at least in part on comparing the map semantic information to the detected semantic information.

[0341] In one or more embodiments, a method includes: generating LiDAR data using one or more LiDAR sensors of a vehicle; for at least two of a plurality of poses of the vehicle represented in a cost space, generating the cost space from: projecting points corresponding to the LiDAR data into a distance function representation of a LiDAR layer of a map corresponding to the points; comparing the points to the distance function representation; computing a cost based at least in part on the comparison; and updating points of the cost space corresponding to each of the at least two poses based at least in part on the cost; and localizing the vehicle to the map based at least in part on the cost space.

[0342] In one or more embodiments, the method further includes determining the points from a larger set of points represented by the LiDAR data based at least in part on height values corresponding to points within a height range, where the LiDAR point cloud corresponds to the height range.

[0343] In one or more embodiments, for each of the at least two poses of the vehicle represented in the cost space, generating the cost space further by: projecting an intensity representation corresponding to the LiDAR data into a map intensity representation corresponding to the LiDAR layer of the map; comparing the intensity representation to the map intensity representation; and calculating another cost based at least in part on comparing the intensity representation to the map intensity representation.

[0344] In one or more embodiments, comparing the height representation to the map height representation includes adjusting at least one of the height representation or the map height representation based at least in part on a difference between a vehicle origin of the vehicle or a road segment corresponding to the map height representation.

[0345] In one or more embodiments, for each of the at least two poses of the vehicle represented in the cost space, generating the cost space further by: projecting an intensity representation corresponding to the LiDAR data into a map intensity representation corresponding to the LiDAR layer of the map; comparing the intensity representation to the map intensity representation; and calculating another cost based at least in part on comparing the intensity representation to the map intensity representation.

[0346] In one or more embodiments, the origin of the segment to which the vehicle is located is positioned to a plurality of segments of the map, and the method further includes being positioned to a global coordinate system based at least in part on being positioned to the origin of the segment.

[0347] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes: generating a plurality of additional cost spaces corresponding to a plurality of previous frames; generating an aggregate cost space corresponding to the cost space and the plurality of additional cost spaces, generating the aggregate cost space including using a self-motion compensation relative to the current frame; and applying a filter to the aggregate cost space to determine a position representation of the vehicle for the current frame, where the locating is further based at least in part on the position representation.

[0348] In one or more embodiments, the filter includes a Kalman filter.

[0349] In one or more embodiments, the position representation includes an ellipsoid.

[0350] In one or more embodiments, one or more of the plurality of additional cost spaces are generated to correspond to the first road segment, and generating the aggregated cost space includes transforming the one or more additional cost spaces to correspond to a second road segment corresponding to the cost space of the current frame.

[0351] In one or more embodiments, the localization corresponds to a localization modality, and the method further includes: localizing the vehicle using one or more additional localization modalities; and fusing the localization modality with the one or more additional localization modalities to determine a final localization result.

[0352] In one or more embodiments, the localization modality includes a LiDAR modality, and the one or more additional localization modalities include a RADAR modality, an image modality, or a fusion modality.

[0353] In one or more embodiments, a system includes: one or more sensors; one or more processors; and one or more memory devices having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations including: generating sensor data using the one or more sensors; for at least two poses of a plurality of poses represented in a cost space, generating the cost space by: projecting points corresponding to the sensor data into a distance function representation of a point cloud corresponding to a map; comparing the points to the distance function representation; computing a cost based at least in part on the comparison; and updating points of the cost space corresponding to each of the at least two poses based at least in part on the cost; and localizing a vehicle to the map based at least in part on the cost space.

[0354] In one or more embodiments, the system is included in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system of an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system containing one or more virtual machines (VMs); a system implemented using robots; a system implemented at least in part in a data center; or a system implemented at least in part using cloud computing resources.

[0355] In one or more embodiments, the vehicle is localized to an origin of a road segment of a plurality of road segments of the map, and the method further includes localizing to a global coordinate system based at least in part on localizing the vehicle to the origin of the road segment.

[0356] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes generating a plurality of additional cost spaces corresponding to a plurality of previous frames, generating an aggregated cost space corresponding to the cost space and the plurality of additional cost spaces, generating the aggregated cost space including using self-motion compensation relative to the current frame, and applying a filter to the aggregated cost space to determine a position representation of the vehicle for the current frame, wherein the localization is further based at least in part on the position representation.

[0357] In one or more implementations, the sensor data corresponds to one or more of LiDAR data or RADAR data, and the one or more sensors include one or more of a LiDAR sensor or a RADAR sensor.

Claims

1. A method comprising: Enables the vehicle to navigate within the environment based at least on a fused map representing the environment, wherein the fused map is generated at least through the following operations: Receive multiple map streams corresponding to multiple driving sessions; Identify segments of two or more map streams that are within a threshold distance of each other from the plurality of map streams; A frame graph is generated based at least on the registration of the segments from the two or more map streams, the frame graph including the poses associated with the two or more map streams and the corresponding pose links; At least based on dividing the frame into road segments, a pose map is generated, the pose map including: a first part of the pose corresponding to a first position in a first road segment of the road segment and a first part of the pose link, and a second part of the pose corresponding to a second position in a second road segment of the road segment and a second part of the pose link; as well as The fused map is generated at least in part based on data from the two or more map streams fused according to the attitude map.

2. The method of claim 1, wherein the fused map is further generated by at least the following operation: converting the two or more map streams into corresponding maps, wherein fusion of the data from the two or more map streams includes fusion of the data from the corresponding maps.

3. The method of claim 2, wherein a single map in the corresponding map comprises two or more map layers corresponding to different sensor modes.

4. The method according to claim 1, further comprising: The path of the vehicle through the environment is determined at least based on the fused map, wherein navigating the vehicle through the environment includes controlling the vehicle to navigate along the path.

5. The method of claim 1, wherein the plurality of map streams are generated by a plurality of vehicles.

6. The method according to claim 1, further comprising: The vehicle receives at least a subset of the fused map corresponding to the sensor modes mounted on the vehicle.

7. A method comprising: Receive data, which represents sensor data, perception outputs from one or more deep neural networks (DNNs), and trajectory information corresponding to multiple driving operations; The data is converted into first map data representing multiple maps, where each map in the multiple maps corresponds to one of the multiple drives. Based at least on the trajectory information, a first segment of the first map in the plurality of maps is registered to a second segment of the second map in the plurality of maps; A frame map is generated based at least on the registration, the frame map including poses and corresponding pose links associated with the first segment of the first map and the second segment of the second map; A pose map is generated based at least on assigning at least a portion of the pose from the frame map and at least a portion of the pose links to road segments; At least based on the attitude map, a second map data is generated, the second map data representing a fused map corresponding to the first map and the second map, the fused map including the road segment; as well as Data representing the fused map is sent to the vehicle so that the vehicle can navigate in the environment based at least on the fused map.

8. The method of claim 7, further comprising locating the vehicle relative to a global coordinate system.

9. The method of claim 7, wherein a single map comprises one or more map layers corresponding to different sensor modalities.

10. The method of claim 7, further comprising: Additional data corresponding to the additional driving is received, at least in part, based on the quality of the positioning results; And at least based on the additional data, update the fused map.

11. The method of claim 7, further comprising: An updated attitude map is generated based on at least one or more optimization algorithms applied to the attitude map, wherein second map data representing the fused map is generated based at least on the updated attitude map.

12. The method of claim 7, wherein receiving data representing the sensor data, the sensing output, and the trajectory information includes receiving preprocessed data representing the sensor data, the sensing output, and the trajectory information, the preprocessed data reducing the amount of data.

13. The method of claim 7, wherein the data representing at least one of the sensor data, the sensing output, or the trajectory information is received in a compressed format, and the method further comprises: The data in the compressed format is decompressed, wherein converting the data into the first map data includes converting the data after decompression.

14. The method of claim 7, wherein the registration comprises geometric registration.

15. The method of claim 7, further comprising: Assign one or more origin points to the road segment; Calculate one or more transformations between the one or more origins of adjacent road segments in the road segment; and The one or more transformations are encoded in the fused map.

16. The method of claim 7, further comprising: Registration of the first segment and the second segment is determined based at least on GNSS data corresponding to the first map and the second map.

17. The method of claim 7, wherein the perception output corresponds to a three-dimensional 3D representation of the detected landmark, the detected landmark including one or more of a lane divider, road boundary, sign, pole, static object or vertical structure.

18. The method of claim 7, wherein the sensor data corresponds to at least one of LiDAR data, RADAR data, or ultrasound data.

19. A system comprising: One or more processing units, said one or more processing units being used for: Receive multiple map streams corresponding to multiple driving sessions; Identify segments of two or more map streams that are within a threshold distance of each other from the plurality of map streams; At least based on the registration of the segment, a frame map is generated, the frame map including the pose associated with the segment and the corresponding pose link; At least based on dividing the frame into road segments, a pose map is generated, the pose map including: a first part of the pose corresponding to a first road segment in the road segment and a first part of the pose link, and a second part of the pose corresponding to a second road segment in the road segment and a second part of the pose link; At least based on the attitude graph, data from the map stream is fused; At least based on the fusion of data from the map stream, a merged map is generated; and The fused map is sent to the vehicle so that the vehicle navigates at least partially based on the fused map.

20. The system of claim 19, wherein the one or more processing units are further configured to: convert the two or more map streams into corresponding maps, wherein the data from the two or more map streams are fused using the corresponding maps.

21. The system of claim 20, wherein a single corresponding map in the corresponding map comprises a map layer corresponding to different sensor modes.

22. The system of claim 19, wherein enabling the vehicle to navigate based on the fused map includes enabling the vehicle to: The path of the vehicle through the environment is determined at least based on the fused map; and Control the vehicle to navigate along the path.

23. The system of claim 19, wherein the one or more processing units are further configured to receive the plurality of map streams from the plurality of vehicles.

24. The system of claim 19, wherein a first subset of the fused map corresponding to a first sensor mode fitted to the vehicle is sent to the vehicle, and a second subset of the fused map corresponding to a second sensor mode not fitted to the vehicle is not sent to the vehicle.

25. The system of claim 19, wherein the system is located in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system for merging one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images