Mapping and Localization for Autonomous Driving Applications

Abstract

An end-to-end system for data generation, map creation using the generated data, and localization onto the created map is disclosed. Map streams—or streams of sensor data, sensory output from deep neural networks (DNNs), and / or relative trajectory data—corresponding to any number of drives by any number of vehicles can be generated and uploaded to the cloud. The map streams can be used to generate map data—and ultimately a fused high-definition (HD) map—representing the data generated across multiple drives. When localizing onto the fused HD map, individual localization results can be generated based on a comparison of real-time data from one sensor modality with map data corresponding to that sensor modality. This process can be repeated for any number of sensor modalities, and the results can be fused together to determine a final fused localization result.

Description

[Technical field]

[0001] The present invention relates to map creation and localization for autonomous driving applications. [Background technology]

[0002] Mapping and localization are crucial processes for autonomous driving functionality. High definition (HD) maps, sensor perception, or a combination thereof are often used to localize vehicles with respect to HD maps to make planning and control decisions. Typically, conventional HD maps are generated using survey vehicles equipped with advanced, highly accurate sensors. However, these sensors are prohibitively expensive to implement in consumer vehicles. In a typical deployment, survey vehicles have the capability to generate HD maps suitable for localization after a single drive. Unfortunately, due to a shortage of survey vehicles with such high-cost sensors, the availability of HD maps for any particular area may be concentrated around metropolitan areas or hubs where survey vehicles operate. For more remote areas, less-frequented areas, and areas further away from these hubs of activity, the data used to generate the HD map may have been collected from a single, small drive - that is, if data is available at all. As a result, if a single drive results in sensor data of poor quality or not well suited for this purpose - for example, due to occlusions, dynamic objects, inclement weather effects, debris, construction artifacts, temporary hardware failures, and / or other issues that may compromise the quality of the collected sensor data - the HD map generated from the sensor data may be unsafe or unreliable for use in localization.

[0003] To remedy these quality concerns, another survey vehicle may be required to perform another drive at the compromised location. However, the identification, deployment, sensor data generation, and map update processes can take a long time due to systematic data collection and map creation processes, rendering the HD map unusable until updates are made. This problem is exacerbated when road conditions or layout change frequently or dramatically over time - for example, due to construction - because there may be no mechanism to identify these changes, and even if identified, there is no way to generate updated data without deploying another survey vehicle.

[0004] In addition, consumer vehicles may not be equipped with the same high-quality, high-cost sensors, so localization to HD maps - even when available - cannot be performed using multiple sensor modalities - e.g., cameras, LiDAR, RADAR, etc. - because the quality and type of data may not match the data used to generate the HD maps. As a result, localization relies solely on global navigation satellite system (GNSS) data - even for the most expensive and accurate sensor models - which still only achieves an accuracy of a few meters under certain circumstances and / or has variable accuracy. An inaccuracy of a few meters can place the vehicle in a different lane than the current driving lane, or on one side of the road other than the one currently being driven on. Thus, the use of conventional solutions to generate HD maps, when compromised by their inaccurate localization, can result in inaccurate maps that pose a significant obstacle to the achievement of safe or reliable highly autonomous vehicles (e.g., Level 3, 4, and 5) autonomous vehicles. Summary of the Invention [Means for solving the problem]

[0005] Embodiments of the present disclosure relate to techniques for map creation and localization for autonomous driving applications. Specifically, embodiments of the present disclosure include an end-to-end system for data generation, map creation using the generated data, and localization to a created map that can be used with consumer-grade sensors common in commercial vehicles. For example, during the data generation process, data collection vehicles and / or consumer vehicles using consumer-grade sensors can be used to generate sensor data. The resulting data can correspond to a map stream - which can include streams of sensor data, perception output from a deep neural network (DNN), and / or relative trajectory (e.g., rotation and translation) data - corresponding to any number of drives by any number of vehicles. As such, the present system can crowdsource data generation using a large number of vehicles and a large number of drives, in contrast to the systematic data collection efforts of conventional systems. To reduce the bandwidth and memory requirements of the system, data from the map stream can be minimized (e.g., by filtering out dynamic objects, performing LiDAR plane slicing or LiDAR point reduction, converting sensory or camera-based output into 3D location information, running campaigns only on certain data types, etc.) and / or compressed (e.g., using delta compression techniques). As a result of the map stream data being generated using consumer-grade sensors, the sensor data - after being converted into a map format for localization - can be used directly for localization rather than relying solely on GNSS data. Furthermore, since relative trajectory information corresponding to each maneuver is tracked, this information can be used to generate individual road segments (e.g., road segments of size 25 meters, 50 meters, etc.) onto which can be localized, thereby enabling localization accuracy within the centimeter range.

[0006] During map creation, the map streams may be used to generate map data representing data generated across multiple maneuvers - and ultimately, a fused HD map. Additionally, as new map streams are generated, these additional maneuvers may be merged, combined, or integrated with existing map stream data to further increase the robustness of the HD map. For example, each map stream may be converted to a respective map, and any number of maneuver segments from any number of maps (or corresponding map streams) may be used to generate a fused HD map representation of a particular maneuver segment. Pairs of maneuver segments may be geometrically registered with respect to each other to determine pose links representing rotations and translations between the poses (or frames) of the pair of maneuvers. The frame graph representing the pose links may be divided into road segments - e.g., road segments used for relative localization - and the poses corresponding to each road segment may undergo optimization. The resulting finalized poses within each segment may be used to fuse various sensor data and / or sensory outputs to generate the final fused HD map. As a result, and because the map data corresponds to consumer quality sensors, sensor data and / or perception results (e.g., landmark locations) from the HD map may be used directly for localization (e.g., by comparing current real-time sensor data and / or perception with corresponding map information), in addition to using GNSS data in embodiments.

[0007] For example, when localizing to a fused HD map, individual localization results may be generated based on a comparison of sensor data and / or sensory output from a sensor modality with map data corresponding to the same sensor modality. For example, a cost space may be sampled at each frame using data corresponding to the sensor modality, an aggregate cost space may be generated using multiple individual cost spaces, and filtering (e.g., using a Kalman filter) may be used to finalize the localization results for a particular sensor modality. This process may be repeated for any number of sensor modalities - e.g., LiDAR, RADAR, camera, etc. - and the results may be fused together to determine a final fused localization result for the current frame. The fused localization result may then be carried forward to the next frame and used to determine the fused localization for the next frame, etc. As a result of the HD map including individual road segments for localization, and each road segment having a corresponding global position, a global localization result may also be realized when the vehicle localizes to a local or relative coordinate system corresponding to the road segment.

[0008] The present systems and methods for map creation and localization for autonomous driving applications are described in detail below with reference to the accompanying drawings. [Brief description of the drawings]

[0009] [Figure 1] 1 is a data flow diagram of a process of a map creation and localization system according to some embodiments of the present disclosure. [Diagram 2] 1 illustrates a data flow diagram of a process for map stream generation according to some embodiments of the present disclosure. [Diagram 3] 1 is a flow diagram illustrating a method for map stream generation according to some embodiments of the present disclosure. [Figure 4]1 is a data flow diagram of a process for map creation according to some embodiments of the present disclosure. [Figure 5A] 1 is a data flow diagram of a process for converting a map stream into a map for registration according to some embodiments of the present disclosure. [Figure 5B] 1 is a data flow diagram of a process for converting a map stream into a map for registration according to some embodiments of the present disclosure. [Figure 5C] 1 is an illustration of partitioning and determining pair-wise connections between partitions for registration, according to some embodiments of the present disclosure. [Figure 5D] 11 is a table illustrating the number of pair-wise connections for registration for different numbers of sections using different approaches, according to some embodiments of the present disclosure. [Figure 5E] FIG. 2 illustrates an example visualization of registering two base layer map partitions using a front-facing camera and a rear-facing camera, according to some embodiments of the present disclosure. [Figure 5F] FIG. 2 illustrates an example visualization of registering two base layer map partitions using a front-facing camera and a rear-facing camera, according to some embodiments of the present disclosure. [Figure 6A] 1 is an exemplary flame graph generated after registration of multiple driving segments, according to some embodiments of the present disclosure. [Figure 6B] FIG. 6B illustrates road segment generation using the flame graph of FIG. 6A in accordance with some embodiments of the present disclosure. [Figure 6C] 6C is a segment graph corresponding to the determined road segments of the flame graph of FIG. 6B, in accordance with some embodiments of the present disclosure. [Figure 6D] FIG. 13 illustrates an example pose-link error calculation for a road segment pose, according to some embodiments of the present disclosure. [Figure 6E] FIG. 1 illustrates an example pose graph and corresponding optimization algorithm, according to some embodiments of the present disclosure. [Figure 6F]FIG. 1 illustrates an example pose graph and corresponding optimization algorithm, according to some embodiments of the present disclosure. [Figure 6G] FIG. 1 illustrates an example pose graph and corresponding optimization algorithm, according to some embodiments of the present disclosure. [Figure 6H] FIG. 6C illustrates a finalized road segment determined from the pose graph of FIG. 6G, in accordance with some embodiments of the present disclosure. [Figure 6I] FIG. 2 illustrates a diagram showing a transformation between origins of road segments of a segment graph, according to some embodiments of the present disclosure. [Figure 7] 1 is a flow diagram illustrating a method of map creation according to some embodiments of the present disclosure. [Figure 8A] 1 is a data flow diagram of a localization process according to some embodiments of the present disclosure. [Figure 8B] FIG. 2 illustrates a diagram showing the relationship between a vehicle origin, a road segment origin, and a global origin for localization, according to some embodiments of the present disclosure. [Figure 9A] FIG. 1 illustrates a sampled cost space corresponding to frames generated during localization, according to some embodiments of the present disclosure. [Figure 9B] FIG. 1 illustrates an aggregate cost space corresponding to a sampled cost space of multiple frames generated during localization, according to some embodiments of the present disclosure. [Figure 9C] FIG. 1 illustrates a filtered aggregate cost space generated during localization, according to some embodiments of the present disclosure. [Figure 10A] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for camera-based localization, in accordance with some embodiments of the present disclosure. [Figure 10B] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for camera-based localization, in accordance with some embodiments of the present disclosure. [Figure 10C]FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for camera-based localization, in accordance with some embodiments of the present disclosure. [Figure 11A] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for RADAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 11B] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for RADAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 12A] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for LiDAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 12B] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for LiDAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 12C] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for LiDAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 12D] FIG. 1 illustrates an example visualization for sampling poses of a sampled cost space for LiDAR-based localization, in accordance with some embodiments of the present disclosure. [Figure 13] FIG. 2 illustrates an example of fusing multiple localization results to generate a final localization result, according to some embodiments of the present disclosure. [Figure 14] 1 is a flow diagram illustrating a method for localization according to some embodiments of the present disclosure. [Figure 15A] 1 is an illustration of an example autonomous vehicle, according to some embodiments of the present disclosure. [Figure 15B]15B is an illustration of camera positions and fields of view for the example autonomous vehicle of FIG. 15A, according to some embodiments of the present disclosure. [Figure 15C] FIG. 15B is a block diagram of an example system architecture of the example autonomous vehicle of FIG. 15A in accordance with some embodiments of the present disclosure. [Figure 15D] FIG. 15B is a system diagram of communication between a cloud-based server and the example autonomous vehicle of FIG. 15A, according to some embodiments of the present disclosure. [Figure 16] FIG. 1 is a block diagram of an example computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 17] FIG. 1 is a block diagram of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Systems and methods related to map creation and localization for autonomous driving applications are disclosed. The present disclosure may be described with respect to an exemplary autonomous vehicle 1500 (examples of which are described herein with respect to FIGS. 15A-15D and alternatively referred to herein as “vehicle 1500” or “ego vehicle 1500”), but this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more Advanced Driver Assistance System(s) (ADAS)), robots, warehouse vehicles, off-road vehicles, flying ships, boats, and / or other types of vehicles. Additionally, the present disclosure may be described with respect to autonomous driving, but this is not intended to be limiting. For example, the systems and methods described herein may be used in robotics (e.g., mapping and localization for robotics), aviation systems (e.g., mapping and localization for drones or other aircraft), marine systems (e.g., mapping and localization for ships), simulated environments (e.g., for mapping and localization of virtual vehicles within virtual simulated environments), and / or other technology fields, for example, for data generation and curation, map creation, and / or localization.

[0011] Referring to FIG. 1, FIG. 1 is a data flow diagram of a map creation and localization system process 100 according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are merely illustrative. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by an entity may be implemented by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in a memory.

[0012] In some embodiments, the vehicle 1500 may include similar components, features, and / or functionality of the vehicle 1500 described herein with respect to FIGS. 15A-15D. Additionally, map creation 106 may be performed in an embodiment at a data center and may be performed using similar components, features, and / or functionality as described herein with respect to the exemplary computing device 1600 and / or the exemplary data center 1700. In some embodiments, the entire end-to-end process 100 of FIG. 1 may be performed within a single vehicle 1500. Although only a single vehicle 1500 is shown in FIG. 1, this is not intended to be limiting. For example, any number of vehicles 1500 may be used to generate sensor data 102 used for map stream generation 104, and any number of (different) vehicles 1500 may be used to generate sensor data 102 for localization 110. Additionally, the make, model, year, and / or type of vehicle, in addition to the sensor configuration and / or other vehicle attributes, may be the same, similar, and / or different for each vehicle 1500 used in the map stream generation 104, map creation 106, and / or localization 110 process.

[0013] Process 100 may include operations for map stream generation 104, map creation 106, and localization 110. For example, process 100 may be performed as part of an end-to-end system that relies on a map stream generated using sensor data 102 from any of several vehicles 1500 over any number of drives, map creation 106 using map stream data received from vehicles 1500, and localization 110 into a map (e.g., a high-definition (HD) map) generated using the map creation 106 process. Maps represented by map data 108 may include maps generated for or by data collected from different sensor modalities or for individual sensors of individual sensor modalities, in some non-limiting examples. For example, map data 108 may represent a first map (or map layer) corresponding to image-based localization, a second map (or map layer) corresponding to LiDAR-based localization, a third map (or map layer) corresponding to RADAR-based localization, etc. In some embodiments, for example, image-based localization may be performed using a first map (or map layer) corresponding to a forward-facing camera and a second map (which may or may not correspond to the same map layer) corresponding to a rear-facing camera. As such, depending on the sensor configuration of the vehicle 1500 receiving the map data 108 and performing its localization 110, only the necessary portions of the map data 108 may be transmitted to the vehicle 1500. As a non-limiting example, if the first vehicle 1500 includes a camera and a RADAR sensor, but does not include a LiDAR sensor, the camera map layer and the RADAR map layer may be transmitted to the vehicle 1500, and the LiDAR map layer may not be transmitted. As a result, the LiDAR map layer may not be transmitted and / or stored at the vehicle 1500, thereby reducing memory usage and preserving bandwidth at the vehicle 1500.

[0014] Additionally, as the vehicle 1500 navigates the environment using the map data 108 for localization 110, a health check 112 may be performed to ensure that the map is up-to-date or accurate, taking into account changes in road conditions, road structures, construction, and / or the like. As such, when a portion or segment of the map is determined to be less than a desired quality - for example, as a result of difficulty in localizing to the map - map stream generation 104 may be performed and used to update the map corresponding to the particular portion of the segment via map creation 106. As a result, an end-to-end system may be used not only to generate maps for localization 110, but also to ensure that the maps are kept up-to-date for accurate localization over time. The map stream generation 104, map creation 106, and localization 116 processes are each described in more detail herein.

[0015] Map Stream Generation To generate the map stream, any number of vehicles 1500, e.g., consumer vehicles, data collection vehicles, combinations thereof, may perform any number of drives. For example, each vehicle 1500 may drive various road segments from locations in a town, city, state, country, continent, and / or around the world and may generate sensor data 102 during the drive using any number of sensors, e.g., LiDAR sensors 1564, RADAR sensors 1560, cameras 1568, 1570, 1572, 1574, 1598, etc., inertial measurement unit (IMU) sensors 1566, ultrasonic sensors 1562, microphones 1596, speed sensors 1544, steering sensors 1540, global navigation satellite system (GNSS) sensors 1558, etc. Each individual vehicle 1500 may generate sensor data 102 and use the sensor data 102 for map stream generation 104 corresponding to the particular drive of the vehicle 1500. Map streams generated to correspond to different drives from a single vehicle 1500 and drives from any number of other vehicles 1500 may be used in map creation 106, as described in more detail herein.

[0016] As a result of multiple map streams being used to generate map data 108 for any particular road segment, the individual map streams from each drive do not need to be as highly accurate or high fidelity as in conventional systems. For example, conventional systems use survey vehicles equipped with sensor types that are prohibitively expensive and therefore undesirable to install in consumer vehicles (e.g., because the cost of the vehicle would increase significantly). However, the sensors in these survey vehicles can generate sensor data that may be sufficiently reliable after a single drive. However, the downside is that if a particular drive contains a lot of dynamic or temporary factors, such as, but not limited to, traffic, construction artifacts, debris, blockages, inclement weather effects, temporary hardware failures, or other sources of sensor data quality concerns, a single drive may not yield data suitable for generating an accurate map for localization. Additionally, because road conditions change, and due to the small number of survey vehicles available, maps may not be updated quickly - for example, the map is not updated until another survey vehicle traverses the same route. In contrast, in the disclosed system, by leveraging consumer vehicles with low-cost mass-market sensors, any number of map streams from any number of drives can be used to generate map data 108 more quickly and more frequently. As a result, individual map streams from drives where blockages or other quality concerns exist can be relied upon less, and map streams from higher quality sensor data can be relied upon more heavily. In addition, health checks 112 can be performed to update map data 108 more quickly, e.g., in real-time or substantially real-time, as road structure, layout, conditions, surroundings, and / or other information changes. The result of this process is a more crowdsourced approach to map stream generation rather than the systematic data collection efforts of traditional approaches.

[0017] Reference is now made to Figure 2, which is a data flow diagram of a process 104 of map stream generation, according to some embodiments of the present disclosure. For example, the process 104 may correspond to generating a map stream from a single drive by a vehicle 1500. This process 104 may be repeated by any number of vehicles 1500 over any number of drives. The sensor data 102, as described herein, may correspond to sensor data 102 from any number of different sensor modalities and / or for any number of sensors of a single modality. For example, the sensor data 102 may correspond to any of the sensor types described herein with respect to the vehicle 1500 of FIGS. 15A-15D , such as a GNSS sensor 1558 (e.g., a global positioning system sensor), a RADAR sensor 1560, an ultrasonic sensor 1562, a LiDAR sensor 1564, an ultrasonic sensor, an IMU sensor 1566 (e.g., an accelerometer, gyroscope, magnetic compass, magnetometer, etc.), a microphone 1576, a stereo camera 1568, a wide-view camera 1570 (e.g., a fisheye camera), an infrared camera 1572, a surround camera 1574 (e.g., a 360 degree camera), a long-range and / or medium-range camera 1578, a speed sensor 1544 (e.g., for measuring the speed of the vehicle 1500), and / or other sensor types. In some embodiments, the sensor data 102 may be included directly in the map stream 210, e.g., with or without compression using the data compressor 208. For example, for LiDAR data and / or RADAR data, the detections represented by the sensor data 102 may be in a three-dimensional (3D) coordinate space (e.g., world space), and the LiDAR points and / or RADAR points (or detections) may be used directly to generate the LiDAR map layer and / or RADAR map layer, respectively. In some embodiments, the sensor data 102 may be converted from a two-dimensional (2D) coordinate space (e.g., image space) to a 3D coordinate space, e.g., using the data converter 206, and then included in the map stream 210 (e.g., after compression using the data compressor 208, in embodiments).

[0018] In some examples, LiDAR slicing may be performed on the LiDAR data - e.g., on a point cloud generated using raw LiDAR data - to slice the LiDAR data into different elevation ranges. The LiDAR data may be sliced ​​into any number of elevation ranges. In some examples, the LiDAR elevation ranges may be defined relative to the origin of the vehicle 1500 or the rig. By way of non-limiting example, the LiDAR data may be sliced ​​into ground slices (e.g., 5 meters to 300 meters relative to the origin of the vehicle 1500), giraffe plane slices (e.g., 2.5 meters to 5 meters relative to the origin of the vehicle 1500), and / or ground plane slices (e.g., -2.5 meters to 0.5 meters relative to the origin of the vehicle 1500). When LiDAR slicing is performed, the different slices may be stored as separate LiDAR layers in the map stream 210 and / or used to generate separate LiDAR map layers during map creation 106 for localization 110. In some examples, data corresponding to certain slices may be filtered out or removed such that less data is encoded into the map stream 210 and less data is sent to the cloud for map creation 106. For example, ground slices may not be as valuable as giraffe plane slices or ground plane slices because detections far from the ground plane may not be usable, accurate, and / or clear enough for localization. In such instances, ground slices (e.g., 5 meters to 300 meters) may be filtered out.

[0019] The sensor data 102 may include data that may be used to track the absolute position of the vehicle 1500 and / or the local or relative position of the vehicle 1500, e.g., generated by an IMU sensor 1566, a GNSS sensor 1558, a speed sensor 1544, a camera sensor, a LiDAR sensor 1564, and / or other sensor types. For example, in each frame of generated sensor data 102 from any number of sensor modalities, a global position, e.g., using a GNSS sensor 1558, may be recorded for that frame of sensor data 102. This global position, e.g., in the WGS84 reference system, may be used to generally place the vehicle 1500 within a global coordinate system. However, when using HD maps for autonomous driving operations, global localization accuracy is not as valuable as localization accuracy at a given road segment. For example, when driving on highway 101 in Santa Clara, California, the position of vehicle 1500 relative to Interstate 95 in Boston, Massachusetts is not as critical as the vehicle's position relative to 50 meters, 100 meters, 1 mile, etc. ahead on highway 101. In addition, GNSS sensors - even the highest quality ones - may not be accurate to within more than 5 or 10 meters, and thus global localization may be of low precision and / or variable accuracy affected by the relative positioning of the satellites. This may still be the case - for example, global localization may be 5 or 10 meters away - but localization relative to the current road segment may be accurate to within 5 or 10 centimeters. To ensure safety when driving autonomously, localization relative to the local layout or to that road segment provides greater precision and accuracy than a global-only approach.As such, when performing localization, as described in more detail herein, the system of the present disclosure can use GNSS coordinates to determine which road segment the vehicle 1500 is currently traveling on, and can then use the local or relative coordinate system of the determined road segment to more accurately localize the vehicle 1500 (e.g., in an embodiment, without requiring costly GNSS sensor types that are impractical for consumer vehicle implementations). As such, GNSS coordinates may not be required for accurate and precise localization, since once the vehicle 1500 is localized to a given road segment, the vehicle 1500 can localize itself from road segment to road segment as the vehicle 1500 travels. In some non-limiting embodiments, as described herein, each road segment may be 25 meters, 50 meters, 80 meters, 100 meters, and / or another distance.

[0020] To generate data for the map stream 210 that can be used to generate HD maps for accurate and precise local or relative localization, the sensor data 102 IMU sensors 1566, GNSS sensors 1558, speed sensors 1544, wheel sensors (e.g., counting wheel marks of the vehicle 1500), perception sensors (e.g., cameras, LiDAR, RADAR, etc.), and / or other sensor types may be used to track the motion (e.g., rotation and translation) of the vehicle 1500 at each frame or time step. The trajectory or ego-motion of the vehicle 1500 may be used to generate a trajectory layer of the map stream 210. As a non-limiting example, the motion of the perception sensors may be tracked to determine the corresponding motion of the vehicle 1500, e.g., referred to as a visual odometer. The IMU sensors 1566 may be used to track the rotation or pose of the vehicle 1500, and the speed sensors 1544 and / or wheel sensors may be used to track the distance traveled by the vehicle 1500. As such, in a first frame, a first pose (e.g., angles along the x, y, and z axes) and a first position (e.g., (x,y,z)) of the rig or origin of the vehicle 1500 may be determined using the sensor data 102. In a second frame, a second pose and a second position (e.g., relative to the first position) of the rig or origin of the vehicle 1500 may be determined using the sensor data, and so on. As a result, a trajectory may be generated of points (corresponding to frames), where each point may encode information corresponding to the relative position of the vehicle 1500 with respect to the previous point. Additionally, the sensor data 102 and / or output 204 of the DNN 202 captured at each of these points or frames may be associated with the point or frame.As such, when creating an HD map (e.g., map data 108 of FIG. 1), the sensor data 102 and / or output 204 of the DNN 202 may have a known position relative to the origin or rig of the vehicle 1500, and since the origin or rig of the vehicle 1500 may have a corresponding position on a global coordinate system, the sensor data 102 and / or output 204 of the DNN 202 may also have a position on the global coordinate system.

[0021] By using the relative motion of the vehicle 1500 from frame to frame, accuracy can be maintained even when in tunnels, cities, and / or other environments where GNSS signals may be weak or lost. However, even using the relative motion and adding a vector (e.g., representing the translation and rotation between frames) for each frame or time step, the relative position may drift after an advanced period or distance. As a result, at predefined intervals, when drift is detected, and / or based on some other criteria, the relative motion may be reset or realigned. For example, there may be an anchor point in the global coordinate system that may have a known position, and the anchor point may be used to realign or reset the relative motion in the frame.

[0022] In some embodiments, the sensor data 102 is applied to one or more deep neural networks (DNNs) 202 that are trained to compute a variety of different outputs 204. Prior to application or input to the DNNs 202, the sensor data 102 may undergo pre-processing, for example, to transform, crop, expand, shrink, zoom in, rotate, and / or otherwise modify the sensor data 102. For example, if the sensor data 102 corresponds to camera image data, the image data may be cropped, reduced, expanded, flipped, rotated, and / or otherwise adjusted to an appropriate input format for the respective DNN 202. In some embodiments, the sensor data 102 may include image data representing an image, image data representing a video (e.g., a snapshot of a video), and / or sensor data representing a representation of a sensor's perception field (e.g., a depth map for a LIDAR sensor, a value graph for an ultrasonic sensor, etc.). For example, any type of image data format may be used, such as, but not limited to, compressed images in Joint Photographic Experts Group (JPEG) or Luminance / Chrominance (YUV) formats, compressed images as frames resulting from compressed video formats such as H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC), raw images from Red Clear Blue (RCCB), Red Clear (RCCC), or other types of imaging sensors, and / or other formats. Additionally, in some instances, the sensor data 102 may be used without preprocessing (e.g., in a raw or captured format), while in other instances, the sensor data 102 may be subjected to preprocessing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data preprocessor (not shown)).

[0023] Where the sensor data 102 corresponds to LiDAR data, for example, the raw LiDAR data may be accumulated, ego-motion corrected, and / or otherwise adjusted, and / or converted to another representation, such as a 3D point cloud representation (e.g., from a top-down view, a sensor perspective, etc.), a 2D projected image representation (e.g., a LiDAR range image), and / or another representation. Similarly, for RADAR and / or other sensor modalities, the sensor data 102 may be converted to a suitable representation for input to the respective DNN 202. In some embodiments, the DNN 202 may process two or more different sensor data inputs - from any number of sensor modalities - to generate the output 204. As such, as used herein, the sensor data 102 may refer to raw sensor data, preprocessed sensor data, or a combination thereof.

[0024] Although examples are described herein with respect to use of DNN 202, this is not intended to be limiting. For example, and without limitation, DNN 202 may include any type of machine learning model or algorithm, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptrons, long / short-term memory / LSTM, Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid state machines, etc.), area of ​​interest detection algorithms, machine learning models using computer vision algorithms, and / or other types of algorithms or machine learning models.

[0025] As an example, the DNN 202 can process the sensor data 102 to produce detections of lane markings, road boundaries, signs, poles, trees, static objects, vehicles and / or other dynamic objects, wait states, intersections, distance, depth, object dimensions, etc. For example, the detections may correspond to location (e.g., in 2D image space, in 3D space, etc.), geometry, pose, semantic information, and / or other information related to the detection. As such, for lane lines, the location of the lane lines and / or the type of lane lines (e.g., dashed, solid, yellow, white, crosswalk, bike path, etc.) may be detected by the DNN 202 processing the sensor data 102. For signs, the location of the sign or other wait state information and / or its type (e.g., yield ahead, stop, crosswalk, traffic light, yield ahead warning light, construction, speed limit, exit, etc.) may be detected using the DNN 202. For detected vehicles, motorcyclists, and / or other dynamic actors or road users, the position and / or type of the dynamic actor may be identified and / or tracked and / or used to determine waiting conditions within the scene (e.g., if a vehicle moves in a certain way with respect to an intersection, e.g., by coming to a stop, its corresponding intersection or waiting condition may be detected as an intersection with a stop sign or traffic lights).

[0026] The output 204 of the DNN 202, in an embodiment, may be subject to post-processing, for example, by converting the raw output into a useful output - for example, where the raw output corresponds to a confidence level for each point (e.g., in LiDAR, RADAR, etc.) or pixel (e.g., in a camera image) that the point or pixel corresponds to a particular object type, post-processing may be performed to determine which each point or pixel corresponds to a single instance of the object type. This post-processing may include temporal filtering, weighting, outlier removal (e.g., removing pixels or points determined to be outliers), upscaling (e.g., the output may be predicted at a lower resolution than the input sensor data instances and the output may be scaled back to the input resolution), downscaling, curve fitting, and / or other post-processing techniques. The output 204 - after post-processing, in an embodiment - may be in a 2D coordinate space (e.g., image space, LiDAR range image space, etc.) and / or in a 3D coordinate system. In embodiments where the output is in a 2D coordinate space and / or in a 3D coordinate space other than the 3D world space, the data converter 206 may convert the output 204 to the 3D world space.

[0027] In some non-limiting examples, the DNN 202 and / or the output 204 may be implemented using any of the techniques described in U.S. patent application Ser. No. 16 / 286,329, filed February 26, 2019, U.S. patent application Ser. No. 16 / 355,328, filed March 15, 2019, U.S. patent application Ser. No. 16 / 356,439, filed March 18, 2019, U.S. patent application Ser. No. 16 / 385,921, filed April 16, 2019, U.S. patent application Ser. No. 535,440, filed August 8, 2019, U.S. patent application Ser. No. 535,440, filed December 27, 2019, each of which is incorporated herein by reference in its entirety. No. 16 / 728,595, filed on Dec. 27, 2019, U.S. Ser. No. 16 / 728,598, filed on Mar. 9, 2020, U.S. Ser. No. 16 / 813,306, filed on Apr. 14, 2020, U.S. Ser. No. 16 / 814,351, filed on Mar. 10, 2020, U.S. Ser. No. 16 / 911,007, filed on Jun. 24, 2020, and / or U.S. Ser. No. 16 / 514,230, filed on Jul. 17, 2019.

[0028] The data converter 206, in an embodiment, may convert all of the output 204 and / or sensor data 102 to a 3D world space coordinate system having the rig of the vehicle 1500 as the origin (e.g., (0,0,0)). The origin of the vehicle 1500 may be the most forward or rearward point on the vehicle 1500, along the axles of the vehicle 1500, and / or anywhere on the vehicle or relative to the vehicle 1500. In some non-limiting embodiments, the origin may correspond to the center of the rear axle of the vehicle 1500. For example, at a given frame or time step, the sensor data 102 may be generated. As a non-limiting example, a first subset of the sensor data 102 may be generated in a 3D world space relative to the origin and may be used directly - e.g., after compression by the data compressor 208 - to generate the map stream 210. A second subset of the sensor data 102 may be generated in a 3D world space, but may not be relative to the origin of the vehicle 1500. As such, the data converter 206 may transform the sensor data 102—e.g., using the intrinsic and / or extrinsic parameters of each sensor—such that the 3D world space position of the sensor data 102 is relative to the origin of the vehicle 1500. A third subset of sensor data 102 may be generated in 2D space. The data converter 206 may transform this sensor data 102—e.g., using the intrinsic and / or extrinsic parameters of each sensor—such that the 2D space position of the sensor data 102 is in 3D space and relative to the origin of the vehicle 1500.

[0029] In embodiments where a DNN 202 is implemented, the output 204 - e.g., before or after post-processing - may be generated in 2D space (relative or not relative to an origin) and / or 3D space. Similar to the description herein of converting a position from the sensor data 102 directly into 3D world space relative to the origin of the vehicle 1500, the 2D and / or 3D output 204 (not relative to an origin) may be converted by the data converter 206 into 3D space relative to the origin of the vehicle 1500. As such, and as described herein, since the origin of the vehicle 1500 has a known relative position relative to a current section of road or within a series of map stream frames, and the current road section has a relative position in a global coordinate system (e.g., the WGS84 reference system), the position of the sensor data 102 and / or output 204 from the DNN 202 may also have a relative position relative to the current road segment and the global coordinate system.

[0030] With respect to detected road and lane boundaries, e.g., detected using the DNN 202 processing the sensor data 102, landmark filtering may be performed to stitch and / or smooth the detected road and / or lane boundaries. For example, the 3D positions of the lane and / or road boundaries may include gaps in the detection, may include noise, and / or may not be as accurate, precise, or include artifacts as would be otherwise optimal or desirable. As a result, landmark filtering may be performed to stitch together detections within and / or across frames such that virtual continuous lane dividers and road boundaries are generated. These continuous lane dividers and / or road boundaries may be similar to lane graphs used to define several lanes, lane positions, and / or road boundary positions on a driving surface. In some embodiments, smoothing may be performed on the generated solid lines to more accurately reflect known geometric information of the lane and road boundaries. For example, if a detection of a lane line in a frame is distorted with respect to previous and / or subsequent detections, the distorted portions of the lane lines may be smoothed to more closely match the known pattern of lane lines. As such, the encoded information in map stream 210 may correspond to these successive lane lines and / or road lines in addition to, or as an alternative to, encoding each detection into map stream 210.

[0031] In some embodiments, sensor data 102 and output 204 may be generated at all times and for each frame, and all of the data may be sent to a mapping cloud or server as map stream 210. However, in some embodiments, sensor data 102 and / or output 204 may not be generated at each frame, not all data may be sent in map stream 210, or a combination thereof. For example, a map stream campaign may be implemented that identifies and directs what type and amount of data to collect, where to collect the data, how frequently to collect the data, and / or other information. The campaign may allow for targeted or selective generation of data of certain types and / or at certain locations to fill gaps, provide additional data to increase accuracy, update maps when road changes are detected (e.g., via health check 112), and / or for other reasons. As one example, a mapstream campaign may be executed that identifies a vehicle 1500 at a particular location and instructs the vehicle 1500 to generate (or prioritize the generation of) certain types of data, such as LiDAR and RADAR data starting at a certain location and over some distance, in order to reduce computation (e.g., by the vehicle 1500 during map creation 106 when generating the mapstream 210 and processing the mapstream data) and bandwidth (e.g., for transmitting the mapstream 210 to the cloud). In such an example, some drives through a particular section of road may have encountered many blockages, or the vehicle 1500 that performed the drive was not equipped with a certain sensor modality. As such, the mapstream campaign may instruct the vehicle 1500 to collect data corresponding to previously blocked data and / or generate data for the missing modality.As another example, a mapstream campaign may be generated to more accurately identify lane markings, signs, traffic lights, and / or other information, and the instruction to the vehicle 1500 may be to generate sensor data 102 and / or output 204 that can be used to generate or update an HD map with this information. As such, a DNN 202 that calculates information regarding lane markings, signs, traffic lights, etc., can be run using the respective sensor data type, and the output 204 - e.g., after post-processing, data conversion, compression, etc. - can be sent via mapstream 210 to the cloud for map creation.

[0032] In some embodiments, the map stream campaign may be part of a map health check 112 - for example, after an HD map has been generated and used for localization. For example, if a discrepancy is detected between the HD map represented by map data 108 and current sensor data or DNN detection, the health checker may trigger the vehicle 1500 to generate and / or upload new map stream data for its location. For example, in some embodiments, the vehicle 1500 may be generating data for the map stream 210 and not uploading the map stream 210, while in other embodiments, the vehicle 1500 may simply generate and upload map stream data when triggered. As such, if localization to the map results in poor planning and / or control actions, the map stream 210 may be uploaded to the cloud to update the map information throughout the map creation process. As a result, the map may not be updated periodically, but only when localization errors, planning errors, and / or control errors are detected.

[0033] In some instances, the system may minimize how often a trajectory point or frame is generated and / or included in the map stream 210. For example, instead of including every frame in the map stream 210, a distance threshold, a time threshold, or a combination thereof may be used to determine which frames should be included in the map stream 210. As such, a trajectory point or frame may be included in the map stream if a certain distance (e.g., 0.5 meters, 1 meter, 2 meters, 5 meters, 10 meters, etc.) has been traveled by the vehicle 1500 and / or a certain amount of time has passed (e.g., 0.5 seconds, 1 second, 2 seconds, etc.). This distance or time threshold may be used based on which was met first or which was met last. For example, the first frame may be included in the map stream 210, when the distance threshold may be met, and the second frame at the distance threshold may be included in the map stream 210. Once the second frame is included, the distance and time thresholds may be reset, at which time the time threshold may be met, a third frame may be included in the map stream 210, etc. As a result, less duplicated data may be included in the map stream 210.

[0034] For example, if the vehicle 1500 is at a traffic light for 30 seconds, the time threshold is 1 second, and the frame rate is 30 frames per second (fps), instead of including 900 (e.g., 30*30) frames in the map stream 210, only 30 frames may be included in the map stream 210. In some embodiments, after the vehicle 1500 has not moved for a threshold time (e.g., 2 seconds, 4 seconds, etc.), frame generation and / or inclusion of frames in the map stream 210 may be suspended (e.g., until motion is detected). As another non-limiting example, if the vehicle 1500 is traveling at a speed of 1 meter / second (or 2.24 miles / hour), the distance threshold is 2 meters, the frame rate is 30 fps, and instead of including 60 (e.g., 30*2) frames in the map stream 210 during a distance of 2 meters, only a single frame may be included in the map stream 210. As a result, the amount of data to be sent from the vehicle 1500 to the cloud for map creation 106 is reduced, but does not affect the accuracy of the map creation process - for example, because at least some of the data may be redundant or simply incrementally different and therefore may not be necessary for accurate map creation.

[0035] In addition to or as an alternative to sending less data in the map stream 210 (e.g., minimizing the amount of data), in an embodiment, the data may be compressed - e.g., to reduce bandwidth and reduce run time. In some embodiments, extrapolation and / or interpolation may be used to determine the points or frames, such that fewer points or frames (e.g., rotation / translation information for the points or frames) need to be transmitted in the map stream 210, and extrapolation and / or interpolation may be used to generate additional frames or points. For example, rotation and translation information may be data intensive - e.g., requiring many bits to fully encode (x,y,z) position information and x-axis, y-axis, and z-axis rotation information - so the fewer points along the trajectory that require rotation and / or translation information encoded therein, the less data that needs to be transmitted. As such, the history of the trajectory may be used to extrapolate future points in the trajectory. As another example, additional frames between frames may be generated using interpolation. As an example, if the trajectory corresponds to the vehicle 1500 driving straight at a substantially constant speed, then the first and last frames of the sequence may be included in the map stream 210, and frames in between may be interpolated from the first and last frames of the sequence of frames. However, if the velocity changes, more frames may need to be encoded in the map stream 210 to more accurately linearly interpolate where the vehicle 1500 was at a particular time. In such an instance, the system may use more frames than in a straight, constant speed driving instance, although it may still not use all the frames. In some embodiments, cubic or cubic polynomial interpolation may be used to encode derivatives that can be used to determine the rate of change of velocity, and thus can be used to determine - or interpolate - other points along the trajectory without having to directly encode them into the map stream 210.As such, instead of encoding the rotation, translation, position, and / or pose in each frame, the encoded interpolated and / or extrapolated information may instead be used to recreate additional frames. In some embodiments, the frequency of trajectory data generation may be adaptively updated to achieve a maximum error in accuracy of the pose used in map generation 106 that is interpolated between two other frames or time steps compared to the correct pose known at the time of data reduction.

[0036] As another example, delta compression may be used to encode the difference or delta between the current position of the vehicle 1500 (e.g., pose and position) and a previous position of the vehicle 1500. For example, 64 bits may be used to encode each of the latitude, longitude, and altitude or x, y, and z coordinates of the vehicle 1500 in a frame for a total of 192 bits. However, as the vehicle 1500 moves, these values ​​may not change much from frame to frame. As a result, rather than using the full 64 bits for each of the latitude, longitude, and altitude in each frame, the encoded value of a frame may instead correspond to the difference from the previous frame, which may be encoded in fewer bits. For example, in an embodiment, the difference may be encoded using 12 bits or less, thereby producing significant memory and bandwidth savings for the system. Delta compression encoding may be used for local layout relative coordinates and / or global coordinates.

[0037] In addition to or as an alternative to compressing position, pose, rotation, and / or translation information, sensor data 102 and / or output 204 may be compressed for inclusion in map stream 210. For example, for the position, pose, dimensions, and / or other information regarding lanes, lane markers, lane dividers, signs, traffic lights, waiting conditions, static and / or dynamic objects, and / or other output 204, the data may be delta encoded, extrapolated, interpolated, and / or otherwise compressed to reduce the amount of data included in map stream 210. For sensor data 102, such as LiDAR data, RADAR data, ultrasonic data, and / or the like, the points represented by the data may be voxelized such that duplicate points are removed and instead, a volume is represented by the data. This may allow a lower density of points to be encoded in map stream 210 while still containing enough information for accurate and detailed map creation 106. For example, RADAR data may be encoded using octrees. Additionally, quantization of RADAR points and / or points from other sensor modalities may be performed to minimize the number of bits used to encode the RADAR information. For example, because a RADAR point may have an x ​​and y location, the geometry of lane dividers, signs, and / or other information may be used - in addition to the x and y location - to encode the RADAR point using fewer bits.

[0038] In some embodiments, a buffer or other data structure may be used to encode the map stream data as serialized structured data. As such, instead of having a single field of the data structure describing some amount of information, bit packing into a byte array may be performed - for example, such that the data in the buffer only contains numbers rather than field names to achieve bandwidth and / or storage savings compared to a system in which the data includes field names). As a result, a schema may be defined that associates field names with data types, using integers to identify each field. For example, an interface description language may be used to describe the structure of the data, and a program may be used to generate source code from the interface description language to generate or parse a stream of bytes representing the structured data. As such, a compiler may generate an application programming interface (API) - for example, cAPI, pythonAPI, etc. - that receives the file and instructs the system on how to consume the information from the pin file encoded in this particular manner. Additional compression can be achieved by chunking the data into smaller - e.g., 10,000 byte - chunks, converting the data to a byte array, and then sending or uploading it to the cloud for map creation 106.

[0039] In some examples, dynamic obstacle removal may be performed on LiDAR data, RADAR data, and / or data from other sensor types to remove or filter out sensor data 102 corresponding to dynamic objects (e.g., vehicles, animals, pedestrians, etc.). For example, different frames of sensor data 102 may be compared - e.g., after self-motion compensation - to determine points that are not consistent across frames. In such an instance, for example, if a bird may fly across the perception field of the LiDAR sensor, points corresponding to a detected bird in one or more frames may not be present in one or more previous or subsequent frames. As such, these points corresponding to detected birds may be filtered out or removed - during map creation 106 - such that these points are not used during generation of the final LiDAR layer of the HD map.

[0040] As such, the sensor data 102 may be processed by the system to generate outputs corresponding to image data, LiDAR data, RADAR data, and / or trajectory data, one or more of which may undergo post-processing (e.g., sensory outputs may undergo fusion to generate a fused sensory output, LiDAR data may undergo dynamic obstacle filtering to generate filtered LiDAR data, etc.). The resulting data may be aggregated, merged, edited (e.g., trajectory completion, interpolation, extrapolation, etc.), filtered (e.g., landmark filtering to create continuous lane and / or road boundaries), and / or otherwise processed to generate a map stream 210 representing this data generated from any number of different sensors and / or sensor modalities.

[0041] Referring now to FIG. 3, each block of method 300 described herein includes computational processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 300 may also be implemented as computer usable instructions stored on a computer storage medium. Method 300 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 300 is described with respect to process 104 of FIG. 2, by way of example. However, method 300 may additionally or alternatively be performed in any process by any system, or in any combination of processes and systems, including, but not limited to, those described herein.

[0042] 3 is a flow diagram illustrating a method 300 for map stream generation according to some embodiments of the present disclosure. The method 300 includes, at block B302, generating sensor data using a sensor of the vehicle. For example, the sensor data 102 may be generated.

[0043] The method 300 includes applying at least the first subset of sensor data to a DNN at block B304. For example, the sensor data 102 may be applied to the DNN 202.

[0044] The method 300 includes computing, using the DNN, an output at block B306. For example, the DNN 202 can compute the output 204, which in one or more embodiments can include, but is not limited to, lane divider information, road boundary information, static object information, dynamic object information, waiting condition information, intersection information, signs, poles, or traffic light information, and / or other information corresponding to objects—static and / or dynamic—in an environment of the vehicle 1500.

[0045] The method 300 includes transforming at least the first subset of outputs into a 3D coordinate system having the vehicle as an origin to generate transformed outputs at block B308. For example, the data converter 206 can transform the outputs 204 into a 3D coordinate system having the vehicle 1500 origin as an origin.

[0046] The method 300 includes, at block B310, transforming at least the second subset of sensor data into a 3D coordinate system to generate transformed sensor data. For example, at least some of the sensor data 102 may be used directly in the map stream 210 but may not have been generated in a 3D coordinate system related to an origin of the vehicle 1500. As such, the sensor data 102 may be transformed into a 3D coordinate space having the vehicle 1500 at the origin.

[0047] The method 300 includes compressing and / or minimizing the sensor data, the transformed sensor data, the transformed output, and / or the output of the second subset to generate compressed data at block B312. For example, the sensor data 102 and / or the output (e.g., either without transformation or after transformation if not generated in 3D coordinate space relative to the vehicle 1500) may be compressed and / or minimized. The compression and / or minimization may be performed using any known techniques, including but not limited to those described herein.

[0048] The method 300 includes encoding the compressed data, the sensor data, the transformed sensor data, the transformed output, and / or the output of the second subset to generate a map stream at block B314. For example, the sensor data 102 (with or without the transform) and / or the output 204 (with or without the transform) - e.g., after compression by the data compressor 208 - may be encoded to generate a map stream 210. The map stream 210 may then be transmitted to the cloud for map creation 106.

[0049] 3 may be repeated for any number of trips (or segments thereof) of any number of vehicles 1500. Information from each map stream 210 may then be used for map creation 106.

[0050] Map Creation 4, which is a data flow diagram of a process 106 of map creation according to some embodiments of the present disclosure. In some embodiments, the process 106 may be performed in the cloud using computing devices (e.g., similar to the example computing device 1600 of FIG. 16) of one or more data centers, such as the example data center 1700 of FIG. 17. In some embodiments, the process 106 may be performed using one or more virtual machines, one or more separate computing devices (e.g., servers), or a combination thereof. For example, a virtual graphics processing unit (GPU), a virtual central processing unit (CPU), and / or other virtual components may be used to perform the process 106. In some embodiments, one or more of the process steps described with respect to the process 106 may be performed in parallel using one or more parallel units. For example, the registration 402 of pairs of partitions, such as cost space sampling, aggregation, etc., may be performed in parallel (e.g., a first pair may be registered in parallel with another pair). Additionally, within a single registration, a pose sampled to update the cost of a point in the cost space may be performed in parallel with one or more other poses. Additionally, map data corresponding to individual map layers may be stored in the GPU as textures so that texture lookups may be performed to quickly determine cost values ​​in the cost space - thereby reducing the runtime of each cost space analysis.

[0051] The map creation process 106 may include receiving a map stream 210 from one or more vehicles 1500 corresponding to any number of maneuvers. As described herein, each map stream 210 may include various layers of data generated using a variety of different methods, such as by tracking ego-motion (e.g., relative and global), sensor data 102 generation and processing, perception using one or more DNNs 202, etc. Each layer of the map stream 210 may correspond to a series of frames corresponding to sensor events recorded at a variable frame rate. For example, the map stream 210 layers may correspond to a camera layer, a LiDAR layer (e.g., layers of different slices), a RADAR layer, a trajectory (or ego-motion) layer, and / or other layers. The camera layer may include information obtained by performing perception (e.g., via the DNN 202) on a stream of 2D camera images and converting the (2D and / or 3D) detection or output 204 of the DNN 202 into 3D landmarks and paths (e.g., by combining lane markings to define lane boundary and road boundary positions). The LiDAR and / or RADAR layer (or other sensor modality layer) may correspond to point cloud information collected using the LiDAR sensor 1564 and / or the RADAR sensor 1560, respectively. As described herein, during the map stream generation process 104, pre-processing may be performed on the LiDAR and / or RADAR data, e.g., data reduction, ego-motion compensation, and / or dynamic obstacle removal. The trajectory layer may include information corresponding to the absolute position of the origin or rig of the vehicle 1500 and the relative ego-motion over a certain time frame.

[0052] 5A, map creation process 106 may include, for each map stream 210(1)-210(N)--where N corresponds to the number of map streams 210 used for a particular registration process--converting map stream 210 to maps 504(1)-504(N) via transformations 502(1)-502(N) (e.g., converting the map stream to a maneuver map format). For example, with reference to FIG. 5B, for a single map stream 210(1), transformations 502(1) may include base transformation 506, RADAR transformation 508, LiDAR elevation slicing 510, LiDAR transformation 512, RADAR map image creation 514, LiDAR map image creation 516, and / or LiDAR voxelization using LiDAR voxelizer 518. Base transform 506 may correspond to landmarks—e.g., lane lines, road lines, signs, poles, trees, other vertical structures or objects, crosswalks, etc.—as determined using perception via DNN 202. For example, 3D landmark locations may be transformed—using base transform 506—to a map format to generate base layer 520 (or “camera layer” or “perception layer”) of map 504(1). In addition to landmark locations, base layer 520 may further represent the trajectory or path (e.g., global or relative) of vehicle 1500 that generated map stream 210(1). When generating base layer 520, a 1:1 mapping between aggregated input frames of map stream 210(1) and output base layer 520 map road segments may be maintained.

[0053] RADAR data from map stream 210(1)—e.g., when received or accessed in raw format—may be converted to RADAR point cloud layer 522 via RADAR transform 508. In some embodiments, the RADAR point cloud from RADAR point cloud layer 522 may be used to generate RADAR map image layer 524 via RADAR map image creation 514. For example, the RADAR point cloud may be converted to an image of the RADAR point cloud from one or more different perspectives (e.g., top-down, sensor perspective, etc.). For example, a virtual camera with a top-down field of view may be used to project the RADAR point cloud into the frame of the virtual camera to generate a RADAR map image in RADAR map image layer 524.

[0054] LiDAR data from map stream 210(1)—e.g., when received or accessed in raw format—may be transformed into LiDAR point cloud layer 526 via LiDAR transform 512. In some examples, as described herein, the LiDAR data may be generated in slices (e.g., ground slices (e.g., 5 meters to 300 meters relative to the origin of vehicle 1500), giraffe plane slices (e.g., 2.5 meters to 5 meters relative to the origin of vehicle 1500), ground plane slices (e.g., −2.5 meters to 0.5 meters relative to the origin of vehicle 1500), etc.). In addition to or as an alternative to LiDAR elevation slicing in map stream generation process 104, LiDAR elevation slicing 510 may be performed during transform 502(1) to determine separate slices of LiDAR data for use in generating one or more LiDAR point cloud layers 526. For example, LiDAR data corresponding to a particular slice, e.g., a Giraffe slice, may be used to generate LiDAR point cloud layer 526 of map 504(1). In some examples, the LiDAR point cloud from LiDAR point cloud layer 526 may be used to generate LiDAR map image layer 528 via LiDAR map image creation 516. For example, the LiDAR point cloud, e.g., corresponding to a particular slice, may be converted into images of the LiDAR point cloud from one or more different perspectives (e.g., top-down, sensor perspective, etc.). For example, a virtual camera with a top-down field of view may be used to project the LiDAR point cloud into the frame of the virtual camera to generate a LiDAR map image in LiDAR map image layer 528. LiDAR map image layer 528 may include a LiDAR map image encoded with elevation values ​​determined from the point cloud (e.g., a top-down depth map), a LiDAR map image encoded with intensity values ​​determined from the point cloud, and / or other LiDAR map image types. In some examples, the LiDAR point cloud layer 526 may be used—for example, by the LiDAR voxelizer 518—to generate a LiDAR voxel map layer 530 of the map 504(1).The LiDAR voxel map layer 530 may represent a voxelized representation of the LiDAR point cloud, which may be used for editing the LiDAR data in the individual maps 504 and / or in the fused HD map (as described in more detail herein), for example to filter out dynamic objects.

[0055] 5A , the process described with respect to map stream 210(1) may be performed for each map stream 210(1)-210(N) to generate maps 504(1)-504(N). Maps 504(1)-504(N), or a subset thereof, may then be used for registration 402. As such, after maps 504 are generated for each map stream 210 to be used in the current registration process, registration 402 may be performed to generate an aggregate map corresponding to multiple map streams 210. The aggregate map may include aggregated layers, such as an aggregate camera or base layer, an aggregate LiDAR layer, an aggregate RADAR layer, etc. To determine which map streams 210 (and thereby which maps 504) should be registered together, the location of the map streams 210 - or a segment thereof - may be determined. 5C , a first map stream segment 540 (e.g., representing the trajectory of vehicle 1500 that generated map stream 210(1)) may correspond to first map stream 210(1), a second map stream segment 542 may correspond to second map stream 210(2), and a third map stream segment 544 may correspond to third map stream 210(3). Position or trajectory information, e.g., GNSS data, may be used to determine that map streams 210(1)-210(3) correspond to similar positions or road sections. For example, GNSS coordinates from map stream 210 and / or map 504 may be used to determine whether map streams 210--or sections thereof--are close enough in space over a long enough distance to be registered with one another. Once localized using GNSS coordinates, the relative coordinates from map streams 210(1)-210(3) may be used to determine how close the map stream segments are (e.g., how close the trajectory of vehicle 1500 is). This process may result in a final list of map stream sections to be registered.For example, map stream segments 540, 542, and 544 may be determined to be on the same road - or to have at least an overlap between demarcations 550A and 550B - using GNSS coordinates and / or relative coordinates from map stream 210 and / or map 504.

[0056] In some instances, an individual map stream 210 or maneuver—e.g., map stream 210(1) corresponding to map stream segment 540—may include a loop. For example, during map stream generation process 104, vehicle 1500 may have traversed the same road segment at two or more different times. As a result, map stream segment 540 may be split—via de-duplication—into two separate segments 552A and 552B and treated as separate map stream segments for registration 402. The de-duplication process may be performed as a pre-processing step to identify when vehicle 1500—or its corresponding map stream 210—has been driven in a circle or loop. After loops are identified, the maneuver or map stream 210 may be split into two or more sections to ensure that each physical location is represented only once.

[0057] A layout of all map stream segments may be generated, including map stream segments determined to be within some spatial threshold relative to one another. For example, FIG. 5C may represent an example visualization of the layout of map stream segments 540, 542, and 544. In the example of FIG. 5C, overlapping portions 552A-552D, e.g., between demarcations 550A and 550B, may be used for registration 402. The layout may then be used to determine which pairs of overlapping portions 552A-552D to register together. In some implementations, a minimum spanning tree algorithm may be used to determine the minimum number of portions 552 that will be registered together to have connectivity between the respective portions 552 (e.g., if portions 552A and 552B are registered together, and portions 552B and 552C are registered together, then portions 552A and 552C have connectivity through portion 552B). The minimum spanning tree algorithm may involve randomly selecting a pair of portions 552, then randomly selecting another pair of portions, until the minimum spanning tree is complete. In some embodiments, in addition to the minimum spanning tree algorithm, a margin of error may be used to include additional pairs for robustness and accuracy.

[0058] As an example, with reference to FIG. 5D, table 560 illustrates different numbers of connections or pairs between map streams or map stream sections using a variety of different techniques. The number of segments, M, may correspond to the total number of map streams or map stream sections available for registration 402 for a particular segment (e.g., segments 552A-552D of FIG. 5C). Total connections may correspond to the number of pairs that would be registered if registration was performed for each possible connection between each pair of segments. As an example, the following equation (1) may be used to calculate total connections: Total connections = M*(M-1) / 2 (1)

[0059] As another example, to calculate the minimum spanning tree connection, the following equation (2) may be used: Minimum spanning tree connection = M-1 (2)

[0060] As a further example, to calculate a safety margin connection, e.g., a minimum spanning tree connection plus a safety margin, the following equation (3) may be used: Safety margin connections = MIN [(minimum spanning tree connections * 2), all connections] (3)

[0061] As such, and referring also to Figure 5C, using equation (3) with respect to portions 552A-552D, connections 554A-554F can be determined. For example, since there are four portions or segments to be registered, a safety margin would result in six connections being made between the four portions or sections of the map stream section between demarcations 550A and 550B.

[0062] In some embodiments, the quality of the sections may be analyzed to determine which of the sections to use and / or which sections to use more frequently (e.g., if some sections are registered more than once). For example, the quality may correspond to the number of sensor modalities represented in the map stream section. As such, if one section includes cameras, tracks, RADAR, and LiDAR, and another section includes only cameras, tracks, and RADAR, the section with LiDAR may be weighted or favored such that the section is more likely to be included in more registrations 402. Additionally, in some embodiments, geometric distance may be used as a criterion when determining which sections to register together. For example, two sections that are geometrically closer may be selected over sections that are geometrically farther apart - e.g., because the closer located sections may correspond to the same travel lane as opposed to sections that are farther apart that may correspond to different travel lanes or opposite sides of the road.

[0063] Referring again to FIG. 4 and FIG. 5A, pairs of sections may then be registered with each other to generate pose links between them that may be used for pose optimization 404. Registration 402 may be performed to determine the geometric relationships between the corresponding multi-pole maneuvers or map streams 210 at multiple different locations where the maneuvers or map streams 210 overlap, e.g., portions 552A-552D of FIG. 5C. The output of the registration process may include relative pose links between pairs of frames or sections in different map streams (e.g., rotations and translations between the pose of a first map stream frame or section and the pose of a second map stream frame or section). In addition to the relative poses, the pose links may further represent covariances that represent the confidence in the respective pose links. The relative pose links are then used to align the maps 504 corresponding to each of the map streams 210 so that landmarks and other features, e.g., point clouds, LiDAR image maps, RADAR image maps, etc., are aligned in the final, aggregated, HD map. As such, the registration process is performed by localizing one map 504--or a portion thereof--of the pair to the other map 504--or a portion thereof. Registration 402 may be performed for each sensor modality and / or for each map layer corresponding to different sensor modalities. For example, camera-based registration, LiDAR-based registration, RADAR-based registration, and / or other registrations may be performed separately. Results may include an aggregated base layer of HD maps, an aggregated LiDAR point cloud layer of HD maps, an aggregated RADAR point cloud layer of HD maps, an aggregated LiDAR map imagery layer of HD maps, etc.

[0064] The localization process used to localize one section to another for registration may be performed similarly to the localization process 110 used in live perception, as described in more detail herein. For example, a cost space may be sampled for a single frame or pose, an aggregate cost space may be accumulated across frames or poses, and a Kalman filter may be used on the aggregate cost space to localize one section relative to another. As such, the pose from the trajectory of a first section may be known, and the pose of the trajectory from a second section may be sampled relative to the first section using localization to determine the relative pose between the two - once alignment between the landmarks is achieved. This process may be repeated for each of the poses of the sections such that pose links between the poses are generated.

[0065] For example, with respect to the base layer or perception layer of the map 504, during registration 402, the 3D world space positions of the landmarks may be projected into the virtual field of view of a virtual camera to generate an image corresponding to the 2D image space positions of the landmarks in the virtual image. In some implementations, the 3D world space landmark positions may be projected into the field of view or into multiple virtual cameras to generate images from different perspectives. For example, a front-facing virtual camera and a rear-facing virtual camera may be used to generate two separate images for localization in the registration process. Using more than one image may increase the robustness and accuracy of the registration process. To register to the image, the virtual image space positions of the landmarks from the two sections undergoing registration 402 may be compared to each other - e.g., using a camera-based localization technique - to sample a cost space representing the positions of the matches between the sections. For example, the detections from the first section may be converted to a distance function image (e.g., as described herein with respect to localization 110) and the detections from the second section may be compared to the distance function image to generate a cost space. In addition to comparing geometries, semantic information may also be compared - for example, simultaneously or in separate projections - to calculate costs. For example, if the semantic information - for example, lane marking type, pole, sign type, etc. - does not match for a particular point, the cost of that particular point may be set to the maximum cost. The geometric and semantic costs may then be used together to determine a final cost for each pose. This process may be performed over any number of frames to determine an aggregate cost space that is finer than any individual cost space, and a Kalman filter (or other filter type) may be performed on the aggregate cost space to determine the poses - and pose links - between the two segments.

[0066] 5E and 5F, which show an example of camera or base layer based registration according to some embodiments of the present disclosure. For example, visualization 570A in FIG. 5E may correspond to a front-facing virtual camera registration, and visualization 570B in FIG. 5F may correspond to a rear-facing virtual camera. 3D landmark positions from the base layer of the first map 504 may be projected into the 2D image space of the front-facing and rear-facing virtual cameras. The 2D projections may then be converted into a distance function - shown as a dotted pattern in visualizations 570A and 570B - where the centerline of each dotted section 572A-K may correspond to zero cost, and as the points move away from the centerline of the section, the cost increases up to a white area, which may correspond to maximum cost. The dotted sections 572 may correspond to the distance function equivalent of the landmarks. For example, dotted segments 572A-572C and 572J-572K may correspond to lane and / or road markings, and dotted segments 572D-572I may correspond to poles, signs, and / or other static objects or structures.

[0067] The distance function may then be compared to the 2D projected 3D landmark information from the base layer of the second map 504. For example, the solid black segment 574 (e.g., including lines and / or dots) may represent a 2D projection of the 3D landmarks of the second map 504 at the sampled pose 576. As such, this comparison of the 2D projection with the distance function projection of the first map 504 may correspond to a single location on the cost space that corresponds to the pose 576, and any number of other poses may also be sampled to fill the cost space. As such, for example, each point along the solid black segment 574 may have a relatively low cost since many of the points match points along the dotted segment 572A. In contrast, each point along the solid black segment 574J may have a high cost since many of the points do not match points along the dotted segment 572J. Finally, for pose 576, a cost corresponding to each of the points in solid black partitions 574A-574K may be calculated (e.g., using an average value) and the point in the cost space corresponding to pose 576 may be updated to reflect the calculated cost. This process may be repeated for any number of poses (e.g., each possible pose) until the most likely relative pose of the base layer of second map 504 with respect to the base layer of first map 504 has been determined for a particular time step or frame. This process may then be repeated for each time step or frame of the pair of co-registered sections (e.g., with respect to FIG. 5C, for each corresponding time step or frame from portions 552A and 552B, for each corresponding time step or frame from portions 552C and 552D, etc.).

[0068] As another example, for a LiDAR point cloud layer of map 504, during registration 402, the LiDAR point cloud from a first section may be converted to a distance function point cloud and compared to the LiDAR point cloud of a second section to generate a cost space. For example, costs may be sampled at each possible pose of the second section with respect to the distance function image corresponding to the first section. This process may be performed similarly for LiDAR intensity map images, LiDAR elevation map images, and / or other LiDAR image types. Similarly, registration 402 may be performed in this manner for RADAR.

[0069] The result of registration 402 between any pair of sections is a pose link that defines the rotation, translation, and / or covariance between the poses of the sections. This registration process is performed for each determined pair from the connections, e.g., safety margin connections 554A-554F in FIG. 5C. Using the output of the registration process, a pose graph can be generated that represents the pose links of the time stamp based relative positions, or poses, of different maneuvers or sections.

[0070] 4 and 5A, after the registration process 402 is completed and pose links are generated between registered pairs of frames or poses, pose optimization 404 may be performed to smooth the pose links and separate groups of poses from various different maneuvers into segments corresponding to road segments used for relative localization in the final aggregate HD map. Pose optimization 404 may be performed to obtain a more desirable or optimal geometric alignment of the sub-maps or layers based on the input absolute poses from the map stream 210, the relative pose links generated during registration 402, and the relative trajectory pose links of each individual maneuver generated using ego-motion. Before pose optimization, a pose graph may be generated to represent the absolute poses and the relative poses. In some embodiments, a filtering process may be performed to remove poses with low confidence, for example, using covariance as determined during registration process 402. The (filtered) frames or poses may be clustered into road segments that may differ from the individual input map 504 segments - for example, because the same area only needs to be represented by a single road segment rather than multiple overlapping road segments. Each output road segment may be associated with an absolute pose, or origin, during the pose optimization process 404. Given the initial pose graph layout, optimization may be performed to minimize the pose error of the observed relative poses and absolute poses relative to the input absolute pose, while taking into account the uncertainty information associated with each observation (e.g., maximum likelihood optimization). The optimization may be initialized using a random sample consensus (RANSAC) process to find a maximally consistent subset of pose links measured via a periodic consistency check over multiple periods in the pose graph. Once optimized, during localization 110, each road segment may be compared to connected road segments to determine the relative transformation between the road segments, so that the cost function calculation values ​​may be transferred from road segment to road segment using the transformation so that an accurate accumulated cost space may be calculated.

[0071] 6A-6I show an example pose optimization process 404 corresponding to four maneuvers or segments from the same and / or different maps 504. In some embodiments, the pose optimization process 404 may be performed separately for different sensor data modalities or layers of the maps 504. For example, base layers of any number of maps 504 may be registered together and then pose optimized, LiDAR point cloud layers of any number of maps 504 may be registered together and then pose optimized separately from the pose optimization of the base layer, etc. In other embodiments, results of the registration process of different layers of the maps 504 may be used to generate aggregate poses and pose links, and the pose registration process may be performed in aggregate.

[0072] With reference to FIG. 6A, registered segments from four different maneuvers 602A-602D are shown in frame graph 600A. Frame graph 600A is an example of a portion of a larger frame graph, where the larger frame graph may correspond to any portion—or all portions—of a road structure or layout. The four maneuvers 602A-602D may be registered together to generate pose links 608 (e.g., pose links 608A-608C) between poses 604 or positions of the different maneuvers 602. Each pose 604 or position of a single maneuver 602 may be represented by a link 606. For example, link 606 may represent translation, rotation, and / or covariance from one pose 604 to the next pose 604. As such, link 606A(6) may represent rotation, translation, and / or covariance between the poses 604 of maneuvers 602A that link 606A(5) connects. Pose link 608A may correspond to the output of registration process 402 and may encode translation, rotation, and / or covariance between different driving 602 poses 604. For example, pose link 608A may encode six degree of freedom transformations between poses, such as translation (e.g., (x,y,z) position difference), rotation (e.g., x,y, and z axis angle difference), and / or covariance (e.g., corresponding to confidence in the value of the pose link) between driving 602A pose 604A(1) and driving 602B pose 604B(2).

[0073] FIG. 6B illustrates road segment generation, whereby a group of poses from any number of maneuvers 602 may be combined into a single road segment. The resulting road segments may be included in a final aggregate HD map that is used for local or relative localization. Additionally, the generated road segments may have an origin, which may have a relative position in a global coordinate system, e.g., the WGS84 coordinate system. Frame graph 600B (which may correspond to frame graph 600A but may have road segment identification) illustrates how a frame graph may be divided into different road segments 610A-610D. For example, poses with horizontal striped fills may correspond to the first road segment 610A, poses with vertical striped fills may correspond to the second road segment 610B, poses with dotted fills may correspond to the third road segment 610C, and poses with shaded fills may correspond to the fourth road segment 610D. To determine which poses 604 or frames will be included in each road segment, a first random pose 604, e.g., pose 604A(6), may be selected. After a random pose 604 is selected, the links 606 and / or pose links 608 may be iterated from pose 604 to pose 604, starting with the randomly selected pose 604, until a maximum distance corresponding to a maximum distance of the road segment 610 is reached (e.g., 25 meters, 50 meters, etc.) or a pose 604 or frame already encoded in the road segment 610 is determined. After the road segment 610 is fully encoded, another random unencoded pose 604 may be selected and the process may be repeated. This may be repeated until each pose 604 is encoded or included in the road segment 610. Due to the stochastic nature of this process, some very small road segments 610 may be created. As such, post-processing algorithms may be performed to analyze road segment sizes and to merge road segments below a certain size threshold into adjacent road segments.

[0074] FIG. 6C illustrates a segment graph 620 of road segments 610A-610D after the road segment encoding or generation process. For example, each block labeled 610A-610D may correspond to a collapsed representation of all the poses 604 corresponding to the respective road segment 610. After the road segments 610 are determined, an origin or seed position for each road segment may be determined. The origin or seed position may correspond to the center of the road segment 610 (e.g., where the road segment is 50 meters by 50 meters long, the origin may be (25m, 25m)). In some embodiments, the origin may correspond to the average or median position of each pose 604 within the road segment 610. For example, the (x,y,z) coordinates corresponding to each pose 604 may be averaged and the result may be selected as the origin. In other embodiments, the origin may be selected using a different method.

[0075] FIG. 6D illustrates a pose-link error correction process. For example, pose-link transformations between poses in a frame graph may not be guaranteed to be consistent with each other. As such, when three poses 604 are taken - as in FIG. 6D - pose-link errors 612 may be present. To minimize the pose-link errors 612 within a road segment, a configuration of poses within a road segment 610 with the lowest pose-link errors 612 may be determined. In one or more embodiments, the layout of poses determined during pose optimization 404 may not be globally consistent, but may be optimized such that the layout is consistent within the same road segment 610 and adjacent road segments 610. For example, pose links 614 and 614B should be approximately the same, and pose link 614C should be approximately equal to the combination of pose links 614A and 614B. However, in practice, due to some errors, such as localization errors during registration 402, there may be some delta or pose link errors 612 that appear between the poses 604. As a result, pose optimization 404 may be used to move or shift the poses 604 or frames such that the best fit is achieved within the corresponding road segment 610. With respect to pose links 614A and 614B, the pose 604 may be shifted to distribute the errors among all pose links 614, instead of having the pose link error 612 appear primarily with pose link 614C.

[0076] FIG. 6E illustrates a pose graph 630 that may represent the frame graph 600B of FIG. 6B, but omitting the pose links connecting the road segments 610A-610D to the outer road segment 610. FIG. 6F illustrates a RANSAC operation performed on the pose graph 630. This process may be performed based on confidence that the accuracy of the links 606 between the poses is more accurate than the pose links 608. For example, a minimum set of pose links 608 - or minimum spanning tree - in the pose graph 630 that preserves the connectivity of all nodes or poses 604 may be sampled. After the minimum set is selected, the remaining pose links 608 may be sampled to ascertain how many pose link errors 612 have been detected. This may correspond to one iteration, and in the next iteration, another random minimum set of pose links 608 may be selected, and then all remaining pose links 608 may be sampled for pose link error 612, etc. for a number of iterations (e.g., 100, 1000, 2000, etc.). After that number of iterations are completed, the layout with the most matches may be used. For example, with respect to FIG. 6F, a cost of error may be calculated for each pose link 608 other than the sampled minimum - for example, according to the following equation (4):

number

[0077] Referring to FIG. 6G, FIG. 6G illustrates an updated pose graph 650 after the RANSAC process is completed - for example, pose graph 650 illustrates the pose configuration with the most inliers. Pose graph 650 of FIG. 6G may be subjected to an optimization process, for example, a nonlinear optimization process (e.g., a bundle adjustment process) that aims to minimize the sum of the squared costs of the inliers - for example, using a cost function calculated as described herein with respect to FIG. 6F. After bundle adjustment, pose graph 650 may be fixed and road segments 610A-610D may be fixed such that pose 604 may be in their final fixed positions of road segment 610 - for example, as shown in FIG. 6H with example road segment 610C. As described herein, final road segment 610 may have a relative origin (e.g., for localization with respect to a local or relative coordinate system). As such, since the relative origin has a position in the global coordinate system, then localization to the local coordinate system may also localize to the global coordinate system. For example, with respect to Figure 6H, road segment 610C may have an origin 652C (as shown). Although not shown, the other road segments 610A, 610B, and 610D may also have respective origins 652.

[0078] 6I, each road segment 610 may have a relative transformation calculated for each immediately adjacent road segment 610. In the illustration of FIG. 6I, road segment 610C has a relative transformation (e.g., transformation, T 610C 610A ), a relative transformation determined between road segment 610C and road segment 610B (e.g., transformation, T 610C 610B ), as well as a relative transformation determined between road segment 610C and road segment 610D (e.g., transformation, T 610C 610D), the aggregate cost space may include one or more cost spaces generated when localizing about, e.g., road segment 610A, and a cost space generated when localizing about, e.g., road segment 610C, so that a relative transformation may be used during localization 110, e.g., more specifically, when generating the aggregate cost function and / or when generating the local map layout used for localization. As such, when entering road segment 610C from road segment 610A, the cost space may include a transformation, T, to update the cost space so that each aggregate cost space references the same origin (e.g., origin 652C of road segment 610C). 610C 610A A relative transformation may represent six degrees of freedom, such as rotation (e.g., the difference between the x, y, and z axis rotation angles) and translation (e.g., the difference between the (x, y, z) coordinates).

[0079] 4, after registration 402 and pose optimization 404, an updated pose graph, e.g., pose graph 650, may be fixed and the relative poses of the multiple maneuvers 602 may be fixed as a result. As such, the maps 504 from each maneuver 602 whose outputs are now aligned, e.g., as a result of registration 402 and pose optimization 404, may be used to fuse together (e.g., via fusion 406) the map layers of the individual maps 504 to form an aggregate HD map. The aggregate HD map may include the same separate layers as the individual maps 504, but the aggregate HD map may include aggregate layers, e.g., an aggregate base layer, an aggregate LiDAR point cloud layer, an aggregate RADAR point cloud layer, and an aggregate RADAR map imagery layer. The fusion process 604 may be used to improve the data quality of the map data with respect to the data present in any single maneuver, e.g., the fused map may be more accurate and reliable than a single map 504 from the map stream 210 of a single maneuver. For example, inconsistencies between maps 504 may be removed, the accuracy of map content may be improved, and a more complete representation of the world may be achieved - for example, by combining the geographic extents of two different map layers or sub-maps, or by combining observations regarding waiting conditions across multiple maneuvers through the same intersection. After the final - e.g., more optimal or desired - geometric layout of poses 604 for different maneuvers 602 has been determined by a pose graph (e.g., pose graph 650), information from multiple maps 504 and / or their corresponding map streams 210 may be readily transformed into the same output coordinate frame or pose associated with each road segment 610.

[0080] As an example, for 3D landmark positions represented in the base layer, 3D landmark positions from multiple maps 504 may be fused together to generate a final representation of each lane boundary, each road boundary, each sign, each pole, etc. Similarly, for LiDAR intensity maps, LiDAR elevation maps, LiDAR range function images, RADAR range function images, and / or other layers of map 504, separate map layers may be fused to generate an aggregated map layer. As such, if a first map layer contains data that matches -within some threshold similarity- with data of another map layer, the matching data may be used (e.g., averaged) to generate the final representation. For example, for the base layer, if the data corresponds to a sign, a first representation of the sign in the first map 504 and a second representation of the sign in the second map 504 may be compared. If the first sign and the second sign are within a threshold distance of each other and / or are of the same semantic class, a final (e.g., average) representation of the sign may be included in the aggregated HD map. This same process may be performed for lane dividers, road boundaries, waiting conditions, and / or other map information. In contrast, if data from a map layer of a map 504 does not match data from other map layers of other maps, the data may be filtered out. For example, if a first map 504 includes data for a lane divider at a certain location and semantic class, and one or more other maps 504 do not share this information, the lane divider from the first map 504 may be removed or filtered out from consideration for the aggregate HD map.

[0081] As described herein, the fusion process 406 may be different for different map layers of the map 504 - or different features represented therein. For example, for lane graph fusion - e.g., fusion of lane dividers, road boundary lines, etc. - individually observed lanes, paths, and / or trajectories from multiple base map layers of multiple maps 504 may be fused into a single lane graph in the aggregate HD map. Lane boundary lines and / or lane dividers of various maps 504 may be fused not only for lane graphs (e.g., as delimiters of lane boundaries in which the vehicle 1500 may travel), but also for camera-based localization as 3D landmarks, or 2D landmarks generated from the 3D landmarks (e.g., as a set of stable semantic landmarks). In addition to lane boundary lines or dividers, other road markings may also be collected from the map stream 210 and included in the map 504 for use in the fusion process 604. For example, stop lines, road text, shoulders, and / or other markings may be fused together for use as semantic landmarks for localization and / or for updating waiting condition information. For waiting conditions in the base layer, waiting conditions from multiple maps 504 may be fused to generate a final representation of the waiting conditions. Poles, signs, and / or other static objects may also be fused from multiple base layers of maps 504 to generate an aggregate representation thereof. Poles, signs, and / or other (vertical) static objects may be used for localization.

[0082] In some embodiments, lane dividers, lane centers (e.g., rails), and / or road boundaries may not be clearly identified in the base layer from the perspective. In such embodiments, trajectory information from the base layer and / or map stream 210 may be used to infer rails and / or lane dividers. For example, after bundle adjustment, the pose graph 650 may be viewed from a top-down view to determine the pattern of trajectories. Trajectories determined to be in the same lane may be used to generate lane dividers and / or rails for that particular lane. However, heuristics may be used to determine that two or more trajectories are from the same lane of travel. For example, the current pose or frame may be compared to determine to cluster two trajectories together as belonging to the same lane. If the current frame or pose of two trajectories appears to match the same lane of travel (e.g., based on some distance heuristic), a pose some distance ahead (e.g., 25 meters) of the current pose of the two trajectories and a pose some distance behind (e.g., 25 meters) of the current pose of the two trajectories may also be analyzed. If the current, previous, and future poses all indicate the same lane of travel, the current poses may be clustered together to determine lane divider, lane rail, and / or road boundary locations. As a result, if one trajectory corresponds to one vehicle 1500 changing lanes while another vehicle 1500 remained stable within its lane, the combination of the two trajectories will result in an inaccurate representation of the actual lane divider, lane rail, and / or road boundary. In some embodiments, in addition to or as an alternative to analyzing the distance between the poses of the trajectories, the angle formed between the trajectories may be analyzed. For example, if the difference in angle is greater than some threshold (e.g., 15 degrees, 40 degrees, etc.), the two trajectories may be considered not to be from the same lane and may not be clustered together.

[0083] With respect to LiDAR and / or RADAR map layers, fusion of LiDAR and RADAR points from multiple maps 504 may ensure that the fused or aggregated map contains a more complete point cloud than any individual map 504 (e.g., due to occlusions, dynamic objects, etc.). The fusion process 604 may also reduce redundancy in the point cloud coverage by pruning to reduce the amount of data required to store and / or transmit the point cloud information of the aggregated HD map. For example, if the point cloud points of one or more drives do not line up with the point cloud points from another drive, the points from the non-matching drives may be removed - e.g., via dynamic object removal. The resulting aggregated map layer for LiDAR may include an aggregated slice (e.g., ground plane slice, giraffe plane slice, etc.) point cloud layer, a LiDAR elevation map image layer that stores the average altitude per pixel, and / or a ground reflectance or intensity map image layer that stores the average intensity value per pixel. For RADAR, the resulting aggregate map layers may include a RADAR cross section (RCS) map image layer and / or a RADAR point cloud layer (which may or may not be sliced). In an embodiment, the in-memory representation of all LiDAR and RADAR map image layers may use floating point representation. For the elevation model, ground reflectance model, and / or RCS, pixels that do not have valid data may be encoded with a not a number (NaN) data type. For the ground reflectance and elevation map image layers, the map image generator may search for peak densities in the elevation distribution of each point. Only these points may be considered as ground detections, and noisy measurements may be filtered out (e.g., measurements corresponding to obstacles). In some embodiments, a median filter may be applied to fill in missing measurements where gaps would otherwise have been present.

[0084] For LiDAR, a voxel-based fusion strategy may be implemented. For example, if a voxel is observed only by a few of many maneuvers, the voxel may be determined to be a noisy detection. In an example, this LiDAR fusion may be performed for each road segment 610. In a non-limiting example, LiDAR fusion may be performed according to the following process: (1) find a rigid 3D bounding box for the LiDAR point; (2) calculate the index of the 3D point given a preset voxel resolution and a 3D point, (x,y,z); (3) use a sparse representation of the 3D volume instead of a dense representation to reduce memory usage; (4) store the ID of the maneuver or map stream 210 that observed the voxel and the average color in the voxel data; and (5) set a threshold for the number of maneuvers in each voxel such that only voxels with a number of maneuvers greater than the threshold are kept after updating the voxel volume for all points in the maneuver (e.g., the threshold may be set to half of all maneuvers 602 in the road segment 610). In addition to removing the noisy 3D points, LiDAR fusion may generate and store point clouds in the Giraffe plane and ground plane slices. The 3D points in the corresponding planes or slices may be used later to generate different kinds of LiDAR map images for LiDAR localization. For example, the Giraffe plane may include points in the range of z=[a1, b1], and the ground plane points may be in the range of z=[a2, b2], where a1, b1, a2, b2 may be preset parameters. Thus, the points are selected based on their z coordinates (altitude) in the frame coordinate system. It should be noted that the filtering process is based on the frame coordinates instead of the road segment coordinates, and therefore the points may not be filtered based on their z coordinates after they are transformed to the road segment coordinate system. To solve this problem, it may be noted that the points belong to the Giraffe plane or ground plane before the points are transformed, and the information may be stored inside a data structure related to the voxel. If a voxel passes the noise filtering process and further belongs to one of the two planes, the voxel can be saved into the corresponding point cloud file.A similar process can be performed for RADAR data.

[0085] In some embodiments, for example, when the fusion process 406 is performed during the health check 112, older LiDAR data, RADAR data, and / or map data may be weighted more negatively compared to more recent data. As such, if a discrepancy is determined between the newer map 504 and the older map 504, the data from the older map 504 may be filtered out and the data from the newer map 504 may be retained. This may be the result of changes in road conditions, construction, and / or the like, and more recent or current data may be more useful for navigation of the road segment 610.

[0086] After the fusion process 406, map data 108, e.g., representing an aggregate HD map including the aggregate layer, may have been generated. The map data 108 may then be used for localization 110, as described in more detail herein.

[0087] Referring now to FIG. 7, each block of method 700 described herein includes computational processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 700 may also be implemented as computer usable instructions stored on a computer storage medium. Method 700 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 700 is described with respect to process 106 of FIG. 4, by way of example. However, method 700 may additionally or alternatively be performed in any process by any system, or in any combination of processes and systems, including, but not limited to, those described herein.

[0088] 7 is a flow diagram illustrating a method 700 for map creation according to some embodiments of the disclosure. The method 700 includes receiving data representing multiple map streams corresponding to multiple maneuvers from multiple vehicles at block B702. For example, map stream 210 may be received for the map creation 106 process, where map stream 210 may correspond to any number of maneuvers from any number of vehicles 1500.

[0089] The method 700 includes, at block B704, transforming each map stream into a respective map including multiple layers to generate multiple individual maps. For example, each received map stream 210 may undergo a transformation 502 to generate maps 504 such that multiple maps 504 (e.g., maps 504(1)-504(N)) are generated. Each map 504 may include multiple layers, such as, but not limited to, the layers described with respect to FIG. 5B.

[0090] The method 700 includes, at block B706, geometrically registering pairs of segments from two or more individual maps to generate a frame graph representing pose links between the poses of the pairs of segments. For example, segments (e.g., portions 552A-552D) of various map streams 210 and / or maps 504 may be determined - e.g., using orbit information and GNSS data indicating proximity - and the segments may be registered with each other (e.g., in an embodiment, using a minimum spanning tree plus a safety margin) to generate a pose graph (e.g., frame graph 600A of FIG. 6A). The pose graph may include pose links between different driving 602 poses or frames. In some embodiments, different registration processes may be performed for different map layers and / or sensor modalities - e.g., frame rates of different sensors may be different, so that the pose of vehicle 1500 in each frame may be different for different sensor modalities and therefore for different map layers.

[0091] The method 700 includes, at block B708, assigning a group of poses from the frame graph to a road segment to generate a pose graph including poses corresponding to the road segment. For example, the process described herein with respect to FIG. 6B may be performed to determine poses 604 from different maneuvers 602 that correspond to a single road segment 610. After road segment 610C is determined, a pose graph including poses 604 from road segment 610C and each adjacent road segment (e.g., 610A, 610B, and 610D) may be generated to generate a pose graph 630, e.g., similar to the larger frame graph 600A, but with external pose connections (e.g., pose connections to poses 604 outside of road segments 610A-610D) not omitted.

[0092] The method 700 includes, at block B710, performing one or more pose optimization algorithms on the pose graph to generate an updated pose graph. For example, a RANSAC operation (e.g., FIG. 6F), a pose graph optimization (e.g., FIG. 6G), and / or other optimization algorithms may be performed to generate the updated pose graph 650.

[0093] The method 700 includes fusing map layers from multiple individual maps based on the updated poses from the updated pose graph to generate a fused map at block B712. For example, after the relative poses 604 for each frame of the map layers are known, the data from the map layers may be fused to generate an aggregated or fused HD map layer. For example, 3D landmark positions from two or more base layers of each map 504 may be fused according to their pose-aligned average positions. As such, if a pole in a first base layer of a first map 504 has a position (x1, y1, z1) and a pole in a second base layer of another map 504 has a position (x2, y2, z2), the pole may have a single final position in the fused base layer of the HD map as ((x1+x2) / 2, (y1+y2) / 2, (z1+z2) / 2). Additionally, the origin 652 of the road segment 610 to which the pole corresponds may be known, so the final pole position may be determined relative to the road segment's origin 652. This process may be repeated for all 3D landmarks in the base layer, and may be repeated for each other layer type of map 504 (e.g., a RADAR layer, a LiDAR layer, etc.).

[0094] The method 700 includes, at block B714, transmitting data representing the fused map to one or more vehicles for use in performing one or more operations. For example, the map data 108 representing the final fused HD map may be transmitted to one or more vehicles 1500 for localization, path planning, control decisions, and / or other operations. For example, once localized into the fused HD map, information from the planning and control layer of the autonomous driving stack may be obtained - e.g., lane graphs, waiting conditions, static obstacles, etc. - and this information may be provided to the planning and control subsystem of the autonomous vehicle 1500. In some examples, select layers of the fused HD map may be transmitted to each vehicle 1500 based on the configuration of the vehicle 1500. For example, if the vehicle 1500 does not have a LiDAR sensor, the LiDAR layer of the fused map may not be transmitted to the vehicle 1500 for storage and / or use. As a result of this adaptive approach, the bandwidth requirements for transmitting the map data 108 and / or the storage requirements on the vehicle 1500 for storing the map data 108 may be reduced.

[0095] Localization Referring to FIG. 8A, FIG. 8A is a data flow diagram of a process 110 of localization according to some embodiments of the present disclosure. In some embodiments, the process 110 may be performed using a vehicle 1500. In some embodiments, one or more of the processes described with respect to the process 110 may be performed in parallel using one or more parallel processing devices. For example, localization using a different map layer may be performed in parallel with one or more other map layers. Within a single map layer, the cost space sampling 802, cost space aggregation 804, and / or filtering 806 may be performed in parallel. For example, different poses may be sampled in parallel during the cost space sampling 802 to more efficiently generate a cost space for the current frame or time step. In addition, map data corresponding to a layer of a map may be stored in the GPU as textures, so that texture searches may be performed to quickly determine cost values ​​in the cost space - thereby leading to a runtime reduction of each cost space analysis. Additionally, as described herein, a localization process 110, such as cost space sampling 802, cost space aggregation 804, and / or filtering 806, may be performed during registration 402 in the map creation process 106. For example, the localization process 110 may be used to geometrically register poses from pairs of segments, and the cost space sampling 802, cost space aggregation 804, and / or filtering 806 may be used to align a map layer from one segment with a map layer of another segment.

[0096] The goal of the localization process 110 may be used to localize an origin 820 of the vehicle 1500 relative to a local origin 652 of the road segment 610 of the fused HD map represented by the map data 108. For example, an ellipsoid corresponding to the fused localization 1308, described in further detail herein with respect to FIGS. 9A-9C, may be determined for the vehicle 1500 at a particular time step or frame using the localization process 110. The origin 820 of the vehicle 1500 may correspond to a reference point or origin of the vehicle 1500, for example, the center of the rear axle of the vehicle 1500. The vehicle 1500 may be localized relative to the origin 652 of the road segment 610, and the road segment 610 may have a corresponding position 824 in a global coordinate system. As such, after the vehicle 1500 has been localized to the road segment 610 of the HD map, the vehicle 1500 may be localized globally as well. In some embodiments, an ellipsoid 910 may be determined for each individual sensor modality - e.g., LiDAR localization, camera localization, RADAR localization, etc. - and the output of each localization technique may be fused via localization fusion 810. A final origin position may be generated - e.g., origin 820 of vehicle 1500 - and used as the localization result of vehicle 1500 in the current frame or time step.

[0097] At the start of a drive, a current road segment 610 of the vehicle 1500 may be determined. In some embodiments, the current road segment 610 may be known from the last drive, e.g., when the vehicle 1500 was stopped, and the last known road segment to which the vehicle 1500 was localized may be stored. In other embodiments, the current road segment 610 may be determined. To determine the current road segment 610, GNSS data may be used to globally localize the vehicle 1500 and then determine the road segment 610 that corresponds to the global localization result. If the results return more than one road segment 610, the road segment 610 having an origin 652 closest to the origin of the vehicle 1500 may be determined to be the current road segment 610. After the current road segment 610 is determined, the road segment 610 may be determined to be a seed road segment for the breadth-first search. A breadth-first search may be performed to generate a local layout of road segments 610 adjacent to the current road segment 610 in a first level, then a second level of road segments 610 adjacent to the road segments 610 from the first level, etc. As the vehicle 1500 moves from one road segment 610 to another road segment 610, understanding the road segments 610 adjacent to the current road segment 610 may be useful for the localization process 110 because the relative transformation between the road segments 610 may be used to update the sampled cost space generated for the previous road segment 610 to be used in the aggregate cost space for localization. After the vehicle 1500 moves from the seed road segment to the adjacent road segment, another breadth-first search may be performed for the new road segment to generate an updated local layout, and this process may be repeated as the vehicle 1500 traverses the map from road segment to road segment. Additionally, as described herein, when the vehicle 1500 moves from one road segment 610 to another road segment 610, a previously calculated cost space (e.g., some number of previous cost spaces in a buffer, e.g., 50, 100, etc.) may be updated to reflect the same cost space with respect to the origin 652 of the new road segment 610.As a result, the calculated cost space may be carried over through the road segments to generate an aggregated cost space via cost space aggregation 804.

[0098] The localization process 110 may localize the vehicle 1500 at each time step or frame using the sensor data 102--e.g., real-time sensor data 102 generated by the vehicle 1500--the map data 108, and / or the output 204. For example, the sensor data 102, the output 204, and the map data 108 may be used to perform cost spatial sampling 802. The cost spatial sampling 802 may be different for different sensor modalities corresponding to different map layers. For example, the cost spatial sampling 802 may be performed separately for a LiDAR map layer (e.g., a LiDAR point cloud layer, a LiDAR map image layer, and / or a LiDAR voxel map layer), a RADAR map layer (e.g., a RADAR point cloud layer and / or a RADAR map image layer), and / or for a base layer (e.g., for a landmark or camera-based map layer). Within each sensor modality, cost sampling may be performed using one or more different techniques, and the costs across the different techniques may be weighted to generate a final cost for the sampled pose. This process may be repeated for each pose in the cost space to generate a final cost space for the frame during localization. For example, LiDAR intensity costs, LiDAR elevation costs, and LiDAR (slice) point cloud costs (e.g., using a distance function) may be calculated, then averaged or otherwise weighted, and used for the final cost of the pose or point on the cost space. Similarly, for camera or landmark-based cost space sampling, semantic costs may be calculated, and geometric costs (e.g., using a distance function) may be calculated, then averaged or otherwise weighted, and used for the final cost of the pose or point on the cost space. As a further example, RADAR point cloud costs (e.g., using a distance function) may be calculated, and used for the final cost of the pose or point on the cost space. As such, a cost space sampling 802 may be performed to sample the cost of each different pose in the cost space.The cost space may correspond to several regions within the map, which may include only a portion of the current road segment 610, the entire current road segment 610, the current road segment 610 and one or more adjacent road segments 610, and / or some other region of the entire fused HD map. As such, the size of the cost space may be a programmable parameter of the system.

[0099] The result of the cost space sampling 802, for any individual sensor modality, may be a cost space that represents a geometric match or likelihood that the vehicle 1500 may be located with respect to each particular pose. For example, points in the cost space may have corresponding relative positions with respect to the current road segment 610, and the cost space may indicate the likelihood or possibility that the vehicle 1500 is currently in each particular pose (e.g., an (x,y,z) position with respect to the origin of the road segment 610 and an axial angle with respect to each of the x, y, and z axes).

[0100] 9A, the cost space 902 may represent a sensor-based cost space, e.g., a camera-based cost space generated according to FIGS. 10A-10C. For example, the cost space 902 may represent the likelihood that the vehicle 1500 is currently in each of a plurality of poses, e.g., represented by a point in the cost space 902, in the current frame. The cost space 902, while represented in 2D in FIG. 9A, may correspond to a 3D cost space (e.g., having (x,y,z) positions and / or axis angles with respect to each of the x, y, and z axes). As such, the sensor data 102, e.g., before or after preprocessing, and / or the output 204 (e.g., detection of landmark positions in the 2D image space and / or the 3D image space) may be compared to the map data 108 for each of the plurality of poses. If a pose does not match well with the map data 108, the cost may be high and the point in the cost space corresponding to the pose may be represented as such - e.g., represented in red, or with respect to FIG. 9A, represented by a dotted or white portion. If a pose matches well with the map data 108, the cost may be low and the point in the cost space corresponding to the pose may be represented as such - e.g., represented in green, or with respect to FIG. 9A, represented by a dotted point. For example, with reference to FIG. 9A, if the cost space 902 corresponds to the visualization 1002 of FIG. 10A, the dotted portion 908 may correspond to low costs of poses along the diagonal where the label 1010 may match well with the prediction or output 204 of the DNN 202. For example, in the poses at the bottom left of the dotted portion of the cost space, the prediction of the label may line up well with the label 1010 from the map data 108, and similarly in the top right portion of the dotted portion, the prediction of the label from the corresponding pose may also line up well with the label 1010. As such, these points can be represented at low cost. However, due to noise and abundance of low cost poses, the single cost space 902 may not be accurate for localization - for example, the vehicle 1500 cannot be located at each of the poses represented by the dotted portions 908.As such, the aggregated cost space 904 may be generated via the cost space aggregation 804 as described herein.

[0101] Cost space sampling 802 may be performed separately for different sensor modalities as described herein. For example, with respect to FIGS. 10A-10D, a camera or landmark based cost space (e.g., corresponding to a base layer of a fused HD map) may be generated using geometric and / or semantic cost analysis at each pose of the cost space. For example, at a given time step or frame, outputs 204, such as landmark locations such as lane dividers, road boundaries, signs, poles, etc., may be calculated for an image, such as the image represented in visualization 1002. 3D landmark information from map data 108 may be projected into 2D image space to correspond to the location of the 3D landmarks in the 2D image space for a current prediction of vehicle 1500 at the current pose being sampled in the cost space. For example, landmarks 1010 may correspond to a 2D projection from map data, and landmarks 1012 may correspond to a current prediction or output 204 from one or more DNNs 202. Similarly, lane divider 1014 may correspond to a 2D projection from map data 108, and lane divider 1016 may correspond to a current prediction or output 204 from one or more DNNs 202. To calculate the cost of a current pose, e.g., represented by pose indicator 1018, the current output 204 from DNN 202 may be converted to a distance function corresponding to the geometry of the prediction as represented in visualization 1004 (e.g., where the prediction is divided into points, each point has zero cost at its center, and the cost increases from the center moving outward until a maximum cost is achieved, as represented by the white areas of visualization 1004), and the current output 204 may be individually converted to a semantic label of the prediction as represented in visualization 1006. Additionally, the 2D projection of the 3D landmarks may be projected into image space, and each point from the 2D projection may be compared to the portion of the distance function representation on which the projected point lands to determine the associated cost. As such, the dotted portion may correspond to a distance function representation of the current predictions of the DNN 202, and the solid black lines or dots may represent 2D projections from the map data 108.A cost may be calculated for each point of the 2D projection, and an average cost may be determined using the relative cost from each point. For example, the cost at point 1020A may be high, or the maximum, and the cost at point 1020B may be low - e.g., because the cost at point 1020B is aligned with the center of the distance function representation of the lane divider. This cost may correspond to the geometric cost of the current pose in the current frame. Similarly, the semantic labels corresponding to the 2D projection points from the map data 108 may be compared to the semantic information of the projection, as shown in FIG. 10C. As such, if the points do not match semantically, the cost may be set to a maximum value, and if the points match, the cost may be set to a minimum value. These values ​​may be averaged - or weighted in other ways - to determine a final semantic cost. The final semantic cost and the final geometric cost may be weighted to determine a final overall cost for updating the cost space (e.g., cost space 902). For example, for each point, the semantic cost may need to be low for the corresponding geometric cost to vote. As such, if the semantic information does not match, the cost of that particular point may be set to the maximum value. If the semantic information matches, the cost may be set to the minimum value for the semantic cost, or zero, and the final cost of the point may represent the geometric cost. Finally, the point in the cost space corresponding to the current pose may be updated to reflect the final costs of all the 2D projected points.

[0102] As another example, with respect to FIGS. 11A-11B, a RADAR-based cost space (e.g., corresponding to a RADAR layer of a fused HD map) may be generated using a distance function corresponding to map data 108 (e.g., a top-down projection of a RADAR point cloud with each point converted to a distance function representation). For example, at a given time step or frame, map data 108 corresponding to a RADAR point cloud - as represented in visualization 1102 - may be converted to a distance function as represented in visualization 1104 - e.g., where each RADAR point may have zero cost at its center, and the cost increases as the distance from the center increases, up to a maximum cost. For example, with respect to visualization 1104, the white portion of visualization 1102 may correspond to the maximum cost. To calculate the cost of the current pose - e.g., represented by pose indicator 1106 - RADAR data from sensor data 102 may be converted to a RADAR point cloud and compared to the distance function representation of the RADAR point cloud (e.g., as represented in visualization 1104). The hollow circles in visualizations 1102 and 1104 may correspond to the current RADAR point cloud predictions of the vehicle 1500. As such, for each current RADAR point, a cost may be determined by comparing each current RADAR point with the distance function RADAR value to which the current RADAR point corresponds or lands. As such, the current RADAR point 1108A may have the highest cost, while the current RADAR point 1108B may have a lower cost - for example, because point 1108B lands near the center of points from the RADAR point cloud in the map data 108. Finally, an average or other weighting of each cost from the current RADAR points may be calculated, and the final cost values ​​may be used to update the cost space for the currently sampled pose.

[0103] 12A-12D, a LiDAR-based cost space (e.g., corresponding to a LiDAR layer of a fused HD map) may be generated using a distance function on the (sliced) LiDAR point cloud, the LiDAR intensity map, and / or the LiDAR elevation map. For example, for a given pose—as indicated by pose indicator 1210—current or real-time LiDAR data (e.g., corresponding to sensor data 102) may be generated and converted to values ​​for comparison with an intensity map generated from map data 108 (e.g., as shown in visualization 1202), may be converted to values ​​for comparison with an elevation map generated from map data 108 (e.g., as shown in visualization 1204), and the LiDAR point cloud of map data 108 (e.g., as shown in visualization 1206) may be converted to a distance function representation of the same (e.g., as shown in visualization 1208) for comparison with the current LiDAR point cloud corresponding to sensor data 102. In an embodiment, the LiDAR point cloud may correspond to a slice of the LiDAR point cloud, and one or more separate slices may be converted to a distance function representation and used to calculate a cost. The costs from the elevation comparison, intensity comparison, and distance function comparison may be averaged or otherwise weighted to determine a final cost that corresponds to the current pose on the LiDAR-based cost map.

[0104] For example, with reference to FIG. 12A , the LiDAR layer of the fused HD map represented by map data 108 may include a LiDAR intensity (or reflectivity) image (e.g., a top-down projection of intensity values ​​from the fused LiDAR data). For example, painted surfaces such as lane markers may have higher reflectivity, and this reflectivity intensity may be captured and used to compare map data 108 to the current LiDAR sensor data. The current LiDAR sensor data from the vehicle 1500 may be converted to a LiDAR intensity representation 1212A and compared to the LiDAR intensity image from map data 108 at the current pose. For points in the current LiDAR intensity representation 1212A that have similar or matching intensity values ​​to points from map data 108, the cost may be low, and if the intensity values ​​do not match, the cost may be high. For example, a zero difference in the intensities of the points may correspond to zero cost, a threshold difference or greater may correspond to a maximum cost, and between zero difference and the threshold difference the cost may increase from zero cost to a maximum cost. The cost of each point in the current LiDAR intensity representation 1212A may be averaged or otherwise weighted with each other point to determine the cost of the LiDAR intensity comparison.

[0105] As another example, with respect to FIG. 12B , the LiDAR layer of the fused HD map represented by the map data 108 may include a LiDAR elevation image (e.g., a top-down projection of elevation values ​​resulting in a top-down depth map). Current LiDAR sensor data from the vehicle 1500 may be converted into a LiDAR elevation representation 1212B and compared to the LiDAR elevation image generated from the map data 108 at the current pose. For points in the current LiDAR elevation representation 1212B that have similar or matching elevation values ​​as points from the map data 108, the cost may be low, and if the elevation values ​​do not match, the cost may be high. For example, a zero difference in elevation angles of the points may correspond to zero cost, a threshold difference or greater may correspond to a maximum cost, and between zero difference and the threshold difference the cost may increase from zero cost to a maximum cost. The cost of each point in the current LiDAR elevation angle representation 1212B may be averaged or otherwise weighted with respect to each other to determine the cost of the LiDAR elevation angle comparison.

[0106] In some embodiments, the elevation value from the map data 108 may be determined relative to the origin 652 of the current road segment 610, and the elevation value from the current LiDAR elevation representation 1212B may correspond to the origin 820 or reference point of the vehicle 1500, so a transformation may be performed to compare the LiDAR elevation image from the map data 108 and the values ​​from the LiDAR elevation representation 1212B. For example, if a point from LiDAR elevation representation 1212B has an elevation value of 1.0 meter (e.g., 1.0 meter above the origin of vehicle 1500), then the point in map data 108 corresponding to the point from representation 1212B has a value of 1.5 meters, and the elevation difference between origin 820 of vehicle 1500 and origin 652 of road segment 610 of 0.5 meters (e.g., origin 652 of road segment is 0.5 meters higher than origin 820 of vehicle 1500), the actual difference between the point from map data 108 and representation 1212B may be 0.0 meters (e.g., 1.5 meters - 0.5 meters = 1 meter as the final value of the point from map data 108 relative to origin 820 of vehicle 1500). Depending on the embodiment, the transformation between values ​​in map data 108 or representation 1212B may correspond to a transformation from road segment origin 652 to vehicle origin 820, from vehicle origin 820 to road segment origin 652, or a combination thereof.

[0107] 12C-12D, the LiDAR layer of the fused HD map represented by map data 108 may include a sliced ​​LiDAR point cloud (e.g., corresponding to a ground plane slice, a Giraffe plane slice, another defined slice, e.g., a 1 meter thick slice extending 2 meters to 3 meters from the ground plane, etc.). In some examples, the point cloud may not be sliced ​​and may instead represent the entire point cloud. The sliced ​​LiDAR point cloud (e.g., as shown in visualization 1206) may be transformed into a distance function representation of the same (e.g., as shown in visualization 1208). For example, each point from the LiDAR point cloud may be transformed such that the center of the point has zero cost and the cost increases further from the center of the point to some maximum cost (e.g., as represented by the white region of visualization 1208). To calculate the cost of the current pose—e.g., represented by pose indicator 1210—the LiDAR data from sensor data 102 may be converted to a LiDAR point cloud (or a corresponding slice thereof) and compared to a distance function representation of the LiDAR point cloud (e.g., as represented in visualization 1208). The hollow circles in visualizations 1206 and 1208 may correspond to the current LiDAR point cloud predictions of vehicle 1500. As such, for each current LiDAR point, a cost may be determined by comparing each current LiDAR point to the distance function LiDAR value to which the current LiDAR point corresponds or lands. As such, current LiDAR point 1214A may have the highest cost, while current LiDAR point 1214B may have a lower cost—e.g., because point 1214B lands near the center of the points from the LiDAR point cloud in map data 108. Finally, an average or other weighting of each cost from the current LiDAR point may be calculated, and the final cost value may be used - in addition to the cost values ​​from the elevation and intensity comparisons - to update the cost space of the currently sampled pose.

[0108] In some embodiments, at least one of the LiDAR-based cost space sampling 802, cost space aggregation 804, and / or filtering 806 may be performed on a GPU - e.g., a separate GPU, a virtual GPU, etc. - and / or using one or more parallel processing devices. For example, for a camera-based cost space (e.g., as described with respect to FIGS. 10A-10C), the detection information and map data 108 projections may be stored as textures on the GPU or in memory accessible to the GPU, and the comparison may correspond to a texture lookup performed using the GPU. Similarly, for LiDAR and / or RADAR, the comparison may correspond to a texture lookup. Additionally, in some embodiments, parallel processing may be used to run two or more cost spaces in parallel - e.g., a first cost space corresponding to LiDAR and a second cost space corresponding to RADAR may be generated in parallel using different GPU and / or parallel processing device resources. For example, the separate localizations 808 may be computed in parallel such that the execution time of the system for the fused localization is reduced. As a result, these processes may be executed more efficiently than if they were executed by the CPU alone.

[0109] Referring again to FIG. 8A, after cost space sampling 802 has been performed for a single frame or time step and for any number of sensor modalities, cost space aggregation 804 may be performed. Cost space aggregation 804 may be performed separately for each sensor modality - e.g., LiDAR-based cost space aggregation, RADAR-based cost space aggregation, camera-based cost space aggregation, etc. For example, cost space aggregation 804 may aggregate cost spaces calculated for any number of frames (e.g., 25 frames, 80 frames, 100 frames, 300 frames, etc.). To aggregate the cost spaces, each previous cost space that has been calculated may be self-motion compensated to correspond to the current frame. For example, the rotation and / or translation of vehicle 1500 relative to the current pose of vehicle 1500 and from each previous frame included in the aggregation may be determined and used to carry forward cost space values ​​from the previous frame. In addition to transforming the previous cost space based on ego-motion, the cost spaces may also be transformed - e.g., using a transformation from one road segment to the next, such as that described with respect to FIG. 6I - so that each cost space corresponds to the origin 652 of the current road segment 610 of the fused HD map. For example, several cost spaces to be aggregated may be generated while the vehicle 1500 is localized to a first road segment 610, and several other cost spaces to be aggregated may be generated while the vehicle 1500 is localized to a second road segment 610. As such, the cost spaces from the first or previous road segment 610 may be transformed such that the cost space values ​​are relative to the second or current road segment 610. Once in the same reference frame corresponding to the current frame and the current road segment 610, the cost spaces may be aggregated. As a result, and with reference to FIG. 9B, the ego-motion of the vehicle 1500 over time may help disambiguate the individual cost spaces such that an aggregated cost space 904 may be generated.

[0110] The aggregated cost space 904 may then be filtered 806, e.g., using a Kalman filter or another filter type, to determine an ellipsoid 910 that corresponds to the calculated position of the vehicle 1500 with respect to the current road segment 610. Similar to that described above with respect to the transformation of the aggregated cost space 904, the filtered cost space 906 may also be transformed to correct for ego-motion and road segment switching. The ellipsoid 910 may indicate the current position of the vehicle 1500 with respect to the current road segment 610, and this process may be repeated for each new frame. The result may be a separate localization based on the sensor modality for which the ellipsoid 910 was calculated, and multiple ellipsoids 910 may be calculated for each frame, e.g., one for each sensor modality.

[0111] A localization fusion 810 may then be performed on the individual localizations 808 to generate a final localization result. For example, with reference to FIG. 13, the individual localizations 808 may correspond to the LiDAR-based localization 1302 (e.g., represented by an ellipsoid and origin in the visualization 1300), the RADAR-based localization 1304, the camera-based localization 1306, other sensor modality localizations (not shown), and / or the fused localization 1308. Although only a single localization per sensor modality is described herein, this is not intended to be limiting. In some examples, there may be two or more localization results of different sensor modalities. For example, the vehicle 1500 may be localized with respect to a first camera (e.g., a forward-facing camera) and may be individually localized with respect to a second camera (e.g., a rear-facing camera). In such an instance, the individual localization 808 may include a first camera-based localization and a second camera-based localization. In some embodiments, the fused localization 1308 may correspond to a fusion of the individual localizations 808 in the current frame and / or may correspond to previous fused localization results from one or more previous frames carried forward into the current frame - e.g., based on ego-motion. As such, in embodiments, the fused localization 1308 of the current frame may take into account the individual localizations 808 and previous fused localization results to advance the current localization state through the frames.

[0112] A match / mismatch analysis may be performed on the individual localizations 808 to compute the fused localization 1308 of the current frame. For example, in some embodiments, a distance threshold may be used to determine clusters of the individual localizations 808, and the cluster with the least intra-cluster covariance may be selected for fusion. The individual localizations 808 within the selected cluster may then be averaged or otherwise weighted to determine the fused localization 1308 of the current frame. In some embodiments, a filter, such as a Kalman filter, may be used to generate the fused localization 1308 of the clustered individual localizations 808 of the current frame. For example, a Kalman filter, if used, may not deal well with outliers, and thus a large amount of outliers may have an undesirable effect on the final result. As such, clustering techniques may help filter out or remove outliers so that the Kalman filter-based fusion is more accurate. In some embodiments, the fusion result from a previous frame may drift, for example, if the previous fusion result is carried forward to the current frame as an individual localization 808. For example, if the current individual localization 808 is sufficiently different from the fusion result (e.g., if the fusion result can be filtered out of the cluster), the fusion result may be reinitialized for the current frame, and the reinitialized fusion result may then be carried forward to subsequent frames until a certain amount of drift is again detected.

[0113] In some embodiments, the fused localization 1308 may be determined by factoring on each individual localization 808. For example, instead of grouping the results into clusters, each individual localization 808 may be weighted based on a distance evaluation. For example, covariance may be calculated for the individual localizations 808, and the individual localization 808 with the highest covariance (e.g., corresponding to the largest outlier) may be weighted less in determining the fused localization 1308. This may be performed using a robustified average so that the outliers do not have an undesirable effect on the fused localization 1308. For example, the distance of each individual localization 808 to the robustified average may be calculated, and the greater the distance, the less weight the individual localization 808 may have in determining the fused localization 1308.

[0114] The fused localization 1308 of the current frame may then be used to localize the vehicle 1500 with respect to the road segment 610 of the fused HD map (represented by map data 108) and / or with respect to a global coordinate system. For example, the road segment 610 may have a known global position, so that localization of the vehicle 1500 to the road segment 610 may have a corresponding global localization result. In addition, because the local or relative localization to the road segment 610 is more accurate than global or GNSS localization results alone, planning and control of the vehicle 1500 may be more reliable and secure than in a solely GNSS-based localization system.

[0115] Referring now to FIG. 14, each block of method 1400 described herein includes computational processes that may be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in a memory. Method 1400 may also be implemented as computer usable instructions stored on a computer storage medium. Method 1400 may be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. Additionally, method 1400 is described with respect to process 110 of FIG. 8A, by way of example. However, method 1400 may additionally or alternatively be performed within any one process by any one system, or any combination of processes and systems, including, but not limited to, those described herein.

[0116] FIG. 14 is a flow diagram illustrating a method 1400 for localization according to some embodiments of the present disclosure. The method 1400 includes, at block B1402, generating one or more of sensor data or DNN outputs based on the sensor data at each frame of a plurality of frames. For example, at each frame of some buffered number of frames or time steps (e.g., 25, 50, 70, 100, etc.), the sensor data 102 and / or output 204 may be calculated. For example, if LiDAR-based localization or RADAR-based localization is used, the sensor data 102 may be generated and / or processed (e.g., to generate an elevation representation, an intensity representation, etc.) at each frame or time step. If camera-based localization is used, the sensor data 102 may be applied to the DNN 202 to generate output 204 corresponding to a landmark.

[0117] The method 1400 includes, at block B1404, comparing one or more of the sensor data or the DNN output to the map data at each frame to generate a cost space representative of the likelihood that the vehicle is located at each of a plurality of poses. For example, cost space sampling 802 may be performed at each frame to generate a cost space corresponding to a particular sensor data modality. The comparison of the map data 108 to the sensor data 102 and / or output 204 may include comparing LiDAR elevation angle information, LiDAR intensity information, LiDAR point cloud (slice) information, RADAR point cloud information, camera landmark information (e.g., comparing 2D projections of 3D landmark positions from the map data 108 to current real-time predictions of the DNN 202), etc.

[0118] The method 1400 includes aggregating each cost space from each frame to generate an aggregated cost space at block B1406. For example, the cost space aggregation 804 may be performed on several buffered or previous cost spaces to generate an aggregated cost space. The aggregation may include ego-motion transformations of the previous cost spaces and / or road segment transformations of previous cost spaces generated for road segments 610 other than the current road segment 610 of the vehicle 1500.

[0119] The method 1400 includes applying a Kalman filter to the aggregated cost space to calculate a final position of the vehicle in a current frame of the multiple frames at block B 1408. For example, the filter process 806 may be applied to the aggregated cost space to generate an ellipsoid or other representation of an estimated position of the vehicle 1500 for a particular sensor modality in the current frame or time step.

[0120] Blocks B1402-B1408 of process 1408 may be repeated - in parallel, in one embodiment - for any number of different sensor modalities such that two or more ellipsoids or position predictions for vehicle 1500 are generated.

[0121] The method 1400 includes using the final position in addition to one or more other final positions of the vehicle to determine a fused position of the vehicle at block B1410. For example, the ellipsoids of the other representations output after different sensor modality filtering 806 may undergo localization fusion 810 to generate a final fused localization result. In some implementations, previous fused localization results from previous frames or time steps may also be used in determining the current fused localization result, as described herein.

[0122] Exemplary Autonomous Vehicle 15A is a diagram of an example autonomous vehicle 1500 according to some embodiments of the disclosure. Autonomous vehicle 1500 (alternatively referred to herein as “vehicle 1500”) may include, but is not limited to, passenger vehicles, such as cars, trucks, buses, first responder vehicles, shuttles, electric or mopeds, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and previous and future versions of this standard). The mobile vehicle 1500 may be capable of functionality according to one or more of the autonomous driving levels Levels 3 through 5. For example, the mobile vehicle 1500 may be capable of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the embodiment.

[0123] The mobile vehicle 1500 may include components such as a chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the mobile vehicle. The mobile vehicle 1500 may include a propulsion system 1550, such as an internal combustion engine, a hybrid power plant, a fully electric engine, and / or another propulsion system type. The propulsion system 1550 may be connected to a drive train of the mobile vehicle 1500, which may include a transmission, to enable propulsion of the mobile vehicle 1500. The propulsion system 1550 may be controlled in response to receiving a signal from a throttle / accelerator 1552.

[0124] A steering system 1554, which may include a steering wheel, may be used to steer the vehicle 1500 (e.g., along a desired course or route) when the propulsion system 1550 is operating (e.g., when the vehicle is moving). The steering system 1554 may receive signals from a steering actuator 1556. The steering wheel may be an option for fully automated (Level 5) functionality.

[0125] The brake sensor system 1546 may be used to operate the vehicle brakes in response to receiving a signal from a brake actuator 1548 and / or a brake sensor.

[0126] A controller 1536, which may include one or more system on chip (SoC) 1504 (FIG. 15C) and / or a GPU, can provide signals (e.g., representations of commands) to one or more components and / or systems of the vehicle 1500. For example, the controller can send signals to operate vehicle brakes via one or more brake actuators 1548, operate a steering system 1554 via one or more steering actuators 1556, and operate a propulsion system 1550 via one or more throttle / accelerators 1552. The controller 1536 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) to process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a driver in driving the vehicle 1500. The controllers 1536 may include a first controller 1536 for autonomous driving functions, a second controller 1536 for functional safety functions, a third controller 1536 for artificial intelligence functions (e.g., computer vision), a fourth controller 1536 for infotainment functions, a fifth controller 1536 for redundancy in emergency situations, and / or other controllers. In some instances, a single controller 1536 may handle two or more of the aforementioned functions, and two or more controllers 1536 may handle a single function, and / or any combination thereof.

[0127] Controller 1536 may provide signals to control one or more components and / or systems of vehicle 1500 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data may be received from, for example, and without limitation, global navigation satellite system sensors 1558 (e.g., global positioning system sensors), RADAR sensors 1560, ultrasonic sensors 1562, LIDAR sensors 1564, inertial measurement unit (IMU) sensors 1566 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1596, stereo cameras 1568, wide-view cameras 1570 (e.g., fisheye cameras), infrared cameras 1572, surround cameras 1574 (e.g., 360 degree cameras), long-range and / or medium-range cameras 1598, speed sensors 1544 (e.g., for measuring the speed of the moving vehicle 1500), vibration sensors 1542, steering sensors 1540, brake sensors (e.g., as part of a brake sensor system 1546), and / or other sensor types.

[0128] One or more of the controllers 1536 may receive input (e.g., represented by input data) from the instrument cluster 1532 of the vehicle 1500 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1534, an audible annunciator, a loudspeaker, and / or other components of the vehicle 1500. The output may include information such as vehicle velocity, speed, time, map data (e.g., HD map 1522 of FIG. 15C), position data (e.g., the position of the vehicle 1500 on a map, etc.), direction, the positions of other vehicles (e.g., occupancy grid), information regarding objects and object situations as perceived by the controller 1536, etc. For example, the HMI display 1534 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changing traffic signals, etc.) and / or a driving maneuver that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, taking exit 34B in 2 miles, etc.).

[0129] The mobile vehicle 1500 further includes a network interface 1524 that can communicate over one or more networks using one or more wireless antennas 1526 and / or a modem. For example, the network interface 1524 can be capable of communication over LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antennas 1526 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or low power wide-area networks (LPWANs) such as LoRaWAN, SigFox, etc.

[0130] 15B is an illustration of camera positions and fields of view of the example autonomous vehicle 1500 of FIG. 15A, according to some embodiments of the present disclosure. The cameras and their respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different positions on the vehicle 1500.

[0131] The camera type of the camera may include, but is not limited to, a digital camera that may be adapted for use with components and / or systems of the mobile vehicle 1500. The camera may operate at automotive safety integrity level (ASIL) B and / or at another ASIL. The camera type may be capable of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some instances, the color filter array may include a red clear clear clear (RCCC) color filter array, a red clear clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras with RCCC, RCCB, and / or RBGC color filter arrays, may be used in an effort to increase light sensitivity.

[0132] In some instances, one or more of the cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) may simultaneously record and provide image data (e.g., video).

[0133] One or more of the cameras may be mounted in a mounting part, such as a custom designed (3D printed) part, to filter out stray light and reflections from within the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capabilities. With reference to a side mirror mounting part, the side mirror part may be custom 3D printed such that the camera mounting plate fits the shape of the side mirror. In some instances, the camera may be integrated into the side mirror. For side view cameras, the cameras may also be integrated into four posts at each corner of the cabin.

[0134] A camera (e.g., a forward-facing camera) with a field of view that includes a portion of the environment in front of the vehicle 1500 may be used for surround view to help identify the forward path and obstacles and, with the aid of one or more controllers 1536 and / or control SoCs, provide information essential to generating an occupancy grid and / or determining a preferred vehicle path. Forward-facing cameras may be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras may also be used for ADAS functions and systems, including other functions such as lane departure warning ("LDW"), autonomous cruise control ("ACC"), and / or traffic sign recognition.

[0135] A variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform including a complementary metal oxide semiconductor (CMOS) color imager. Another example may be a wide-view camera 1570 that may be used to capture objects entering the view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although only one wide-view camera is shown in FIG. 15B, there may be any number of wide-view cameras 1570 in the vehicle 1500. In addition, a long-range camera 1598 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which the neural network has not yet been trained. The long-range camera 1598 may also be used for object detection and classification, as well as basic object tracking.

[0136] One or more stereo cameras 1568 may also be included in the forward-facing configuration. The stereo camera 1568 may include an integrated control unit with an extensible processing unit that may provide programmable logic (FPGA) and a multi-core microprocessor with CAN or Ethernet interfaces integrated on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 1568 may include a compact stereo vision sensor that may include two camera lenses (one left and one right) and an image processing chip that can measure distances from the vehicle to objects of interest and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1568 may be used in addition to or instead of those described herein.

[0137] A camera having a field of view that includes a portion of the environment to the side of the moving vehicle 1500 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and to generate side impact collision warnings. For example, surround cameras 1574 (e.g., four surround cameras 1574 as shown in FIG. 15B) may be positioned on the moving vehicle 1500. The surround cameras 1574 may include wide-view cameras 1570, fish-eye cameras, 360-degree cameras, and / or the like. For example, four fish-eye cameras may be positioned at the front, back, and sides of the moving vehicle. In an alternative arrangement, the moving vehicle may use three surround cameras 1574 (e.g., left, right, and rear) and may utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0138] A camera having a field of view that includes the portion of the environment behind the moving vehicle 1500 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creating and updating an occupancy grid. As described herein, a wide variety of cameras may be used, including, but not limited to, cameras that are also suitable as forward-facing cameras (e.g., long-range and / or mid-range cameras 1598, stereo cameras 1568, infrared cameras 1572, etc.).

[0139] FIG. 15C is a block diagram of an example system architecture of the example autonomous vehicle 1500 of FIG. 15A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted altogether. Furthermore, many of the elements described herein are functional entities that may be implemented as separate or distributed components or in conjunction with other components, and in any suitable combination and location. Various functions described herein as being performed by entities may be implemented by hardware, firmware, and / or software. For example, various functions may be implemented by a processor executing instructions stored in a memory.

[0140] Each of the components, features, and systems of the mobile vehicle 1500 of FIG. 15C are shown as connected via a bus 1502. The bus 1502 may include a controller area network (CAN) data interface (alternatively referred to as a "CAN bus"). The CAN may be a network within the mobile vehicle 1500 that is used to help control various features and functions of the mobile vehicle 1500, such as actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL B compliant.

[0141] The bus 1502 is described herein as being a CAN bus, but this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or as an alternative to a CAN bus. Additionally, a single line is used to represent the bus 1502, but this is not intended to be limiting. There may be any number of buses 1502, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some instances, two or more buses 1502 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1502 may be used for collision avoidance functions and a second bus 1502 may be used for actuation control. In any instance, each bus 1502 may communicate with any of the components of the mobile vehicle 1500, and two or more buses 1502 may communicate with the same components. In some examples, each SoC 1504, each controller 1536, and / or each computer in the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1500) and may be connected to a common bus, such as a CAN bus.

[0142] The mobile vehicle 1500 may include one or more controllers 1536, such as those described herein with respect to FIGURE 15A. The controllers 1536 may be used for a variety of functions. The controllers 1536 may be coupled to any of a variety of other components and systems of the mobile vehicle 1500 and may be used for control of the mobile vehicle 1500, artificial intelligence of the mobile vehicle 1500, infotainment for the mobile vehicle 1500, and / or the like.

[0143] The mobile vehicle 1500 may include a system on a chip (SoC) 1504. The SoC 1504 may include a CPU 1506, a GPU 1508, a processor 1510, a cache 1512, an accelerator 1514, a data store 1516, and / or other components and features not shown. The SoC 1504 may be used to control the mobile vehicle 1500 in a variety of platforms and systems. For example, the SoC 1504 may be coupled in a system (e.g., the system of the mobile vehicle 1500) having an HD map 1522 that can obtain map refreshes and / or updates via a network interface 1524 from one or more servers (e.g., server 1578 of FIG. 15D).

[0144] The CPU 1506 may include a CPU cluster or CPU complex (alternatively referred to as a "CCPLEX"). The CPU 1506 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1506 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1506 may include four dual-core clusters, each with its own dedicated L2 cache (e.g., a 2M B L2 cache). The CPU 1506 (e.g., a CCPLEX) may be configured to support concurrent cluster operation allowing any combination of the CPU 1506 clusters to be active at any given time.

[0145] The CPU 1506 may implement power management capabilities including one or more of the following features: individual hardware blocks may be automatically clock gated when idle to conserve dynamic power, each core clock may be gated when the core is not actively executing instructions by execution of WFI / WFE instructions, each core may be independently power gated, each core cluster may be independently clock gated when all cores are clock gated or power gated, and / or each core cluster may be independently power gated when all cores are power gated. The CPU 1506 may further implement an enhanced algorithm for managing power states where allowed power states and expected wake-up times are specified and hardware / microcode determines the best power state for entering the cores, clusters, and CCPLEXes. The processing cores may support simplified power state entry sequences in software where work is offloaded to microcode.

[0146] The GPU 1508 may include an integrated GPU (alternatively referred to herein as an "iGPU"). The GPU 1508 may be programmable and efficient for parallel workloads. In some instances, the GPU 1508 may use an enhanced tensor instruction set. The GPU 1508 may include one or more streaming microprocessors, where each streaming microprocessor may include an L1 cache (e.g., an L1 cache having at least 96 KB storage capacity) and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache having 512 KB storage capacity). In some embodiments, the GPU 1508 may include at least eight streaming microprocessors. The GPU 1508 may use a computational application programming interface (API). In addition, the GPU 1508 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0147] The GPU 1508 may be power optimized for best performance in automotive and embedded use cases. For example, the GPU 1508 may be fabricated on FinFETs (Fin field-effect transistors). However, this is not intended to be limiting and the GPU 1508 may be fabricated using other semiconductor fabrication processes. Each streaming microprocessor may incorporate several mixed-precision processing cores partitioned into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores may be partitioned into four processing blocks. In such an example, each processing block may be assigned 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads having a mix of computational and addressing operations. Streaming microprocessors may include independent thread scheduling capabilities to allow finer grain synchronization and coordination among concurrent threads. Streaming microprocessors may include a combined L1 data cache and shared memory unit to improve performance while simplifying programming.

[0148] The GPU 1508 may include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide about 900GB / sec peak memory bandwidth in some instances. In some instances, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used in addition to or in place of the HBM memory.

[0149] The GPU 1508 may include unified memory technology including access counters to enable more accurate movement of memory pages to the processors that access them most frequently, thereby improving the efficiency of storage ranges shared between processors. In some instances, address translation service (ATS) support may be used to enable the GPU 1508 to directly access the CPU 1506 page tables. In such instances, when the GPU 1508 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU 1506. In response, the CPU 1506 may consult its page tables for a virtual-to-real mapping of the address and send the translation back to the GPU 1508. As such, the unified memory technology may enable a single unified virtual address space for both the CPU 1506 and GPU 1508 memories, thereby simplifying GPU 1508 programming and porting of applications to the GPU 1508.

[0150] In addition, the GPU 1508 may include access counters that can record the frequency of accesses of the GPU 1508 to the memory of other processors. The access counters can help ensure that memory pages are moved to the physical memory of the processor that is accessing the page most frequently.

[0151] The SoC 1504 may include any number of caches 1512, including those described herein. For example, the cache 1512 may include an L3 cache available to both the CPU 1506 and the GPU 1508 (e.g., connected to both the CPU 1506 and the GPU 1508). The cache 1512 may include a write-back cache that can record line state, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although smaller cache sizes may be used.

[0152] The SoC 1504 may include an arithmetic logic unit (ALU) that may be utilized in performing processing for any of the various tasks or operations (e.g., processing DNNs) of the vehicle 1500. In addition, the SoC 1504 may include a floating point unit (FPU) (or other math co-processor or numeric co-processor type) for performing mathematical operations within the system. For example, the SoC 1504 may include one or more FPUs integrated as execution units within the CPU 1506 and / or GPU 1508.

[0153] The SoC 1504 may include one or more accelerators 1514 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 1504 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other operations. The hardware acceleration cluster may be used to complement the GPU 1508 and to offload some of the tasks of the GPU 1508 (e.g., to free up more cycles for the GPU 1508 to perform other tasks). As an example, the accelerator 1514 may be used for target workloads that are stable enough to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). As used herein, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and Faster RCNNs (e.g., as used for object detection).

[0154] The accelerator 1514 (e.g., a hardware accelerated cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating point operations, as well as inference. The design of the DLA may provide more performance per millimeter than a general-purpose GPU, greatly exceeding the performance of a CPU. The TPU may perform several functions, including, for example, single instance convolution functions, supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processor functions.

[0155] The DLA can quickly and efficiently run neural networks, in particular CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object identification and detection using data from camera sensors, CNNs for distance estimation using data from camera sensors, CNNs for emergency vehicle detection and identification using data from microphones, CNNs for face recognition and moving vehicle owner identification using data from camera sensors, and / or CNNs for security and / or safety related events.

[0156] The DLA can perform any function of the GPU 1508, and by using an inference accelerator, for example, a designer can target either the DLA or the GPU 1508 for any function. For example, a designer can focus on processing CNNs and floating point operations on the DLA and offload other functions to the GPU 1508 and / or other accelerators 1514.

[0157] The accelerator 1514 (e.g., hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors, for example.

[0158] The RISC cores may interact with an image sensor (e.g., an image sensor of any of the cameras described herein), an image signal processor, and / or the like. Each RISC core may include any amount of memory. The RISC cores may use any of a number of protocols, depending on the embodiment. In some instances, the RISC cores may run a real-time operating system (RTOS). The RISC cores may be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores may include an instruction cache and / or tightly coupled RAM.

[0159] The DMA may allow components of the PVA to access system memory independent of the CPU 1506. The DMA may support any number of features used to provide optimizations to the PVA, including but not limited to supporting multi-dimensional addressing and / or circular addressing. In some instances, the DMA may support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0160] The vector processor may be a programmable processor that may be designed to efficiently and flexibly execute computer vision algorithm programming and provide signal processing capabilities. In some instances, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a single instruction, multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW may increase throughput and speed.

[0161] Each vector processor may include an instruction cache and may be coupled to dedicated memory. As a result, in some instances, each vector processor may be configured to execute independently of other vector processors. In other instances, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other instances, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequential images or portions of an image. In particular, any number of PVAs may be included in a hardware accelerated cluster, and any number of vector processors may be included in each PVA. In addition, the PVAs may include additional error correcting code (ECC) memory to increase overall system security.

[0162] The accelerator 1514 (e.g., a hardware acceleration cluster) may include a computer vision network-on-chip and SRAM to provide high bandwidth, low latency SRAM for the accelerator 1514. In some instances, the on-chip memory may include, for example and without limitation, at least 4 MB of SRAM consisting of eight field configurable memory blocks that may be accessible by both the PVA and DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, controllers, and multiplexers. Any type of memory may be used. The PVA and DLA may access the memory through a backbone that provides the PVA and DLA with high speed access to the memory. The backbone may include a computer vision network-on-chip that interconnects the PVA and DLA to the memory (e.g., using the APB).

[0163] The computer vision network-on-chip may include an interface that determines, prior to transmission of any control signals / addresses / data, that both the PVA and DLA provide ready and valid signals. Such an interface may provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst type communication for continuous data transfer. This type of interface may follow ISO26262 or IEC61508 standards, although other standards and protocols may be used.

[0164] In some examples, the SoC 1504 may include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application Serial No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and scale of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, for acoustic propagation synthesis and / or analysis, for simulation of SONAR systems, for general wave propagation simulation, for comparison against LIDAR data for localization and / or other functions, and / or for other uses. In some examples, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0165] The accelerator 1514 (e.g., a hardware accelerator cluster) has a variety of applications for autonomous driving. The PVA may be a programmable vision accelerator that can be used for critical processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are well suited to the domain of algorithms that require predictable processing at low power and low latency. In other words, the PVA works well with semi-dense or dense regular computations on small data sets that require predictable execution times with low latency and low power. Hence, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms, since the PVA is efficient in object detection and operating on integer computations.

[0166] For example, according to one embodiment of the present technology, the PVA is used to perform computer stereo vision. A semi-global matching based algorithm may be used in some instances, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions with input from two monocular cameras.

[0167] In some instances, PVA may be used to perform dense optical flow by processing raw RADAR data (e.g., using a 4D Fast Fourier Transform) to provide processed RADAR. In other instances, PVA is used for time of flight depth processing, for example, by processing raw time of flight data to provide processed time of flight data.

[0168] DLA can be used to implement any type of network to enhance control and driving safety, including, for example, a neural network that outputs a measure of the confidence of each object detection. Such confidence value can be interpreted as a probability or as providing a relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions regarding which detections should be considered true positive detections rather than false positive detections. For example, the system can set a confidence threshold and consider only detections above the threshold as true positive detections. In an automatic emergency braking (AEB) system, a false positive detection would cause a moving vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered to trigger AEB. DLA can implement a neural network that regresses the confidence value. The neural network may receive as its inputs at least some subset of parameters, such as bounding box dimensions, a ground plane estimate obtained (e.g., from another subsystem), inertial measurement unit (IMU) sensor 1566 output that correlates with the vehicle 1500 orientation, distance, and 3D position estimates of the object obtained from the neural network and / or other sensors (e.g., a LIDAR sensor 1564 or a RADAR sensor 1560), and so forth.

[0169] The SoC 1504 may include a data store 1516 (e.g., a memory). The data store 1516 may be an on-chip memory of the SoC 1504 and may store the neural network to be executed by the GPU and / or DLA. In some instances, the data store 1516 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1512 may comprise an L2 or L3 cache 1512. References to the data store 1516 may include references to memory associated with the PVA, DLA, and / or other accelerators 1514, as described herein.

[0170] The SoC 1504 may include one or more processors 1510 (e.g., embedded processors). The processors 1510 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may be part of the SoC 1504 boot sequence and may provide run-time power management services. The boot power and management processor may provide clock and voltage programming, assist with system low power state transitions, management of the SoC 1504 thermal and temperature sensors, and / or management of the SoC 1504 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 1504 may use the ring oscillator to detect the temperature of the CPU 1506, the GPU 1508, and / or the accelerator 1514. If the temperature is determined to exceed a threshold, the boot and power management processor may enter a temperature fault routine, place the SoC 1504 in a lower power state and / or place the vehicle 1500 in a Chauffeur safe shutdown mode (e.g., bring the vehicle 1500 to a safe shutdown).

[0171] The processor 1510 may further include a set of embedded processors that may perform the functions of an audio processing engine. The audio processing engine may be an audio subsystem that allows full hardware support of multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some instances, the audio processing engine is a dedicated processor core that includes a digital signal processor with dedicated RAM.

[0172] The processor 1510 may further include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0173] The processor 1510 may further include a safety cluster engine that includes a processor subsystem dedicated to handling safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations.

[0174] The processor 1510 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0175] The processor 1510 may further include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0176] The processor 1510 may include a video image compositor, which may be a processing block (e.g., implemented in a microprocessor) that implements video post-processing functions required by the video playback application to produce a final image for the player window. The video image compositor may perform lens distortion correction on the wide-view camera 1570, on the surround camera 1574, and / or on the in-cabin surveillance camera sensor. The in-cabin surveillance camera sensor is preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify in-cabin events and respond appropriately. The in-cabin system may perform lip reading to activate cellular service and make phone calls, dictate emails, change the destination of the vehicle, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain features are available to the driver only when operating in autonomous mode and are disabled otherwise.

[0177] The video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, when motion occurs in the video, the noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. When an image or part of an image does not contain motion, the temporal noise reduction performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0178] The video image composer may also be configured to perform stereo rectification on the input stereo lens frames. The video image composer may further be used for user interface compositing when the operating system desktop is in use and the GPU 1508 is not required to continuously render new surfaces. Even when the GPU 1508 is powered on and actively performing 3D rendering, the video image composer may be used to offload the GPU 1508 for improved performance and responsiveness.

[0179] The SoC1504 may further include a mobile industry processor interface (MIPI) camera serial interface for receiving video and input from a camera, a high-speed interface, and / or a video input block that may be used for camera and related pixel input functions. The SoC1504 may further include an input / output controller that may be controlled by software and that may be used to receive I / O signals that are not committed to a specific role.

[0180] The SoC 1504 may further include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 1504 may be used to process data from cameras (e.g., connected via gigabit multimedia serial links and Ethernet), sensors (e.g., LIDAR sensor 1564, RADAR sensor 1560, etc., which may be connected via Ethernet), data from the bus 1502 (e.g., vehicle 1500 speed, steering wheel position, etc.), and GNSS sensor 1558 (e.g., connected via Ethernet or CAN bus). The SoC 1504 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to offload the CPU 1506 from routine data management tasks.

[0181] The SoC1504 may be an end-to-end platform with a flexible architecture that spans levels 3-5 of automation, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC1504 may be faster, more reliable, and more energy- and space-efficient than traditional systems. For example, when the accelerator 1514 is combined with the CPU 1506, the GPU 1508, and the data store 1516, the SoC1504 may provide a fast and efficient platform for levels 3-5 of autonomous vehicles.

[0182] Thus, the present technology provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms may be executed on a CPU, which may be configured using a high-level programming language, such as the C programming language, to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical Level 3-5 autonomous vehicles.

[0183] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the techniques described herein allow multiple neural networks to run simultaneously and / or sequentially and the results to be combined to enable Level 3-5 autonomous driving capabilities. For example, a CNN running on the DLA or dGPU (e.g., GPU1520) can include text and word recognition that allows the supercomputer to read and understand traffic signs, including signs for which the neural network was not specifically trained. The DLA can further include neural networks that can identify, interpret, and provide a semantic understanding of the signs and pass the semantic understanding to a route planning module running on the CPU complex.

[0184] As another example, multiple neural networks may be run simultaneously, as required for level 3, 4, or 5 operation. For example, a warning sign consisting of "Caution: Flashing lights indicate icy conditions" along with a lightning flash may be interpreted by several neural networks, independently or collectively. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" may be interpreted by a second deployed neural network that informs the vehicle's route planning software (preferably running on a CPU complex) that icy conditions exist when a flashing light is detected. The flashing light may be identified by running a third deployed neural network through multiple frames, informing the vehicle's route planning software of the presence (or absence) of the flashing light. All three neural networks may be run simultaneously, such as within the DLA and / or on the GPU 1508.

[0185] In some instances, a CNN for facial recognition and vehicle owner identification can use data from the camera sensor to identify the presence of a legitimate driver and / or owner of the vehicle 1500. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in security mode, to stop operation of the vehicle when the owner leaves the vehicle. In this manner, the SoC 1504 provides security against theft and / or vehicle takeover.

[0186] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from the microphone 1596. In contrast to conventional systems that use general classifiers to detect sirens and manually extract features, the SoC 1504 uses CNNs for classification of environmental and urban sounds, as well as classification of visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative terminal velocity of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the mobile vehicle is operating, as identified by the GNSS sensor 1558. Thus, for example, when operating in Europe, the CNN will attempt to detect European sirens, and when in the United States, the CNN will attempt to identify only North American sirens. After an emergency vehicle is detected, the control program may be used to execute emergency vehicle safety routines, such as slowing down the mobile vehicle, stopping it at the side of the road, parking the mobile vehicle, and / or idling the mobile vehicle, with the assistance of the ultrasonic sensor 1562, until the emergency vehicle has passed.

[0187] The vehicle may include a CPU 1518 (e.g., a separate CPU, or a dCPU) that may be coupled to the SoC 1504 via a high-speed interconnect (e.g., PCIe). The CPU 1518 may include, for example, an X86 processor. The CPU 1518 may be used to perform any of a variety of functions including, for example, reconciling potentially inconsistent results between the ADAS sensors and the SoC 1504 and / or monitoring the status and health of the controller 1536 and / or the infotainment SoC 1530.

[0188] The vehicle 1500 may include a GPU 1520 (e.g., a discrete GPU, or a dGPU) that may be coupled to the SoC 1504 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 1520 may provide additional artificial intelligence functionality, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input (e.g., sensor data) from sensors of the vehicle 1500.

[0189] The mobile vehicle 1500 may further include a network interface 1524, which may include one or more wireless antennas 1526 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1524 may be used to enable wireless connections with the cloud over the Internet (e.g., with a server 1578 and / or other network devices), with other mobile vehicles, and / or with computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between the two mobile vehicles and / or an indirect link may be established (e.g., through a network and via the Internet). A direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link may provide the mobile vehicle 1500 information about mobile vehicles in its vicinity (e.g., mobile vehicles in front of, beside, and / or behind the mobile vehicle 1500). This feature may be part of a collaborative adaptive cruise control feature of the vehicle 1500 .

[0190] The network interface 1524 may include a SoC that provides modulation and demodulation functions and enables the controller 1536 to communicate over a wireless network. The network interface 1524 may include a radio frequency front end for upconversion from baseband to radio frequency and downconversion from radio frequency to baseband. The frequency conversion may be performed through well known processes and / or may be performed using a superheterodyne process. In some instances, the radio frequency front end functions may be provided by a separate chip. The network interface may include wireless functions for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0191] The mobile vehicle 1500 may further include a data store 1528, which may include off-chip (e.g., off-SoC 1504) storage. The data store 1528 may include one or more memory elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0192] The vehicle 1500 may further include a GNSS sensor 1558. The GNSS sensor 1558 (e.g., a GPS, an aided GPS sensor, a differential GPS (DGPS) sensor, etc.) aids in mapping, perception, occupancy grid generation, and / or route planning functions. Any number of GNSS sensors 1558 may be used, including, but not limited to, a GPS using a USB connector with an Ethernet to serial (RS-232) bridge, for example.

[0193] The mobile vehicle 1500 may further include a RADAR sensor 1560. The RADAR sensor 1560 may be used by the mobile vehicle 1500 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some instances, the RADAR sensor 1560 may use the CAN and / or bus 1502 for control and to access object tracking data (e.g., to transmit data generated by the RADAR sensor 1560), with access to Ethernet to access raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 1560 may be suitable for front, rear, and side RADAR use. In some instances, a pulsed Doppler RADAR sensor is used.

[0194] The RADAR sensor 1560 may include different configurations, such as long range with a narrow field of view, short range with a wide field of view, and short range side coverage. In some instances, the long range RADAR may be used for adaptive cruise control functions. Long range RADAR systems may provide a wide field of view achieved by two or more independent scans, such as within a 250m range. The RADAR sensor 1560 may help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. Long range RADAR sensors may include monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high speed CAN and FlexRay interfaces. In one instance with six antennas, the center four antennas may create a focused beam pattern designed to record the surroundings of the moving vehicle 1500 at high speeds with minimal interference from traffic in adjacent lanes. The other two antennas may widen the field of view, allowing for rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 1500.

[0195] As an example, a medium-range RADAR system may include a range of up to 1560m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1550 degrees (rear). A short-range RADAR system may include, but is not limited to, a RADAR sensor designed to be mounted on either end of a rear bumper. When mounted on either end of a rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and adjacent to a moving vehicle.

[0196] Short-range RADAR systems may be used in ADAS systems for blind spot detection and / or lane change assist.

[0197] The mobile vehicle 1500 may further include ultrasonic sensors 1562. The ultrasonic sensors 1562, which may be located at the front, rear, and / or sides of the mobile vehicle 1500, may be used for parking assist and / or for creating and updating an occupancy grid. A wide variety of ultrasonic sensors 1562 may be used, with different ultrasonic sensors 1562 being used for different ranges of detection (e.g., 2.5m, 4m). The ultrasonic sensors 1562 may operate at a functional safety level of ASIL B.

[0198] The mobile vehicle 1500 may include a LIDAR sensor 1564. The LIDAR sensor 1564 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 1564 may be functional safety level ASIL B. In some instances, the mobile vehicle 1500 may include multiple (e.g., 2, 4, 6, etc.) LIDAR sensors 1564 that may use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0199] In some instances, the LIDAR sensor 1564 may have the ability to provide a list of objects and their distances in a 360-degree field of view. A commercially available LIDAR sensor 1564 may have an advertised range of about 1500m, for example, with an accuracy of 2cm to 3cm, and with support for a 1500Mbps Ethernet connection. In some instances, one or more non-protruding LIDAR sensors 1564 may be used. In such instances, the LIDAR sensor 1564 may be implemented as a small device that may be integrated into the front, rear, sides, and / or corners of the moving vehicle 1500. In such instances, the LIDAR sensor 1564 may have a range of 200m even for low-reflecting objects and provide up to 120-degree horizontal and 35-degree vertical fields of view. A front-mounted LIDAR sensor 1564 may be configured for a horizontal field of view between 45 and 135 degrees.

[0200] In some instances, LIDAR technology such as 3D flash LIDAR may also be used. 3D flash LIDAR uses a laser flash as a transmitter to illuminate the surroundings of the vehicle up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transit time and the reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR may allow a highly accurate and distortion-free image of the surroundings to be generated with every laser flash. In some instances, four flash LIDAR sensors may be deployed, one on each side of the vehicle 1500. Available 3D flash LIDAR systems include solid-state 3D steering array LIDAR cameras that have no moving parts other than the blower (e.g., non-scanning LIDAR devices). Flash LIDAR devices may use 5 nanosecond Class I (eye-safe) laser pulses per frame and may capture reflected laser light in the form of a 3D range point cloud and coregistered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 1564 may be less susceptible to motion blur, vibration, and / or shock.

[0201] The vehicle may further include an IMU sensor 1566. In some instances, the IMU sensor 1566 may be positioned at the center of the rear axle of the vehicle 1500. The IMU sensor 1566 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some instances, such as in a 6-axis application, the IMU sensor 1566 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1566 may include an accelerometer, a gyroscope, and a magnetometer.

[0202] In some embodiments, the IMU sensor 1566 may be implemented as a miniature, high-performance GPS-Aided Inertial Navigation System (GPS / INS) that combines micro-electro-mechanical system (MEMS) inertial sensors, highly sensitive GPS receivers, and advanced Kalman filtering algorithms to provide estimates of position, velocity, and attitude. As such, in some instances, the IMU sensor 1566 may enable the vehicle 1500 to estimate heading without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 1566. In some instances, the IMU sensor 1566 and the GNSS sensor 1558 may be combined in a single integrated unit.

[0203] The vehicle may include microphones 1596 placed in and / or around the vehicle 1500. The microphones 1596 may be used for emergency vehicle detection and identification, among other things.

[0204] The vehicle may further include any number of camera types, including stereo cameras 1568, wide view cameras 1570, infrared cameras 1572, surround cameras 1574, long and / or mid-range cameras 1598, and / or other camera types. The cameras may be used to capture image data around the entire exterior of the vehicle 1500. The type of cameras used depends on the embodiment and requirements of the vehicle 1500, and any combination of camera types may be used to achieve the required coverage around the vehicle 1500. Additionally, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may support, by way of example only, Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet, but are not limited thereto. Each camera is described in further detail herein in conjunction with Figures 15A and 15B.

[0205] The mobile vehicle 1500 may further include a vibration sensor 1542. The vibration sensor 1542 may measure vibration of a component of the mobile vehicle, such as an axle. For example, a change in vibration may indicate a change in the surface of the road. In another example, when two or more vibration sensors 1542 are used, the difference in vibration may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a powered axle and a free-running axle).

[0206] The mobile vehicle 1500 may include an ADAS system 1538. In some instances, the ADAS system 1538 may include a SoC. The ADAS system 1538 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0207] The ACC system may use a RADAR sensor 1560, a LIDAR sensor 1564, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. The longitudinal ACC monitors and controls the distance to the vehicle directly ahead of the vehicle 1500 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. The lateral ACC performs distance keeping and advises the vehicle 1500 to change lanes when necessary. The lateral ACC is related to other ADAS applications such as LCA and CWS.

[0208] CACC uses information from other moving vehicles, which may be received from other moving vehicles via a wireless link via the network interface 1524 and / or the wireless antenna 1526, or indirectly via a network connection (e.g., via the Internet). A direct link may be provided by a vehicle-to-vehicle (V2V) communication link, while an indirect link may be an infrastructure-to-vehicle (I2V) communication link. In general, a V2V communication concept provides information about the immediately preceding moving vehicle (e.g., the moving vehicle directly ahead of the moving vehicle 1500, in the same lane as the moving vehicle 1500), while an I2V communication concept provides information about the traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given the information of moving vehicles ahead of the moving vehicle 1500, CACC may be more reliable and may potentially make traffic flow smoother and reduce congestion on the roads.

[0209] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audio, visual alerts, vibrations, and / or quick brake pulses.

[0210] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a forward-facing camera and / or RADAR sensor 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, the AEB system typically first alerts the driver to take corrective action to avoid the collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes as part of an effort to prevent, or at least mitigate, the effects of the predicted collision. The AEB system can include techniques such as Dynamic Brake Support and / or Collision Imminent Braking.

[0211] The LDW system provides visual, audible, and / or tactile warnings, such as vibrations in the steering wheel or seat, to alert the driver when the mobile vehicle 1500 crosses a lane marking. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. The LDW system may use a forward-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0212] The LKA system is a modification of the LDW system, which provides steering input or braking to correct the vehicle 1500 if it begins to wander out of its lane.

[0213] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear-facing cameras and / or RADAR sensors 1560 coupled to dedicated processors, DSPs, FPGAs, and / or ASICs electrically coupled to driver feedback, e.g., displays, speakers, and / or vibration components.

[0214] The RCTW system may provide visual, audible, and / or tactile notifications when an object is detected outside the range of the rear camera when the vehicle 1500 is backing up. Some RCTW systems include AEB to ensure vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1560 coupled to a dedicated processor, DSP, FPGA, and / or ASIC electrically coupled to driver feedback, e.g., a display, speaker, and / or vibration components.

[0215] Because conventional ADAS systems alert the driver and allow the driver to determine whether a safety condition truly exists and act accordingly, conventional ADAS systems can be prone to producing false positive results that are usually not catastrophic but can be annoying and distracting to the driver. However, in an autonomous vehicle 1500, the vehicle 1500 itself must decide whether to listen to the results from the primary computer or the secondary computer (e.g., the first controller 1536 or the second controller 1536) when the results are conflicting. For example, in some embodiments, the ADAS system 1538 may be a backup and / or secondary computer to provide perception information to a backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect failures in perception and dynamic driving tasks. Output from the ADAS system 1538 may be provided to a supervisory MCU. When the outputs from the primary and secondary computers conflict, the supervisory MCU must decide how to reconcile the conflict to ensure safe operation.

[0216] In some instances, the primary computer may be configured to provide a reliability score to the overseer MCU indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the overseer MCU may follow the instructions of the primary computer regardless of whether the secondary computers provide conflicting or inconsistent results. If the reliability score does not meet the threshold, and if the primary and secondary computers show different (e.g., conflicting) results, the overseer MCU may arbitrate between the computers to determine the appropriate outcome.

[0217] The supervisory MCU may be configured to execute a neural network that is trained and configured to determine conditions under which the secondary computer provides a false alarm based on the outputs from the primary and secondary computers. Thus, the neural network in the supervisory MCU may learn when the output of the secondary computer may be trusted and when it may not be trusted. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU may learn when the FCW identifies a metal object that is not actually a hazard, such as a sewer grate or manhole cover, that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU may learn to ignore the LDW when a bicyclist or pedestrian is present and lane departure is, in fact, the safest maneuver. In embodiments that include a neural network executing on the supervisory MCU, the supervisory MCU may include at least one of a DLA or a GPU suitable for executing the neural network with associated memory. In a preferred embodiment, the director MCU may comprise and / or be included as a component of the SoC1504.

[0218] In other instances, the ADAS system 1538 may include a secondary computer that performs ADAS functions using classical rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then) and the presence of a neural network in the supervisory MCU may improve reliability, safety and performance. For example, the diverse implementations and intentional non-identity may make the overall system more fault-tolerant, especially to failures caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the primary computer and non-identical software code running on the secondary computer provides the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that a bug in the software or hardware on the primary computer has not caused a critical error.

[0219] In some instances, the output of the ADAS system 1538 may be fed to the perception block of the primary computer and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1538 indicates a forward collision warning due to an object directly ahead, the perception block may use this information when identifying the object. In other instances, the secondary computer may have its own neural network that is trained as described herein, thus reducing the risk of false positives.

[0220] The mobile vehicle 1500 may further include an infotainment SoC 1530 (e.g., an in-vehicle infotainment system (IVI)). Although shown and described as an SoC, the infotainment system need not be an SoC and may include two or more separate components. The infotainment SoC 1530 may include a combination of hardware and software that may be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, wireless data system, mobile vehicle related information such as fuel level, total distance traveled, brake fuel level, oil level, door opening / closing, air filter information, etc.) to the mobile vehicle 1500. For example, the infotainment SoC 1530 may be a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, a car computer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display 1534, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1530 may further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from an ADAS system 1538, autonomous driving information such as planned vehicle maneuvers, trajectory, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0221] The infotainment SoC 1530 may include GPU functionality. The infotainment SoC 1530 may communicate with other devices, systems, and / or components of the mobile vehicle 1500 via the bus 1502 (e.g., CAN bus, Ethernet, etc.). In some instances, the infotainment SoC 1530 may be coupled to a supervisory MCU such that the infotainment system's GPU can perform some self-driving functions in the event that the primary controller 1536 (e.g., the primary and / or backup computer of the mobile vehicle 1500) fails. In such instances, the infotainment SoC 1530 may place the mobile vehicle 1500 in a Chauffeur safe stop mode, as described herein.

[0222] The mobile vehicle 1500 may further include an instrument cluster 1532 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1532 may include a controller and / or a supercomputer (e.g., a separate controller or a supercomputer). The instrument cluster 1532 may include a set of instruments such as a speedometer, fuel level, oil pressure, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine malfunction light, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some instances, information may be displayed and / or shared between the infotainment SoC 1530 and the instrument cluster 1532. In other words, the instrument cluster 1532 may be included as part of the infotainment SoC 1530, or vice versa.

[0223] 15D is a system diagram of communication between a cloud-based server and an example autonomous vehicle 1500 of FIG. 15A in accordance with some embodiments of the present disclosure. The system 1576 may include a server 1578, a network 1590, and a mobile vehicle including the mobile vehicle 1500. The server 1578 may include a number of GPUs 1584(A)-1584(H) (collectively referred to herein as GPUs 1584), PCIe switches 1582(A)-1582(H) (collectively referred to herein as PCIe switches 1582), and / or CPUs 1580(A)-1580(B) (collectively referred to herein as CPUs 1580). The GPUs 1584, CPUs 1580, and PCIe switches may be interconnected with a high-speed interconnect, such as, but not limited to, an NVLink interface 1588 developed by NVIDIA and / or a PCIe connection 1586. In some instances, the GPUs 1584 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 1584 and PCIe switch 1582 are connected via a PCIe interconnect. Although eight GPUs 1584, two CPUs 1580, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each server 1578 may include any number of GPUs 1584, CPUs 1580, and / or PCIe switches. For example, the servers 1578 may each include 8, 16, 32, and / or more GPUs 1584.

[0224] The server 1578 can receive image data from the mobile vehicle over the network 1590, the image data representing images showing unexpected or changed road conditions, such as recently started road construction. The server 1578 can transmit the neural network 1592, the updated neural network 1592, and / or the map information 1594, including information about traffic and road conditions, to the mobile vehicle over the network 1590. The updates to the map information 1594 can include updates to the HD map 1522, such as information about construction sites, potholes, detours, flooding, and / or other obstacles. In some instances, the neural network 1592, the updated neural network 1592, and / or the map information 1594 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment and / or based on training performed at a data center (e.g., using the server 1578 and / or other servers).

[0225] The server 1578 may be used to train a machine learning model (e.g., a neural network) based on the training data. The training data may be generated by a moving vehicle and / or generated in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). The training may be performed according to any one or more classes of machine learning techniques, including, but not limited to, the following classes: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multi-linear subspace learning, manifold learning, representation learning (including preliminary dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, the machine learning model may be used by the vehicle (e.g., transmitted to the vehicle via network 1590) and / or the machine learning model may be used by server 1578 to remotely monitor the vehicle.

[0226] In some instances, the server 1578 can receive data from the moving vehicles and apply the data to state-of-the-art real-time neural networks for real-time intelligent inference. The server 1578 can include deep learning supercomputers and / or dedicated AI computers powered by the GPU 1584, such as the DGX and DGX Station machines developed by NVIDIA. However, in some instances, the server 1578 can include a deep learning infrastructure that uses only CPU-powered data centers.

[0227] The deep learning infrastructure of the server 1578 may be capable of rapid real-time inference and may use that capability to evaluate and validate the health of the processor, software, and / or associated hardware within the moving vehicle 1500. For example, the deep learning infrastructure may receive periodic updates from the moving vehicle 1500, such as sequences of images and / or objects that the moving vehicle 1500 has located within the images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by the moving vehicle 1500, and if the results do not match and the infrastructure concludes that the AI ​​within the moving vehicle 1500 is not functioning properly, the server 1578 may send a signal to the moving vehicle 1500 to infer control, notify passengers, and command the failsafe computer of the moving vehicle 1500 to complete a safe parking maneuver.

[0228] For inference, the server 1578 may include a GPU 1584 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration can enable real-time responsiveness. In other instances, such as when less performance is required, servers powered by CPUs, FPGAs, and other processors may be used for inference.

[0229] Exemplary Computing Device 16 is a block diagram of an example computing device 1600 suitable for use in implementing some embodiments of the present disclosure. The computing device 1600 may include an interconnect system 1602 that indirectly or directly couples the following devices: memory 1604, one or more central processing units (CPUs) 1606, one or more graphic processing units (GPUs) 1608, a communication interface 1610, input / output (I / O) ports 1612, input / output components 1614, a power supply 1616, one or more presentation components 1618 (e.g., displays), and one or more logic units 1620. In at least one embodiment, the computing device 1600 may include one or more virtual machines (VMs) and / or any of its components may include virtual components (e.g., virtual hardware components). As non-limiting examples, one or more of GPUs 1608 may include one or more vGPUs, one or more of CPUs 1606 may include one or more vCPUs, and / or one or more of logical units 1620 may include one or more virtual logical units. As such, computing device 1600 may include discrete components (e.g., an entire GPU dedicated to computing device 1600), virtual components (e.g., a portion of a GPU dedicated to computing device 1600), or a combination thereof.

[0230] Although the various blocks in FIG. 16 are shown connected via interconnect system 1602 with lines, this is not intended to be limiting and is merely for clarity. For example, in some embodiments, a presentation component 1618, such as a display device, may be considered an I / O component 1614 (e.g., where the display is a touch screen). As another example, CPU 1606 and / or GPU 1608 may include memory (e.g., memory 1604 may represent a storage device in addition to the memory of GPU 1608, CPU 1606, and / or other components). In other words, the computing devices of FIG. 16 are merely exemplary. Categories such as “workstations,” “servers,” “laptops,” “desktops,” “tablets,” “client devices,” “mobile devices,” “handheld devices,” “gaming machines,” “electronic control units (ECUs),” “virtual reality systems,” and / or other device or system types are all intended to be within the scope of the computing devices of FIG. 16 and therefore are not to be differentiated.

[0231] The interconnect system 1602 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnect system 1602 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standards association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. As an example, the CPU 1606 may be directly connected to the memory 1604. Additionally, the CPU 1606 may be directly connected to the GPU 1608. When there is a direct or point-to-point connection between components, the interconnect system 1602 may include a PCIe link to implement the connection. In these instances, a PCI bus need not be included in the computing device 1600 .

[0232] Memory 1604 may include any of a variety of computer readable media. Computer readable media may be any available media that can be accessed by computing device 1600. Computer readable media may include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer readable media may comprise computer storage media and communication media.

[0233] Computer storage media may include both volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, and / or other data types. For example, memory 1604 may store computer readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disk (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that may be used to store the desired information and that may be accessed by computing device 1600. As used herein, computer storage media does not include the signals themselves.

[0234] Computer storage media may embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism and include any information delivery media. The term "modulated data signal" may refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media may include wired media, such as a wired network or direct-wired connection, and wireless media, such as acoustic, RF, infrared and other wireless media. Combinations of any of the above should also be included within the scope of computer-readable media.

[0235] The CPU 1606 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 1600 to perform one or more of the methods and / or processes described herein. The CPU 1606 may include one or more (e.g., 1, 2, 4, 8, 28, 72, etc.) cores each capable of simultaneously processing multiple software threads. The CPU 1606 may include any type of processor, and may include different types of processors depending on the type of computing device 1600 implemented (e.g., a processor with fewer cores for a mobile device and a processor with more cores for a server). For example, depending on the type of computing device 1600, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). Computing device 1600 may include one or more CPUs 1606 within one or more microprocessors or auxiliary coprocessors, such as computational coprocessors.

[0236] In addition to or in place of the CPU 1606, the GPU 1608 may be configured to execute at least some of the computer readable instructions to control one or more components of the computing device 1600 to perform one or more of the methods and / or processes described herein. One or more of the GPUs 1608 may be integrated GPUs (e.g., with one or more of the CPUs 1606 and / or one or more of the GPUs 1608 may be discrete GPUs. In an embodiment, one or more of the GPUs 1608 may be a coprocessor or coprocessors of the CPUs 1606. The GPUs 1608 may be used by the computing device 1600 to render graphics (e.g., 3D graphics) or perform general purpose computations. For example, the GPUs 1608 may be configured to perform general purpose computing on a GPU (GPGPU). A graphics processing unit (GPU) 1608 may be used for processing graphics data. The GPU 1608 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 1608 may generate pixel data for an output image in response to rendering commands (e.g., rendering commands from the CPU 1606 received via a host interface). The GPU 1608 may include graphics memory, e.g., display memory, for storing the pixel data or any other suitable data, e.g., GPGPU data. The display memory may be included as part of the memory 1604. GPU 1608 may include two or more GPUs operating in parallel (e.g., via links). The links may connect the GPUs directly (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When coupled together, each GPU 1608 may generate pixel data or GPGPU data for a different portion of the output or for a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with the other GPUs.

[0237] In addition to or instead of the CPU 1606 and / or GPU 1608, the logic unit 1620 may be configured to execute at least some of the computer readable instructions to control one or more of the computing devices 1600 to perform one or more of the methods and / or processes described herein. In an embodiment, the CPU 1606, the GPU 1608, and / or the logic unit 1620 may execute any combination of the methods, processes, and / or portions thereof, either discretely or jointly. One or more of the logic units 1620 may be part of and / or integrated with one or more of the CPU 1606 and / or GPU 1608, and / or one or more of the logic units 1620 may be a discrete component to or otherwise external to the CPU 1606 and / or GPU 1608. In an embodiment, one or more of the logic units 1620 may be a co-processor of one or more of the CPU 1606 and / or GPU 1608.

[0238] Examples of logic unit 1620 include one or more processing cores and / or components thereof, such as a Tensor Core (TC), a Tensor Processing Unit (TPU), a Pixel Visual Core (PVC), a Vision Processing Unit (VPU), a Graphics Processing Cluster (GPC), a Texture Processing Cluster (TPC), a Streaming Multiprocessor (SM), a Tree Traversal Unit (TTU), an Artificial Intelligence Accelerator (AIA), a Deep Learning Accelerator (DLA), an Arithmetic Logic Unit (ALU), an Application Specific Integrated Circuit (ASIC), a Floating Point Unit (FPU), an Input / Output (I / O) element, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or the like.

[0239] Communications interface 1610 may include one or more receivers, transmitters, and / or transceivers that enable computing device 1600 to communicate with other computing devices over electronic communications networks, including wired and / or wireless communications. Communications interface 1610 may include components and functionality to enable communication over any of a number of different networks, such as a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communicating over Ethernet or InfiniBand), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0240] The I / O ports 1612 may enable the computing device 1600 to be logically coupled to other devices, including I / O components 1614, presentation components 1618, and / or other components, some of which may be built-in (e.g., integrated) to the computing device 1600. Exemplary I / O components 1614 include a microphone, a mouse, a keyboard, a joystick, a game pad, a game controller, a satellite dish, a scanner, a printer, a wireless device, and the like. The I / O components 1614 may provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by a user. In some cases, the input may be sent to an appropriate network element for further processing. The NUI may implement any combination of voice recognition, stylus recognition, facial recognition, biometric recognition, gesture recognition on and next to the screen, air gestures, head and eye tracking, and touch recognition in conjunction with the display of the computing device 1600 (as described in more detail below). The computing device 1600 may include a depth camera, such as a stereoscopic camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1600 may include an accelerometer or gyroscope (e.g., as part of an inertia measurement unit (IMU)) to enable detection of motion. In some instances, the output of the accelerometer or gyroscope may be used by the computing device 1600 to render immersive augmented or virtual reality.

[0241] The power supply 1616 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1616 can provide power to the computing device 1600 to enable the components of the computing device 1600 to operate.

[0242] The presentation component 1618 may include a display (e.g., a monitor, a touch screen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component 1618 can receive data from other components (e.g., GPU 1608, CPU 1606, etc.) and output data (e.g., as images, video, sound, etc.).

[0243] Exemplary Data Center 17 illustrates an example data center 1700 that may be used in at least one embodiment of the present disclosure. The data center 1700 may include a data center infrastructure layer 1710, a framework layer 1720, a software layer 1730, and / or an application layer 1740.

[0244] 17, data center infrastructure layer 1710 may include a resource orchestrator 1712, grouped computational resources 1714, and node computational resources ("node CRs") 1716(1)-1716(N), where "N" represents any integer, natural number. In at least one embodiment, node CRs 1716(1)-1716(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules, etc. In some embodiments, one or more of the nodes CR 1716(1)-1716(N) may correspond to a server having one or more of the aforementioned computing resources. Additionally, in some embodiments, the nodes CR 1716(1)-1716(N) may include one or more virtual components, such as a vGPU, a vCPU, and / or the like, and / or one or more of the nodes CR 1716(1)-1716(N) may correspond to a virtual machine (VM).

[0245] In at least one embodiment, the grouped computing resources 1714 may include separate groups of nodes CR 1716 housed in one or more racks (not shown), or multiple racks housed in data centers in various geographic locations (also not shown). The separate groups of nodes CR 1716 in the grouped computing resources 1714 may include grouped computing, network, memory, or storage resources that may be configured or assigned to support one or more workloads. In at least one embodiment, several nodes CR 1716 including CPUs, GPUs, and / or other processors may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0246] The resource orchestrator 1722 may configure or otherwise control one or more nodes CR 1716(1)-1716(N) and / or grouped computational resources 1714. In at least one embodiment, the resource orchestrator 1722 may include a software design infrastructure ("SDI") management entity of the data center 1700. The resource orchestrator 1722 may include hardware, software, or some combination thereof.

[0247] In at least one embodiment, as shown in FIG. 17 , framework layer 1720 may include a job scheduler 1732, a configuration manager 1734, a resource manager 1736, and / or a distributed file system 1738. Framework layer 1720 may include frameworks to support software 1732 in software layer 1730 and / or one or more applications 1742 in application layer 1740. Software 1732 or applications 1742 may include web-based service software or applications, such as those offered by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. Framework layer 1720 may be a type of free and open source software web application framework, such as, but not limited to, Apache Spark™ (hereinafter “Spark”), which may use distributed file system 1738 for large-scale data processing (e.g., “big data”). In at least one embodiment, the job scheduler 1732 may include a Spark driver to facilitate scheduling of workloads supported by various tiers of the data center 1700. The configuration manager 1734 may be capable of configuring different tiers, for example, the software layer 1730 and the framework layer 1720 including Spark and a distributed file system 1738 to support large scale data processing. The resource manager 1736 may be capable of managing clustered or grouped computing resources that are mapped or assigned in support of the distributed file system 1738 and the job scheduler 1732. In at least one embodiment, the clustered or grouped computing resources may include the computing resources 1714 grouped in the data center infrastructure layer 1710. The resource manager 1736 may coordinate with the resource orchestrator 1712 to manage these mapped or assigned computing resources.

[0248] In at least one embodiment, software 1732 included in software layer 1730 may include software used by at least a portion of nodes CR 1716(1)-1716(N), grouped computational resources 1714, and / or distributed file system 1738 of framework layer 1720. The one or more types of software may include, but are not limited to, Internet web page searching software, email virus scanning software, database software, and streaming video content software.

[0249] In at least one embodiment, applications 1742 included in application layer 1740 may include one or more types of applications used by at least a portion of nodes CR 1716(1)-1716(N), grouped computational resources 1714, and / or distributed file system 1738 of framework layer 1720. The one or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0250] In at least one embodiment, any of configuration manager 1734, resource manager 1736, and resource orchestrator 1712 can implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. The self-rewriting actions can free data center operators of data center 1700 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming portions of the data center.

[0251] Data center 1700 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculation of weight parameters by a neural network architecture using the software and / or computing resources described above with respect to data center 1700. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described above with respect to data center 1700, for example, by using weight parameters calculated via one or more training techniques, including but not limited to those described herein.

[0252] In at least one embodiment, data center 1700 may use CPUs, application specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or corresponding virtual computing resources) for training and / or performing inference using the aforementioned resources. Additionally, one or more of the aforementioned software and / or hardware resources may be configured as services, such as image recognition, speech recognition, or other artificial intelligence services, to enable a user to train or perform inference on information.

[0253] Example Network Environment A network environment suitable for use in implementing embodiments of the present disclosure may include one or more client devices, servers, network attached storage (NAS), other back-end devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be implemented with one or more instances of computing device 1600 of FIG. 16, e.g., each device may include similar components, features, and / or functionality of computing device 1600. In addition, if a back-end device (e.g., server, NAS, etc.) is implemented, the back-end device may be included as part of data center 1700, examples of which are further detailed herein with respect to FIG. 17.

[0254] Components of a network environment may communicate with each other via a network, which may be wired, wireless, or both. A network may include multiple networks or a network of networks. Illustratively, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. When a network includes a wireless telecommunications network, components such as base stations, communication towers, or access points (as well as other components) may provide wireless connectivity.

[0255] Compatible network environments may include one or more peer-to-peer network environments (wherein a server may not be included in the network environment) and one or more client-server network environments (wherein a server or servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to a server may be implemented in any number of client devices.

[0256] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support software in the software layer and / or one or more applications in the application layer. The software or applications may include web-based service software or applications, respectively. In an embodiment, one or more of the client devices may use the web-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open source software web application framework that may use a distributed file system for large-scale data processing (e.g., "big data"), for example.

[0257] A cloud-based network environment may provide cloud computing and / or cloud storage performing any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across a state, territory, country, or world). When a connection to a user (e.g., a client device) is relatively close to an edge server, the core server may delegate at least a portion of the functionality to the edge server. A cloud-based network environment may be private (e.g., restricted to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0258] A client device may include at least some of the components, features, and functionality of the exemplary computing device 1600 described herein with respect to Figure 16. By way of illustration, and not by way of limitation, a client device may be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smart watch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a boat, an airship, a virtual machine, a drone, a robot, a handheld communications device, a hospital device, a gaming device or system, an entertainment system, a vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these depicted devices, or any other suitable device.

[0259] The present disclosure may be described in the general context of computer code or machine usable instructions, including computer executable instructions, such as program modules, being executed by a computer or other machine, such as a personal digital assistant or other handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs particular tasks or implements particular abstract data types. The present disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0260] As used herein, the statement "and / or" with respect to two or more elements should be interpreted to mean only one element or a combination of elements. For example, "element A, element B, and / or element C" may include element A only, element B only, element C only, elements A and element B, elements A and element C, elements B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Furthermore, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0261] The subject matter of the present disclosure has been described with specificity to meet statutory requirements. However, the description itself is not intended to limit the scope of the present disclosure. Instead, the inventors intend that the claimed subject matter may be implemented in other ways, including different steps or combinations of steps similar to those described in this document, in conjunction with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to connote different elements of the method used, these terms should not be construed as implying any particular order between the various steps disclosed herein unless and when the order of the individual steps is explicitly described.

[0262] Illustrative clauses In one or more embodiments, a method includes generating sensor data using one or more sensors of the vehicle for at least two frames of a plurality of frames corresponding to driving of the vehicle, calculating an output indicative of a location of a landmark using one or more neural networks (NNs) and based at least in part on the sensor data, converting the location to a three-dimensional (3D) world space location relative to an origin of the vehicle, encoding the 3D world space location to generate encoded data corresponding to the frames, and transmitting the encoded data to a server to cause the server to generate a map including the landmarks.

[0263] In one or more embodiments, the sensor data includes image data representing an image, one or more of the outputs are calculated in two-dimensional (2D) image space, and converting the position includes converting the 2D image space position to a 3D world space position based at least in part on at least one of intrinsic or extrinsic camera parameters corresponding to a camera that generated the image data.

[0264] In one or more embodiments, the landmarks include one or more of the following: lane dividers; road boundaries, signs, poles, waiting conditions, vertical structures, other road users, static objects, or dynamic objects.

[0265] In one or more embodiments, the output further represents at least one of a pose of the landmark or a geometry of the landmark.

[0266] In one or more embodiments, the output further represents semantic information corresponding to the landmarks.

[0267] In one or more embodiments, the landmarks include lane dividers, and the method further includes, for at least two lane dividers, combining respective detections of the at least two lane dividers to generate a continuous lane divider representation.

[0268] In one or more embodiments, the method further includes determining, for each frame of the at least two frames, a translation and rotation of the vehicle relative to a previous frame based at least in part on the sensor data, where encoding further includes encoding the rotation and translation.

[0269] In one or more embodiments, the method further includes compressing the data representing the rotation and translation using a delta compression algorithm to generate delta compressed data, where encoding the rotation and translation further includes encoding the delta compressed data.

[0270] In one or more embodiments, the sensor data includes Global Navigation Satellite System (GNSS) data, and the method further includes determining, for each frame of the at least two frames, a global position of the vehicle based at least in part on the GNSS data, where encoding further includes encoding the global position.

[0271] In one or more embodiments, the sensor data includes LiDAR data, and the method further includes filtering out dynamic objects from the LiDAR data to generate filtered LiDAR data, where the encoded data further represents the filtered LiDAR data.

[0272] In one or more embodiments, the encoded data further represents sensor data, the sensor data including one or more of LiDAR data, RADAR data, ultrasonic data, ultrasound data, GNSS data, image data, or inertial measurement unit (IMU) data.

[0273] In one or more embodiments, the method further includes receiving, from the server, data representing a request for the location of the landmark, where transmitting the encoded data is based at least in part on the request.

[0274] In one or more embodiments, the method further includes determining to generate each frame of the at least two frames based at least in part on one or more of a time threshold being met or a distance threshold being met.

[0275] In one or more embodiments, the encoded data is encoded as structured data that is serialized using protocol buffers.

[0276] In one or more embodiments, the server includes at least one of a data center server, a cloud server, or an edge server.

[0277] In one or more embodiments, the method includes generating one or more of LiDAR data or RADAR data using one or more sensors of the vehicle for at least two frames of a plurality of frames corresponding to driving of the vehicle, filtering out points corresponding to the one or more of the LiDAR data or RADAR data to generate filtered data, determining a pose of the vehicle relative to a previous frame, encoding the filtered data and the pose to generate encoded data, and transmitting the encoded data to a cloud server to cause the cloud server to generate at least one of a LiDAR layer of the map or a RADAR layer of the map using the pose.

[0278] In one or more embodiments, the filtering out includes at least one of dynamic object filtering or deduplication.

[0279] In one or more embodiments, the method further includes generating sensor data using one or more sensors of the vehicle and calculating an output indicative of a location of the landmark using one or more neural networks (NNs) and based at least in part on the sensor data, where the transmitting further includes transmitting data indicative of the location of the landmark.

[0280] In one or more embodiments, encoding the filtered data includes encoding the filtered data using an octree.

[0281] In one or more embodiments, the method further includes compressing the filtered data using quantization.

[0282] In one or more embodiments, the system includes one or more sensors, one or more processors, and one or more memory devices having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations including generating sensor data using the one or more sensors, in a current frame, calculating an output indicative of a position of a landmark using one or more neural networks (NNs) and based at least in part on the sensor data, transforming the position to a three-dimensional (3D) world space position relative to an origin of the vehicle, determining a pose for the current frame relative to a previous pose for the previous frame, encoding the 3D world space position and pose to generate encoded data corresponding to the current frame, and transmitting the encoded data.

[0283] In one or more embodiments, the system may be included in at least one of a control system of an autonomous or semi-autonomous machine, a perception system of an autonomous or semi-autonomous machine, a system for performing simulation operations, a system for performing deep learning operations, a system implemented using edge devices, a system implemented using a robot, a system incorporating one or more virtual machines (VMs), a system implemented at least partially in a data center, or a system implemented at least partially using cloud computing resources.

[0284] In one or more embodiments, the method includes computing, using one or more neural networks (NNs) and based at least in part on sensor data generated by one or more sensors of the vehicle, an output indicative of a location in a two-dimensional (2D) image space corresponding to the detected landmarks; generating a distance function representation of the detected landmarks based at least in part on the locations; and, for at least two of a plurality of poses of the vehicle represented in the cost space, projecting corresponding map landmarks in the map into the 2D image space to generate projected map landmarks; comparing the projected map landmarks to the distance function representation, computing a cost based at least in part on the comparison, and updating points in the cost space corresponding to each pose of the at least two poses based at least in part on the cost to generate a cost space; and localizing the vehicle on the map based at least in part on the cost space.

[0285] In one or more embodiments, the localizing is to an origin of one road segment of a plurality of road segments of the map, and the method further includes localizing the vehicle in a global coordinate system based at least in part on localizing the vehicle to the origin of the road segment.

[0286] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes generating a plurality of additional cost spaces corresponding to a plurality of previous frames; generating an aggregate cost space corresponding to the cost space and the plurality of additional cost spaces, the generating aggregate cost space including using self-motion compensation for the current frame; and applying a filter to the aggregate cost space to determine a position representation of the vehicle for the current frame, where the localizing is further based at least in part on the position representation.

[0287] In one or more embodiments, the filter includes a Kalman filter.

[0288] In one or more embodiments, the position representation includes an ellipsoid.

[0289] In one or more embodiments, each of the multiple cost spaces is generated based at least in part on a respective output calculated based at least in part on respective sensor data corresponding to a respective frame of the multiple frames.

[0290] In one or more embodiments, one or more of the multiple additional cost spaces are generated to correspond to a first road segment, and generating the aggregate cost space includes transforming one or more of the additional cost spaces to correspond to a second road segment that corresponds to the cost space of the current frame.

[0291] In one or more embodiments, the transforming includes a translation transformation and a rotation transformation.

[0292] In one or more embodiments, the localizing corresponds to a modality of localization, and the method further includes localizing the vehicle using one or more additional modalities of localization and fusing the modality of localization with the one or more additional modalities of localization to determine a final localization result.

[0293] In one or more embodiments, the one or more additional modalities of localization include a LiDAR modality, a RADAR modality, or a fusion modality.

[0294] In one or more embodiments, the localization modality corresponds to a camera modality.

[0295] In one or more embodiments, the fusing includes applying a Kalman filter to the localization results from the localization modality and the additional modality of localization.

[0296] In one or more embodiments, one or more of the map landmarks or detected landmarks correspond to at least one of a lane divider, a road boundary, a sign, a pole, a vertical structure, a waiting condition, a static object, or a dynamic actor.

[0297] In one or more embodiments, generating the cost space further includes, for at least two of the vehicle poses represented in the cost space, projecting map landmarks corresponding to the map into another 2D image space to generate additional projected map landmarks, comparing map semantic information of the additional projected map landmarks to the detected semantic information of the detected landmarks, and calculating another cost based at least in part on comparing the map semantic information to the detected semantic information.

[0298] In one or more embodiments, a method includes generating LiDAR data using one or more LiDAR sensors of a vehicle, projecting points corresponding to the LiDAR data for at least two of a plurality of poses of the vehicle represented in a cost space onto a distance function representation of a LiDAR point cloud corresponding to a LiDAR layer of a map, comparing the points to the distance function representation, calculating a cost based at least in part on the comparison, and updating points in the cost space corresponding to each pose of the at least two poses based at least in part on the cost, generating a cost space; and localizing the vehicle on the map based at least in part on the cost space.

[0299] In one or more embodiments, the method further includes determining a point from within a larger set of points represented by the LiDAR data based at least in part on an elevation angle value corresponding to the point being within an elevation angle range, where the LiDAR point cloud corresponds to the elevation angle range.

[0300] In one or more embodiments, generating the cost space further includes, for each of the at least two poses of the vehicle represented in the cost space, projecting an elevation representation corresponding to the LiDAR data onto a map elevation representation corresponding to a LiDAR layer of the map, comparing the elevation representation to the map elevation representation, and calculating a further cost based at least in part on comparing the elevation representation to the map elevation representation.

[0301] In one or more embodiments, comparing the elevation angle representation to the map elevation angle representation includes adjusting at least one of the elevation angle representation or the map elevation angle representation based at least in part on a difference between a vehicle origin of the vehicle or a road origin of the road segment to which the map elevation angle representation corresponds.

[0302] In one or more embodiments, generating the cost space further includes, for each of the at least two poses of the vehicle represented by the cost space, projecting an intensity representation corresponding to the LiDAR data onto a map intensity representation corresponding to a LiDAR layer of the map, comparing the intensity representation to the map intensity representation, and calculating a further cost based at least in part on comparing the intensity representation to the map intensity representation.

[0303] In one or more embodiments, the localizing is to an origin of one road segment of a plurality of road segments of the map, and the method further includes localizing the vehicle in a global coordinate system based at least in part on localizing the vehicle to the origin of the road segment.

[0304] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes generating a plurality of additional cost spaces corresponding to a plurality of previous frames; generating an aggregate cost space corresponding to the cost space and the plurality of additional cost spaces, the generating aggregate cost space including using self-motion compensation for the current frame; and applying a filter to the aggregate cost space to determine a position representation of the vehicle for the current frame, where the localizing is further based at least in part on the position representation.

[0305] In one or more embodiments, the filter includes a Kalman filter.

[0306] In one or more embodiments, the position representation includes an ellipsoid.

[0307] In one or more embodiments, one or more of the multiple additional cost spaces are generated to correspond to a first road segment, and generating the aggregate cost space includes transforming one or more of the additional cost spaces to correspond to a second road segment that corresponds to the cost space of the current frame.

[0308] In one or more embodiments, the localizing corresponds to a modality of localization, and the method further includes localizing the vehicle using one or more additional modalities of localization, and blending the modality of localization with the one or more additional modalities of localization to determine a final localization result.

[0309] In one or more embodiments, the modality of localization includes a LiDAR modality, and the one or more additional modalities of localization include a RADAR modality, an imaging modality, or a fusion modality.

[0310] In one or more embodiments, the system includes one or more sensors, one or more processors, and one or more memory devices having instructions stored thereon that, when executed by the one or more processors, cause the one or more processors to perform operations including generating sensor data using the one or more sensors; projecting points corresponding to the sensor data for at least two of the plurality of poses represented in the cost space into a distance function representation of a point cloud corresponding to a map; comparing the points to the distance function representation; calculating a cost based at least in part on the comparison; and updating points in the cost space corresponding to each pose of the at least two poses based at least in part on the cost to generate a cost space; and localizing the vehicle on the map based at least in part on the cost space.

[0311] In one or more embodiments, the system may be included in at least one of: a control system of an autonomous or semi-autonomous machine; a perception system of an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system incorporating one or more virtual machines (VMs); a system implemented using a robot; a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

[0312] In one or more embodiments, the localizing is to an origin of one road segment of a plurality of road segments of the map, and the method further includes localizing the vehicle in a global coordinate system based at least in part on localizing the vehicle to the origin of the road segment.

[0313] In one or more embodiments, the cost space corresponds to a current frame, and the method further includes generating a plurality of additional cost spaces corresponding to a plurality of previous frames; generating an aggregate cost space corresponding to the cost space and the plurality of additional cost spaces, the generating aggregate cost space including using self-motion compensation for the current frame; and applying a filter to the aggregate cost space to determine a position representation of the vehicle for the current frame, where the localizing is further based at least in part on the position representation.

[0314] In one or more embodiments, the sensor data corresponds to one or more of LiDAR data or RADAR data, and the one or more sensors include one or more of a LiDAR sensor or a RADAR sensor.

Claims

1. 1. A method comprising: receiving a plurality of map streams corresponding to a plurality of maneuvers; determining segments of two or more of the map streams that are within a threshold distance of each other; registering at least one pair of segments from the two or more map stream segments to generate a frame graph; partitioning the frame graph into road segments to generate a pose graph including poses corresponding to vehicle positions within each of the road segments and corresponding pose links; fusing data from the two or more map stream segments based at least in part on the pose graph to generate a fused map; and transmitting the fused map to the vehicle to perform one or more actions based at least in part on the fused map.

2. 2. The method of claim 1 , further comprising converting segments of the two or more map streams into respective maps, and wherein the fusing of the data from segments of the two or more map streams comprises fusing the data from the respective maps.

3. The method of claim 2 , wherein the respective maps include two or more map layers, each corresponding to a different sensor modality.

4. The method of claim 1 , wherein the one or more actions include at least one of localizing the vehicle, performing path planning for the vehicle, or controlling the vehicle.

5. The method of claim 1 , wherein the multiple map streams are generated by multiple vehicles.

6. The method of claim 1 , wherein the step of transmitting the fused map comprises transmitting a subset of the fused map that corresponds to a sensor modality present on the vehicle.

7. 11. A method comprising: receiving, from each of a plurality of maneuvers, sensory data, sensory output from one or more deep neural networks (DNNs), and data representing trajectory information corresponding to the maneuvers; converting the data into first map data representing a plurality of individual maps, each corresponding to one of the plurality of maneuvers; registering a first segment of a first individual map of the plurality of individual maps to a second segment of a second individual map of the plurality of individual maps based at least in part on the trajectory information to generate a frame graph; assigning poses from the frame graph to road segments to generate a pose graph; generating second map data representing a fused map corresponding to the first individual map and the second individual map based at least in part on the pose graph, the fused map including the road segment; and transmitting the data representing the fused map to the vehicle to perform vehicle localization with respect to the road segment.

8. The method of claim 7 , wherein the road segment has a corresponding global position, and the step of localizing the vehicle with respect to the road segment further comprises the step of localizing the vehicle with respect to a global coordinate system.

9. The method of claim 7 , wherein each individual map includes one or more map layers corresponding to different sensor modalities.

10. 8. The method of claim 7, further comprising: receiving additional data corresponding to additional maneuvers based at least in part on a quality of the localization results; and updating the fused map based at least in part on the additional data.

11. 8. The method of claim 7, further comprising: running one or more optimization algorithms on the pose graph to generate an updated pose graph, wherein the generating the second map data representing the fused map is based at least in part on the updated pose graph.

12. The method of claim 7 , wherein the data representing the sensor data, the sensory outputs, and the trajectory information are pre-processed by each vehicle that generated the data to minimize the amount of the data.

13. 8. The method of claim 7, wherein the data representing at least one of the sensor data, the sensory output, or the trajectory information is received in a compressed format, the method further comprising the step of expanding the data in the compressed format, and wherein the step of converting the data into the first map data comprises the step of converting the data after the expansion.

14. The method of claim 7 , wherein the step of registering comprises geometric registration.

15. 8. The method of claim 7, further comprising: assigning one or more origins to the road segment; computing a transformation between the one or more origins of adjacent road segments of the road segment; and encoding the transformation in the fused map, wherein the localizing the vehicle is based at least in part on the transformation.

16. 8. The method of claim 7, further comprising determining to register the first and second segments based at least in part on Global Navigation Satellite System (GNSS) data corresponding to the first and second individual maps.

17. 8. The method of claim 7, wherein the sensory output corresponds to a three-dimensional (3D) representation of detected landmarks, the detected landmarks including one or more of a lane divider, a road boundary, a sign, a pole, a static object, or a vertical structure.

18. The method of claim 7 , wherein the sensor data corresponds to at least one of LiDAR data, RADAR data, or ultrasound data.

19. 1. A system comprising: one or more processors; and one or more memory devices having instructions stored thereon, the instructions, when executed by the one or more processors, causing the one or more processors to perform the following: receive a plurality of map streams corresponding to a plurality of maneuvers; determine segments of two or more map streams that are within a threshold distance of each other; geometrically register pairs of the segments to generate a frame graph; partition the frame graph into road segments to generate a pose graph including poses and corresponding pose links corresponding to each of the road segments; fuse data from the map streams based at least in part on the pose graph to generate a fused map; and transmit the fused map to a vehicle to perform one or more actions based at least in part on the fused map.

20. 20. The system of claim 19, further comprising: converting the map stream into a respective map, and wherein the fusing of the data from the map stream comprises fusing the data from the respective maps.

21. 21. The system of claim 20, wherein each of the respective maps includes a map layer corresponding to a different sensor modality.

22. 20. The system of claim 19, wherein the one or more operations include at least one of localizing the vehicle, performing route planning for the vehicle, or controlling the vehicle.

23. The system of claim 19 , wherein the map stream is generated by a plurality of vehicles.

24. 20. The system of claim 19, wherein said transmitting the fused map comprises transmitting a subset of the fused map that corresponds to a sensor modality present on the vehicle.

25. 20. The system of claim 19, wherein the system is included in at least one of a control system of an autonomous or semi-autonomous machine, a perception system of an autonomous or semi-autonomous machine, a system for performing simulation operations, a system for performing deep learning operations, a system implemented using edge devices, a system incorporating one or more virtual machines (VMs), a system implemented at least partially in a data center, or a system implemented at least partially using cloud computing resources.

Images