Intersection region detection and classification for autonomous machine applications

Abstract

In various examples, live perception from sensors of a vehicle can be utilized to detect and classify intersection contest areas in a vehicle’s environment in real-time or near real-time. For example, a deep neural network (DNN) can be trained to compute outputs (e.g., signed distance functions) that can correspond to boundary locations that demarcate intersection contest areas. The signed distance functions can be decoded and / or post-processed to determine instance segmentation masks that represent locations and classifications of intersection zones or areas. Locations of intersection zones or areas can be generated in image space and converted to world space coordinates to assist an autonomous or semi-autonomous vehicle in navigating an intersection according to road rules, traffic priority considerations, etc.

Description

BACKGROUND

[0001] Autonomous driving systems and advanced driver assistance systems (ADAS) can use various sensors, such as cameras, LIDAR sensors, RADAR sensors, etc., to perform various tasks - such as lane keeping, lane changing, lane assignment, camera calibration, turning, path planning, and localization. For example, to enable autonomous and ADAS systems to operate independently and efficiently, a real-time or near real-time understanding of the vehicle’s surroundings can be generated. Such an understanding can include information about the location and type of objects, obstacles, lanes, and / or intersections in the environment relative to various demarcations, such as lanes, road boundaries, intersections, etc. Information about the surroundings can be used by the vehicle when making decisions, such as which path or trajectory to follow, how to apply road rules, and when, where, and for how long to stop.

[0002] For example, information about the location, properties, and layout of intersections in the environment of an autonomous or semi-autonomous vehicle - such as where other vehicles or pedestrians can be located, whether other vehicles have priority to cross an intersection, etc. - can prove valuable when making path planning, obstacle avoidance, and / or control decisions. This is especially important when vehicles are operating in urban and / or semi-urban driving environments, where intersection understanding and path planning become critical due to the increased number of variables compared to highway driving environments. For example, in a two-way multi-lane driving environment, determining the location and classification of intersection areas (e.g., pedestrian crosswalks, vehicle entries, vehicle exits, no-lane, etc.) is critical for safe and efficient autonomous and / or semi-autonomous driving when vehicles must slow to a complete stop and wait for other vehicles with priority to cross an intersection.

[0003] In conventional systems, an intersection can be interpreted by separately detecting and combining several features of the intersection and surrounding environment. For example, to detect an intersection, the intersection region, multiple objects (e.g., traffic lights, stop signs), vehicle position, vehicle direction, lanes, free space boundaries, etc. can be separately detected—e.g., using multiple separate deep neural networks (DNNs)—and pieced together to generate a comprehensive understanding of a single intersection. However, such solutions require accurate and detailed networks to identify relevant features for intersection detection and to combine various features for each intersection to safely traverse the intersection while adhering to different traffic priority rules. Thus, the more complex the intersection, the more detailed the annotations required to train the DNNs—thereby increasing the complexity of accurately detecting and classifying intersection regions while reducing the scalability of intersection detection. In particular, these conventional systems can fail to provide a clear and complete understanding of vehicles and / or pedestrians crossing the intersection that can be competitors with priority to cross the intersection. Moreover, many conventional systems also fail to robustly detect intersection regions that encounter occlusions in the sensor data—e.g., due to buildings, or other vehicles or objects blocking certain portions of a vehicle’s sensor’s sensing field.

[0004] Other conventional systems can analyze an intersection by comparing separately detected features to features in a pre-stored high-definition (HD), three-dimensional (3D) map of vehicle drivable surfaces. However, this map-based solution is highly dependent on the accuracy and availability of the map. Thus, these conventional systems fail when the map for a certain region is out-of-date or unavailable. For example, the process can be more logically complex when a large geographic region (e.g., a city, state, country) needs to be manually labeled for a vehicle to independently and effectively navigate in locations with different intersection types. These conventional systems also fail when there are transient intersection conditions (e.g., a police directing traffic or a school bus stopping) that can not be reflected in the map. SUMMARY

[0005] Embodiments of the present disclosure relate to intersection contested region detection for autonomous machine applications. Systems and methods are disclosed that use a deep neural network (DNN) to process sensor data from one or more sensors of a vehicle to detect an intersection region and a classification corresponding thereto, thereby generating a real-time or near real-time understanding of the intersection as a whole.

[0006] In contrast to conventional systems (e.g., the system described above), the current system can use live perception of a vehicle to detect and classify intersection contest areas in the vehicle’s environment. For example, a machine learning model (e.g., a DNN) can be trained to compute an output corresponding to an intersection, such as a signed distance function corresponding to intersection zones or regions and their classes, and a post-processor can use this information to efficiently and accurately determine intersection zone locations and classifications. Various techniques (e.g., smoothing, clustering, thresholding, upsampling, etc.) can be used to post-process the output to determine 3D world space locations of detected intersection zones. The vehicle can use these world space locations and / or classifications of intersection zones to safely and accurately navigate intersections. For example, the output of the DNN can be used directly or indirectly (e.g., by decoding) to determine the location of each intersection zone and the classification corresponding thereto.

[0007] Because the overall approach is used in real-time or near real-time to generate an understanding of each intersection zone of an intersection, the process of detecting and classifying intersection contest areas can be relatively less time-consuming, less computationally intensive, and more scalable because the system can learn to diagnose each intersection without prior experience or knowledge of the intersection and without individually detecting and then combining several features of the intersection. As a result, the vehicle can be able to more freely autonomously drive in urban, city, and / or other environments without relying on, or in addition to, high-definition maps. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present system and method for intersection contest area detection for autonomous machine applications are described in detail below with reference to the attached drawing figures, wherein:

[0009] FIG. 1 is an example dataflow diagram illustrating an example process for training a neural network to detect intersection zones using output from sensors of a vehicle for navigating an intersection according to some embodiments of the present disclosure;

[0010] FIGS. 2A-2B is an illustration of example annotations of images used to generate ground truth data for training a neural network to detect intersection zones according to some embodiments of the present disclosure;

[0011] FIG. 3 is an illustration of an example ground truth signed distance function encoding intersection zones from annotations according to some embodiments of the present disclosure;

[0012] FIG. 4 is a flowchart illustrating an example method for training a neural network to detect intersection zones according to some embodiments of the present disclosure;

[0013] FIG. 5This is a data flow diagram illustrating an example process for detecting an intersection area in real time or near real time using the output of sensors from a vehicle, according to some embodiments of the present disclosure;

[0014] FIG. 6 This is a visualization of example signed distance functions converted into corresponding instance segmentation masks using a post-processor according to some embodiments of the present disclosure;

[0015] FIGS. 7A-7B This is a visualization of an example intersection area predicted in 3D space based on a signed distance function calculated by a neural network, according to some embodiments of the present disclosure;

[0016] FIG. 8 This is a flowchart illustrating an example method for detecting intersection areas according to some embodiments of the present disclosure;

[0017] FIG. 9A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0018] FIG. 9B According to some embodiments of this disclosure FIG. 9A Examples of camera positions and field of view for autonomous vehicles;

[0019] FIG. 9C According to some embodiments of this disclosure FIG. 9A A block diagram of an example system architecture for an example autonomous vehicle;

[0020] FIG. 9D Cloud-based servers and according to some embodiments of this disclosure FIG. 9A A system diagram illustrating communication between example autonomous vehicles; and

[0021] FIG. 10 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0022] Systems and methods related to intersection competition zone detection for autonomous machine applications are disclosed. Although this disclosure may be described with reference to an example autonomous vehicle 900 (which may be alternatively referred to herein as "vehicle 900" or "autonomous vehicle 900"), examples are referenced. FIGS. 9A-9DThe description is made with reference to the drawings, but is not intended to be limiting. For example, the systems and methods described herein can be used by, without limitation, non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), robots, warehouse vehicles, off-road vehicles, spacecraft, watercraft, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, engineering vehicles, underwater vehicles, drones, and / or other vehicle types. Further, while the present disclosure can be described with reference to intersection structures for vehicle applications, this is not intended to be limiting, and the systems and methods described herein can be used for augmented reality, virtual reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical space in which detection of intersections or other environmental structures and / or poses can be used.

[0023] As described herein, the current systems and methods provide techniques for detecting and classifying intersection regions in real-time or near real-time using output from vehicle sensors (e.g., cameras, RADAR sensors, LIDAR sensors, etc.) as compared to conventional approaches to intersection contest region detection using high-definition (HD) maps. Thus, for each intersection, the live perception of the vehicle can be used to detect the location and / or properties or classification of the intersection region corresponding to that intersection. Computer vision and / or machine learning models (e.g., deep neural networks (DNNs), such as convolutional neural networks (CNNs)) can be trained to compute outputs — in embodiments, after decoding — that result in the detected intersection regions and / or their classification or properties, and the vehicle can use these outputs in conjunction with object and / or lane detection to navigate the intersection effectively and accurately while complying with the associated traffic priority rules. In some embodiments, the outputs of the computer vision and / or machine learning models can include signed distance functions that represent the pixel-based locations of the particular regions of the image in which the various classified intersection regions are detected. Further, in some embodiments, post-processing can be performed on the signed distance functions to generate instance segmentation masks corresponding to each detected intersection region and / or type.

[0024] Thus, by using live perception to generate an understanding of the intersection regions, the process of detecting and classifying the intersection regions can be relatively less time-consuming, less computationally intensive, and more scalable, as the system can learn to diagnose each intersection region in real-time or near real-time. Further, as compared to conventional systems, no prior knowledge or experience of the intersection is required — e.g., the system can perform accurately without the availability and up-to-dateness of a high-definition map.

[0025] In deployment, sensor data (e.g., images, video, point clouds, depth maps, other sensor data representations, etc.) can be received and / or generated using sensors (e.g., cameras, RADAR sensors, LIDAR sensors, etc.) located on or otherwise arranged on an autonomous or semi-autonomous vehicle. The sensor data can be applied to a DNN trained to identify regions of interest related to intersections (e.g., intersection interior, intersection entrance, intersection exit, no-lane, crosswalk, unclear area, etc.) and semantic information related thereto represented by the sensor data. More specifically, the DNN can be trained to compute a signed distance function corresponding to each instance of an intersection region detected. For each pixel or point in the sensor data, the signed distance function can correspond to a (pixel) distance to the nearest boundary corresponding to an intersection region, a position relative to the intersection region (e.g., pixels outside the intersection region are negative, pixels inside the intersection region are positive, and vice versa), semantic information corresponding to the intersection region (e.g., crosswalk, intersection entrance, intersection exit, unclear area, no-lane, intersection interior, partially visible, fully visible, etc., or combinations thereof), and / or other information. In some examples, the intersection region in the computed signed distance function can be represented by pixels or points represented by the sensor data, where pixels on the boundary of the intersection region are represented by zero values, pixels inside the boundary are represented by positive distance values corresponding to a distance to the nearest boundary of the region, and pixels outside the boundary are represented by negative values corresponding to a distance to the nearest boundary of the region. One or more instances of each class of intersection region can be represented by a separate signed distance function (e.g., one signed distance function per class, one separate signed distance function per instance of each class, etc.). Thus, the location and class of an intersection region can be determined based on the signed distance function and related information— e.g., using a post-processor. As non-limiting examples, intersection region classes can include crosswalk, intersection entrance, intersection exit, unclear area, no-lane, intersection interior, partially visible, fully visible, and / or other information corresponding to an intersection.

[0026] A DNN can be trained to predict various types of information corresponding to intersection region classifications - e.g., through any number of channels. For example, one or more channels can represent a signed distance function for a class of intersection regions and / or each instance thereof. During training, the DNN can be trained with image or other sensor data representations that are labeled or annotated with polygons representing crosswalk regions, intersection entry regions, intersection exit regions, unclear regions, no-lane regions, intersection interior regions, partially visible regions, fully visible regions, etc. In addition to boundary labels or annotations, ground truth annotations can include semantic information corresponding to each boundary - e.g., a classification. The labeled polygons and semantic information can be used by a ground truth encoder to generate signed distance functions, each corresponding to an intersection region type corresponding to an intersection region determined from the annotations. In some examples, each pixel in a signed distance function can be encoded with a distance value corresponding to a pixel distance to a nearest boundary of an intersection region - e.g., along an annotated boundary shape of the training data. In such examples, pixels within an intersection region can also be encoded with positive distance values, pixels outside an intersection region can be encoded with negative distance values, and vice versa. Pixels forming boundaries of an intersection region can be encoded with zero values to generate ground truth data.

[0027] In some examples, in addition to or instead of using signed distance functions, a ground truth encoder can use the labeled polygons and semantic information to generate a one-hot encoded map for each intersection region type, where pixels within an intersection region boundary are encoded with a value of 1 and pixels outside an intersection region boundary are encoded with a value of 0. However, when using one-hot encoding, adjacent regions of the same type (e.g., with shared boundaries) can not be separately identifiable, thus potentially reducing accuracy as compared to embodiments using signed distance functions. Ultimately, intersection regions can be encoded using signed distance functions and / or one-hot encoded maps requiring limited labeling, as the information can be determined using polygon annotations and semantic information.

[0028] Once the DNN is trained, the DNN can regress outputs in the form of signed distance functions, one-hot encoded maps, and / or other outputs corresponding to the encoded ground truth channels predicted by the training network, as described herein. In cases where the outputs correspond to signed distance functions, the pixel distances corresponding to different classification types and / or instances thereof can be post-processed into 2D coordinates representing the 2D locations of the intersection regions (e.g., every pixel within a region, just the boundary pixels or vertices thereof, etc.). In some embodiments, the signed distance functions can be Gaussian smoothed on a pixel-by-pixel basis to smooth out spikes and fluctuations in the pixel values. Each of the smoothed signed distance functions can then be mapped to a polygonal region representing an intersection region based on the negative and positive pixel values associated with each pixel in the corresponding signed distance function. In some embodiments, a watershed clustering algorithm can be used to generate an instance segmentation map for each signed distance function corresponding to each detected intersection region or instance. In such embodiments, for each intersection region, pixels with distance values that are too high and too low can be filtered out using a predetermined threshold distance value— thereby reducing the computational resources required. The remaining pixels can be used to determine the boundaries of the relevant intersection region, and / or to determine each pixel associated with a given region of an intersection having a relevant classification type. Each pixel within the polygonal boundaries representing the intersection region can then be encoded with a segmentation pixel value delineating the corresponding intersection region. The post-processed outputs can be instance segmentation masks or instances for each classification type corresponding to the locations of the different classification type regions of the intersection.

[0029] In some embodiments, the 2D image space locations can be used to determine three-dimensional (3D) world space coordinates for the intersection regions to help a vehicle navigate in a world space environment. For example, sensor intrinsic and / or extrinsic parameters can be used to map the 2D image space locations to known 3D world space locations. However, in some embodiments, the DNN can be trained on 3D data to predict locations in 3D world space.

[0030] Reference is now made to FIG. 1 , FIG. 1is an example dataflow diagram illustrating an example process 100 for training a neural network to detect intersection zones or regions to assist a vehicle in navigating an intersection in accordance with some embodiments of the present disclosure. It should be appreciated that such and other arrangements described herein are presented by way of example only. At a high level, the process 100 can include one or more machine learning models 104 that receive one or more inputs, such as sensor data 102, and generate one or more outputs, such as one or more signed distance functions 108. In some examples, the sensor data 102 can be referred to as training data when used for training. Although the sensor data 102 is primarily discussed in terms of image data representative of images, this is not intended to be limiting and the sensor data 102 can include other types of sensor data for intersection region detection, such as LIDAR data, SONAR data, RADAR data, and / or the like—e.g., generated by one or more sensors of the vehicle 900 FIGS. 9A-9D

[0031] The process 100 can include generating and / or receiving sensor data 102 from one or more sensors. By way of non-limiting example, the sensor data 102 can be received from one or more sensors of a vehicle, such as the vehicle 900 of FIGS. 9A-9C and described herein. The sensor data 102 can be used by the vehicle and within the process 100 for detecting intersection regions to navigate an intersection in real-time or near real-time. The sensor data 102 can include, but is not limited to, sensor data 102 from any sensor of the vehicle, including, for example and with reference to FIGS. 9A-9C ​Global Navigation Satellite System (GNSS) sensor 958 (e.g., a Global Positioning System sensor), RADAR sensor 960, ultrasonic sensor 962, LIDAR sensor 964, Inertial Measurement Unit (IMU) sensor 966 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), microphone 976, stereo camera 968, wide-angle camera 970 (e.g., a fisheye camera), infrared camera 972, surround camera 974 (e.g., a 360-degree camera), long and / or medium range camera 978, speed sensor 944 (e.g., to measure a speed of vehicle 900), and / or other sensor types. As another example, sensor data 102 can include virtual (e.g., simulated or augmented) sensor data generated from any number of sensors of a virtual vehicle or other virtual object in a virtual (e.g., test) environment. In such examples, the virtual sensors can correspond to a virtual vehicle or other virtual object in a simulated environment (e.g., to test, train, and / or validate neural network performance), and the virtual sensor data can represent sensor data captured by the virtual sensors in the simulated or virtual environment. Thus, by using virtual sensor data, the machine learning models 104 described herein can be tested, trained, and / or validated using simulated or augmented data in a simulated environment, which can allow for testing of more extreme scenarios outside of real-world environments where such testing can be less safe.

[0032] In some embodiments, sensor data 102 can include image data representing images, image data representing video (e.g., a snapshot of a video), and / or sensor data representing a representation of a sensor’s sensory field (e.g., a depth map of a LIDAR sensor, a value map of an ultrasonic sensor, etc.). Where sensor data 102 includes image data, any type of image data format can be used, such as, but not limited to, compressed images such as Joint Photographic Experts Group (JPEG) or luminance / chrominance (YUV) formats, compressed images as frames sourced from a compressed video format (e.g., H.264 / Advanced Video Coding (AVC) or H.265 / High Efficiency Video Coding (HEVC)), raw images, such as sourced from a Red-Cyan-Blue (RCCB), Red-Cyan-Cyan (RCCC), or other type of imaging sensor, and / or other formats. Moreover, in some examples, sensor data 102 can be used within process 100 without any pre-processing (e.g., in raw or captured format), while in other examples, sensor data 102 can undergo pre-processing (e.g., noise balancing, demosaicing, scaling, cropping, enhancement, white balancing, tone curve adjustment, etc., such as using a sensor data pre-processor (not shown)). As used herein, sensor data 102 can refer to unprocessed sensor data, pre-processed sensor data, or a combination thereof.

[0033] The sensor data 102 for training can include raw images (e.g., captured by one or more image sensors), downsampled images, upsampled images, cropped or region of interest (ROI) images, otherwise enhanced images, and / or combinations thereof. The machine learning model 104 can be trained using the images (and / or other sensor data 102) along with corresponding ground truth data. The ground truth data can include annotations, labels, masks, maps, and the like. For example, in some embodiments, the ground truth data can include a ground truth (GT) signed distance function 116. The encoder 112 can encode the GT signed distance function 116 using the annotations 110. In some embodiments, the annotations 110 can include intersection regions 110A and classifications 110B for the intersection regions 110A.

[0034] Referring to the annotations 110, in some examples, the annotations 110 can be generated in a drawing program (e.g., a drafting program), a computer aided design (CAD) program, a labeling program, another type of program suitable for generating the annotations 110 and / or other GT signed distance functions 116, and / or can be hand-drawn. In any examples, the annotations 110 and / or GT signed distance functions 116 can be synthetically produced (e.g., generated from a computer model or rendering), actually produced (e.g., designed and produced from real-world data), machine automated (e.g., using feature analysis and learning to extract features from data, then generating labels), human annotated (e.g., a labeler or annotation specialist defining locations of labels), and / or combinations thereof (e.g., a human identifying a center or origin and dimensions of a region, a machine generating a polygon and / or labels for intersection regions).

[0035] The intersection regions 110A can include annotations or other label types corresponding to boundary shapes (e.g., polygons) delineating regions of interest for the intersection. In some examples, the intersection regions can be delineated by one or more polygons corresponding to a crosswalk region, an intersection entry region, an intersection exit region, an unclear region, a no-lane region, an intersection interior region, a partially visible region, a fully visible region, and the like—e.g., within a sensor data representation of the sensor data 102. The polygons can be generated as bounding boxes. The classifications 110B can be generated for each of the images (or other data representations) and / or each of the one or more polygons in the images represented by the sensor data 102 used to train the machine learning model 104. The number of classifications 110B can correspond to the number and / or types of features that the machine learning model 104 is trained to predict, or to the number and / or types of intersection regions in the corresponding images.

[0036] According to embodiments, the classifications 110B can correspond to classifications or labels corresponding to feature types or intersection region categories, such as, but not limited to, crosswalk region, intersection entry region, intersection exit region, unclear region, no-lane region, intersection interior region, partially visible region, fully visible region, and the like. In some examples, the classifications 110B can initially correspond to intersection interior region and / or intersection exterior region. The intersection interior region classification can refer to intersection regions 110A that include regions within the interior of an intersection where paths of vehicles traversing the intersection in various directions can cross. The intersection exterior region classification can refer to intersection regions 110A that include regions outside of the intersection interior region.

[0037] Each of the intersection regions 110A classified as intersection exterior regions can also be labeled with a classification 110B corresponding to an attribute corresponding to a feature type of an intersection exit region, including crosswalk region, intersection entry region, intersection exit region, unclear region, no-lane region, and the like. In particular, the intersection entry attribute can correspond to one or more intersection regions 110A where one or more vehicles are about to enter the respective intersection from various different directions. The intersection exit region can correspond to one or more intersection regions 110A where one or more vehicles that have recently exited the intersection in various directions can be located. It should be appreciated that information about the intersection exit region can be particularly important because the vehicle 900 itself must safely traverse the intersection exit region in order to safely traverse the intersection. Similarly, the crosswalk region can refer to intersection regions 110A that correspond to a crosswalk located outside of the intersection interior region. Regions classified as “no-lane region” can correspond to intersection regions 110A that do not allow vehicles to traverse, such as a bike lane, a sidewalk, and the like. The “unclear region” attribute can correspond to intersection regions where the direction of vehicle travel is not clear. Further, the intersection interior region and intersection exterior region categories of the classifications 110A can also include one of a fully visible region and / or a partially visible region attribute. In examples, where the classification 110B includes a fully visible region attribute or category label, the respective intersection region 110A can include, for example, a fully visible surface without any obstructions. Conversely, where the classification 110B includes a partially visible region attribute or category label, the respective intersection region 110A can include obstructions, such as occlusions, such that the driving surface in that region is only partially visible in the respective sensor data 102. The labeling schema described herein is for example purposes only, and additional and / or alternative category labels can be used without departing from the scope of the present disclosure.

[0038] As a non-limiting example, and with respect to FIGS. 2A-2B , FIGS. 2A-2BThe illustration shows example annotations applied to sensor data according to some embodiments of the present disclosure, used in ground reality generation to train machine learning models for detecting intersection areas. For example, FIG. 2A The diagram illustrates what can be used based on FIG. 1 The training process 100 generates example labels (e.g., corresponding to annotation 110) for images 200A of ground reality data. Intersection areas or regions in the images can be defined by intersection regions 110A (e.g., regions 204A, 204B, 206, 208, 210A, and 210B) and corresponding classifications (e.g., "intersection interior," "partially visible," "vehicle exit," "vehicle entrance," "partially visible," "pedestrian crossing," etc.). For example, intersection region 204A can be labeled using polygons and classified as having one or more attributes, such as "intersection entrance" and "partially visible." Similarly, intersection areas 204B, 206, 208, 210A, and 210B can be labeled using polygons, where intersection 204B can be classified as having one or more attributes, such as "vehicle entry" and "partially visible"; intersection area 206 can be classified as having one or more attributes, such as "pedestrian crossing" and "partially visible"; intersection area 208 can be classified as having one or more attributes, such as "intersection interior" and "partially visible"; intersection area 210A can be classified as having one or more attributes, such as "vehicle exit" and "partially visible"; and intersection area 210B can be classified as having one or more attributes, such as "vehicle exit" and "fully visible". In some examples, each intersection area belonging to a common category or classification can also be annotated with polygons of matching colors (or some other visual indication of semantic information). For example, the polygons for intersection areas 204A and 204B can have the same color and / or style because they are both classified as vehicle entry. Similarly, the polygons in intersection areas 210A and 210B can be annotated with the same color and / or style because they are both classified as vehicle exit categories. System 100 can know that these markings or annotation styles correspond to specific categories, and this information can be used to generate encoded ground reality data for training machine learning model 104.

[0039] Now for reference FIG. 2B , FIG. 2BAnother example of annotations applied to sensor data to train a machine learning model to detect intersection regions is illustrated in accordance with some embodiments of the present application. As depicted herein, intersection regions 222A-222C, 224A-224C, 226A-226B, 228A-228B, and 230 can be annotated with polygons and corresponding classifications (e.g., “inside intersection,” “partially visible,” “vehicle exit,” “vehicle entry,” “partially visible,” “crosswalk,” etc.). For example, intersection regions 222A, 222B, and 222C can be labeled with similar colored and / or styled polygons and classified as one or more of “vehicle entry” and “partially visible.” Similarly, intersection regions 224A, 224B, and 224C can be labeled with similar colored and / or styled polygons and classified as one or more of “crosswalk,” “fully visible,” and “partially visible.” Intersection regions 226A and 226B can be labeled with similar colored and / or styled polygons and classified as one or more of “no-lane,” “fully visible,” and partially visible. Intersection regions 228A and 228B can be labeled with similar colored and / or styled polygons and classified as one or more of “vehicle exit,” “fully visible,” and “partially visible.” Intersection region 230 can be labeled with a polygon and classified as one or more of “inside intersection” and “partially visible.”

[0040] For the same classification, annotations can have similar visual representations. As shown, intersection regions 222A, 222B, and 222C can be classified as vehicle exit regions. In this way, similar classifications of features of an image can be annotated in a similar manner. Further, it should be noted that classification 110B can be a compound noun. Different classification labels can be represented in FIG. 2B with solid lines, dashed lines, etc. to represent different classifications. Further, different classification labels can be nouns and / or compound nouns. This is not intended to be limiting and any classification naming convention can be used to illustrate differences in classification labels for features (e.g., intersection regions) in an image.

[0041] Referring again to FIG. 1 , encoder 112 can be configured to encode ground truth information corresponding to intersection regions using annotations 110. Annotations 110 can be encoded by encoder 112 to generate ground truth data, such as ground truth (GT) signed distance function 116. Further, because the spatial resolution of sensor data 102 applied to machine learning model 104 can be different (e.g., greater, less, etc.) than the output spatial resolution corresponding to predictions of machine learning model 104, ground truth information can be encoded in various formats.

[0042] For example, the annotations 110 can be encoded in the form of GT signed distance functions 116— such that each intersection region class (e.g., intersection region type) is encoded in a separate GT signed distance function. The intersection regions 110A and corresponding classifications 110B can be used by an encoder 112 to generate the GT signed distance functions 116, each GT signed distance function corresponding to an intersection region type corresponding to an intersection region determined from the annotations 110. In some examples, each pixel in the signed distance functions 116 can be encoded with a distance value corresponding to the pixel distance to the nearest boundary of the intersection region 110A to the intersection region type of the corresponding intersection region. In such examples, pixels inside the intersection region 110A (e.g., polygon) can also be encoded with positive distance values, and pixels outside the intersection region 110A can be encoded with negative distance values, or vice versa. Pixels forming the boundaries of the intersection region 110A can be encoded with zero values to generate the GT signed distance functions 116. Thus, the ground truth can provide a continuous value representation for each pixel in the image, allowing for efficient and accurate performance of up-sampling of results detection during deployment. Moreover, encoding each intersection region class as its own signed distance function can allow for accurate and precise recall and reinforcement of individual instances of intersection regions of each class.

[0043] In some examples, in addition to or instead of encoding the GT signed distance functions 116, the encoder 112 can use the labeled intersection regions 110A and corresponding classifications 110B to generate a one-hot encoded map for each intersection region type. In such examples, the encoder 112 can encode the intersection regions 110A using a binary representation, where pixels inside the intersection region boundary (e.g., polygon boundary) can be encoded with a value of 1, and pixels outside the intersection region boundary can be encoded with a value of 0. In some examples, the one-hot encoded map can include multiple heat maps, each heat map corresponding to a different type of classification 110B of an intersection region determined from the annotations 118. However, when using one-hot encoded maps, adjacent regions of the same intersection region type (e.g., polygons with shared boundaries) can not be individually identifiable, thus potentially reducing accuracy as compared to embodiments using GT signed distance functions 116. In this way, the intersection regions 110A can be encoded using GT signed distance functions 116 and / or require one-hot encoded maps with limited labeling, as the information can be determined using polygon annotations and semantic information.

[0044] As an example, reference is made to FIG. 3 , FIG. 3Illustrated are ground truth signed distance functions 310B, 320B, 330B, 340B, 350B respectively for encoding the intersection regions from annotations 310A, 320A, 330A, 340A, and 350A. Annotation image 310A includes intersection region 312A. In some examples, intersection region 312A can include a polygon representing the intersection region and corresponding signed distance function 310B can be used to encode intersection region 312A of a particular intersection region type. Intersection region 312A can be encoded in signed distance function 310B such that pixels within intersection region 312B are encoded with positive distance values, pixels outside intersection region 312B are encoded with negative distance values, and / or pixels on the boundary of intersection region 312B are encoded with zero values. The distance values for the pixels can correspond to the distance of each pixel to the nearest boundary of the intersection region or area (e.g., of the intersection classification type to which the signed distance function corresponds).

[0045] Similarly, annotation image 320A includes intersection regions 322A, 324A, 326A, and 328A of another intersection region type, as indicated by annotation 110. In some examples, intersection regions 322A, 324A, 326A, and 328A can include respective polygons representing the respective intersection regions. Respective signed distance function 320B can be used to encode intersection regions 322A, 324A, 326A, and 328A of the other intersection region type. Intersection regions 322A, 324A, 326A, and 328A can be encoded in signed distance function 320B such that pixels within each of intersection regions 322B, 324B, 326B, and 328B are encoded with positive distance values, pixels outside intersection regions 322A, 324A, 326A, and 328A are encoded with negative distance values, and pixels on the boundary of each of intersection regions 322A, 324A, 326A, and 328A are encoded with zero values. By encoding intersection regions in annotations 330A, 340A, and 350A respectively, signed distance functions 330B, 340B, and 350B can similarly be generated to represent their respective intersection region types.

[0046] Once the GT signed distance function 116 is generated for each instance of the sensor data 102 (e.g., for each image of the sensor data 102 that includes image data), the GT signed distance function 116 can be used to train the machine learning model 104. For example, the machine learning model 104 can generate a signed distance function 108 and can compare the signed distance function 108 to the GT signed distance function 116 corresponding to the respective instance of the sensor data 102 using a loss function 120. Accordingly, feedback from the loss function 120 can be used to update parameters (e.g., weights and biases) of the machine learning model 104 in view of the GT signed distance function 116 until the machine learning model 104 converges to an acceptable or desired accuracy. Using the process 100, the machine learning model 104 can be trained to accurately predict the signed distance function 108 (and / or related classification) from the sensor data 102 using the loss function 120 and the GT signed distance function 116. In some examples, different loss functions 120 can be used to train the machine learning model 104 to predict different signed distance functions 108 for each semantic class type (e.g., intersection region type). For example, a first loss function 120 can be used to compare the signed distance function 108 and the GT signed distance function 116 for a first semantic class type and a second loss function 120 can be used to compare the signed distance function 108 and the GT signed distance function 116 for a second semantic class type. Accordingly, in non-limiting embodiments, one or more output channels can be trained using a different loss function 120 than another output channel.

[0047] The machine learning model 104 can use the sensor data 102 to compute the signed distance function 108, which can ultimately be applied to a decoder or one or more other post-processing components (described in more detail herein at least with respect to FIG. 5 The machine learning model 104 can use the sensor data 102 to compute the signed distance function 108, which can ultimately be applied to a decoder or one or more other post-processing components (described in more detail herein at least with respect to FIG. 1 and FIG. 5 ) describe examples, but this is not intended to be limiting. For example, and without limitation, the machine learning model 104 can include any type of machine learning model, such as machine learning models that use linear regression, logistic regression, decision trees, support vector machines (SVM), Naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutional, recurrent, perceptron, long / short-term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolutional, generative adversarial, liquid state machine, etc.), region of interest detection algorithms, computer vision algorithms, and / or other types of machine learning models.

[0048] As an example, in the case where the machine learning model 104 includes a CNN, the machine learning model 104 can include any number of layers. One or more layers can include an input layer. The input layer can hold values associated with the sensor data 102 (e.g., before or after post-processing). For example, when the sensor data 102 is an image, the input layer can hold values representing the raw pixel values of the image as a tensor (e.g., width, height, and color channels (e.g., RGB), such as 32x32x3).

[0049] One or more layers can include a convolutional layer. The convolutional layer can compute the outputs of neurons connected to local regions in the input layer, each neuron computing their weights with the dot product between their weights and the small region in the input tensor they are connected to. The result of the convolutional layer can be another tensor, where one dimension is based on the number of filters applied (e.g., width, height, and number of filters, such as 32x32x12 if 12 is the number of filters).

[0050] One or more layers can include a rectified linear unit (ReLU) layer. For example, the ReLU layer can apply an element-wise activation function, such as max(0, x), with a threshold set to zero. The resulting tensor of the ReLU layer can be the same as the input tensor of the ReLU layer.

[0051] One or more layers can include a pooling layer. The pooling layer can perform a down-sampling operation along spatial dimensions (e.g., height and width), which can result in a smaller tensor than the input to the pooling layer (e.g., 16x16x12 from a 32x32x12 input tensor).

[0052] One or more layers can include one or more fully connected layers. Each neuron in the fully connected layer can be connected to every neuron in the preceding tensor. The fully connected layer can compute class scores, and the resulting tensor can be 1x1xclass number. In some examples, the CNN can include one or more fully connected layers such that the output of one or more layers of the CNN can be provided as input to one or more fully connected layers of the CNN. In some examples, one or more convolutional streams can be implemented by the machine learning model 104, and some or all of the convolutional streams can include respective fully connected layers.

[0053] In some non-limiting embodiments, the machine learning model 104 can include a series of convolutional layers and max-pooling layers to facilitate image feature extraction, followed by multi-scale dilated convolutions and up-sampling layers to facilitate global contextual feature extraction.

[0054] Although input layers, convolutional layers, pooling layers, ReLU layers, and fully connected layers are discussed herein with respect to the machine learning model 104, this is not intended to be limiting. For example, additional or alternative layers can be used in the machine learning model 104, such as normalization layers, SoftMax layers, and / or other layer types.

[0055] In embodiments in which the machine learning model 104 includes a CNN, different orders and / or numbers of layers of the CNN can be used according to embodiments. In other words, the order and number of layers of the machine learning model 104 are not limited to any one architecture.

[0056] Further, certain layers can include parameters (e.g., weights and / or biases), such as convolutional layers and fully connected layers, while others can not, such as ReLU layers and pooling layers. In some examples, the parameters can be learned by the machine learning model 104 during training. Further, certain layers can include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), such as convolutional layers, fully connected layers, and pooling layers, while others can not, such as ReLU layers. The parameters and hyperparameters are not limited and can differ according to embodiments.

[0057] Referring now to FIG. 4 , each block of the method 400 described herein includes a computational process that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 400 can also be embodied as computer-usable instructions stored on computer storage media. The method 400 can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. FIG. 1 The method 400 is described with respect to the process 100 of

[0058] FIG. 4 is a flowchart illustrating a method 400 for training a neural network to detect intersection regions, in accordance with some embodiments of the present disclosure. At block B402, the method 400 includes receiving image data representing an image depicting an intersection. For example, the sensor data 102 can be received, where the sensor data 102 includes the image data representing an image depicting an intersection.

[0059] At block B404, the method 400 includes generating annotations representing boundary shapes corresponding to the intersection regions and respective semantic class labels corresponding to a semantic class of each region. For example, the annotations 110 can be generated and / or received, where the annotations represent boundary shapes corresponding to the intersection regions 110A and respective classifications 110B of each of the intersection regions 110A.

[0060] At block B406, the method 400 includes computing a signed distance function for each semantic class type corresponding to the intersection, the signed distance function including a first signed value for a first pixel of the image inside the region, a second signed value for a second pixel of the image outside the region, and a third signed value for a third pixel of the image along the region boundary. For example, a GT signed distance function 116 can be computed for each intersection region type corresponding to the classification 110B of the intersection. The GT signed distance function 116 can include positive distance values for pixels inside the intersection region 110A, negative distance values for pixels outside the intersection region 110A, and zero values for pixels along the boundary of the intersection region 110A.

[0061] At block B408, the method 400 includes training a deep neural network (DNN) using the signed distance function as ground truth data. For example, the GT signed distance function 116 can be used as ground truth data to train the machine learning model 104.

[0062] Reference is now made to FIG. 5 , FIG. 5 is a dataflow diagram illustrating an example process 500 for detecting intersection regions in real-time or near real-time using output from sensors of a vehicle, according to some embodiments of the present disclosure. The sensor data 102 can be similar to at least with respect to the sensor data 102 described herein. During deployment, the sensor data 102 can be applied to the machine learning model 104 that has been trained according to the process 100, for example, to compute the signed distance functions 108 (and corresponding classifications) using the training sensor data 102. As such, the machine learning model 104 can output or regress one or more signed distance functions 108, as described in more detail at least with respect to the signed distance functions 108 in FIG. 1 During deployment, the sensor data 102 can be applied to the machine learning model 104 that has been trained according to the process 100, for example, to compute the signed distance functions 108 (and corresponding classifications) using the training sensor data 102. As such, the machine learning model 104 can output or regress one or more signed distance functions 108, as described in more detail at least with respect to the signed distance functions 108 in FIG. 1 During deployment, the sensor data 102 can be applied to the machine learning model 104 that has been trained according to the process 100, for example, to compute the signed distance functions 108 (and corresponding classifications) using the training sensor data 102. As such, the machine learning model 104 can output or regress one or more signed distance functions 108, as described in more detail at least with respect to the signed distance functions 108 in

[0063] The signed distance functions 108 can be applied to a post-processor 502 to generate instance segmentation masks 510. In some examples, the post-processor 502 can use one or more post-processing algorithms (e.g., temporal smoothing, upsampling, thresholding, clustering, mapping, etc.) to compute the instance segmentation masks 510. For example, the post-processor 502 can use the signed distance functions 108 to generate a representation of the intersection regions, which can in turn be used to navigate the intersection - e.g., by a control component 516 of the vehicle. The instance segmentation masks 510 can include the intersection regions, their classifications, and / or other information corresponding to the intersection (e.g., a polygon corresponding to the intersection region, which can be used to generate a visualization of the intersection for debugging, etc.).

[0064] The post-processor can perform smoothing 504, upsampling 506, and / or mapping 508 to generate instance segmentation masks 510. In some embodiments, the signed distance functions 108 can be subjected to smoothing 504 on a pixel-by-pixel basis to smooth spikes and fluctuations in the pixel values. In some examples, the smoothing 504 can include Gaussian smoothing of each signed distance function 108 output by the machine learning model 104. Spikes and fluctuations in the signed distance functions 108 can be smoothed by applying a smoothing algorithm to each signed distance function 108 to generate a respective smoothed signed distance function.

[0065] In some embodiments, one or more of the signed distance functions 108 and / or the smoothed signed distance functions can be subjected to upsampling 506. For example, the signed distance functions 108 can be generated at a first spatial resolution (e.g., a downsampled spatial resolution) that is different from a second spatial resolution corresponding to the input spatial resolution of the sensor data 102. Accordingly, the upsampling 506 can decode data from the first spatial resolution of the signed distance functions 108 and translate or convert the data to the second spatial resolution of the sensor data 102. In some examples, bilinear upsampling can be applied to each signed distance function 108 and / or each smoothed signed distance function to restore the original resolution of the sensor data 102 and generate a respective upsampled signed distance function. As a result, the processing speed of the machine learning model 104 can be increased - e.g., due to the processing of reduced spatial resolution data - while reserving computational resources for other tasks of the system. Moreover, by increasing the processing speed, the run time of the system can be reduced, thereby enabling deployment of the process 500 in real-time or near real-time within the system (e.g., the vehicle 900).

[0066] The signed distance function 108, the smoothed signed distance function, and / or the upsampled signed distance function can then be mapped (e.g., using a mapping 508, such as a mapping algorithm) to a polygonal region representing the intersection region based on the negative and positive values associated with each pixel in the respective signed distance function. The mapping 508 can include applying the signed distance function 108 to a thresholding and / or clustering algorithm. In some embodiments, a watershed clustering algorithm can be used as the mapping 508 to generate an instance segmentation map 510 for each detected intersection region corresponding to each signed distance function 108. In such embodiments, for each intersection region, pixels with distance values that are too high and too low can be filtered out using a predetermined threshold distance value, thereby reducing the computational resources required. The remaining pixels can be used to determine the boundaries of the relevant intersection region, and / or to determine each pixel associated with a given region of an intersection having a relevant classification type. Each pixel within the polygonal boundary representing the intersection region can then be encoded with a pixel value delineating the segmentation (e.g., class label or value) of the respective intersection region. The output of the post-processor 502 can be an instance segmentation mask 510 for each classification type or instance corresponding to the location (e.g., two-dimensional coordinates) of the different classification type regions of the intersection in the image space.

[0067] The instance segmentation mask 510 can then be applied to a coordinate converter 512 to convert the 2D locations in the image space to 3D world space coordinates for each detected intersection region in the instance segmentation mask 510. The 3D world space coordinates can be used by the vehicle 900 for path planning, control decisions, and / or other processes to safely and efficiently navigate through the intersection in the world space environment. Any known method of projecting 2D locations into 3D space can be used. In some examples, a planar model assumption can be used to project the locations into 3D space. In some other examples, LIDAR data for depth and / or distance estimation can be used for the projection, and / or future motion of the vehicle 900 can be used as the vehicle 900 traverses the road surface (e.g., to generate more accurate ground truth for training the network), such as to determine the slope or profile of the driving surface to more accurately project the 2D points into 3D space. In some embodiments, the machine learning model 104 can be trained on 3D data to directly predict the intersection region locations in 3D world space.

[0068] Therefore, the 3D world space coordinates determined from instance segmentation graph 510 can be used by control component 514 of vehicle 900 to perform one or more operations. As a non-limiting example, the perception layer of the autonomous driving software stack can update information about the environment based on intersection information, the world model manager can update the world model to reflect position, distance, attributes and / or other information about intersections and other vehicles, and / or the control layer can use this information to determine control when approaching, navigating through and / or leaving an intersection (e.g., based on waiting conditions, competition zones, priority vehicles, intersection size, distance to the intersection, etc.).

[0069] For reference FIG. 6 , FIG. 6 The illustration shows that, according to some embodiments of the present disclosure, a post-processor (e.g., post-processor 502) is converted into a corresponding instance segmentation mask (e.g., FIG. 5 Example signed distance function of instance segmentation mask 510 (e.g., FIG. 5 The signed distance function 108. The signed distance functions 610A, 620A, 630A, and 640A include visualizations of predicted intersection regions for each intersection region category. Each of the signed distance functions 610A, 620A, 630A, and 640A can be smoothed and upsampled (e.g., using smoothing 504, upsampling 506, etc.) to generate smoothed and upsampled signed distance functions 610B, 620B, 630B, and 640B, respectively. For example, pixels of the signed distance function 610A can undergo Gaussian smoothing to smooth out spikes and / or fluctuations in the signed distance function 610A to initially generate a smoothed signed distance function. The smoothed signed distance function can then undergo upsampling 506 to restore the original input image resolution of the sensor data (e.g., sensor data 102) to generate a corresponding smoothed and upsampled signed distance function 610B. Furthermore, mapping 508 can be used to map each smoothed and upsampled signed distance function 610B, 620B, 630B, and 640B to generate corresponding instance segmentation maps 610C, 620C, 630C, and 640C, respectively. For example, the smoothed and upsampled signed distance function 610B can undergo a watershed clustering algorithm to generate an instance segmentation mask 612C, which can then be overlaid on the original image, as can be seen in the resulting instance segmentation map 610C. Thus, the intersection regions detected in the image can be represented by a set of polygons used for classification of each intersection region.

[0070] refer to FIGS. 7A-7B , FIGS. 7A-7B The illustration shows an example intersection area predicted in 3D space based on a signed distance function calculated by a neural network, according to some embodiments of the present disclosure. FIG. 7AA visualization of a prediction 700A of the intersection regions 720, 712, 714, 716, 718, 720, and 722 is illustrated, which can be used to determine intersection crossing priorities related to other vehicles in the intersection. Different intersection region classifications can be represented with solid lines, dashed lines, different colors, and the like to represent different classifications. This is not intended to be limiting, and any visual convention of classification can be used to illustrate differences in classification of features (e.g., intersection regions) in an image for visualization, debugging, or other purposes. For example, it can be understood that the intersection regions 712, 714, and 716 belong to a single intersection region classification. Similarly, the intersection regions 720 and 722 can belong to another intersection region classification.

[0071] FIG. 7B A further prediction 700B of the intersection regions 740, 742, 744, 746, 748, 750, and 720 is illustrated, which can be used to determine intersection crossing priorities related to other vehicles by a control component (e.g., the control component 514 of the vehicle 900). Different intersection region classifications can be represented with solid lines, dashed lines, different colors, and the like to represent different classifications. This is not intended to be limiting, and any visual convention of classification can be used to illustrate differences in classification of features (e.g., intersection regions) in an image for visualization, debugging, or other purposes. For example, it can be understood that the intersection regions 742 and 744 belong to a first intersection region classification. Similarly, the intersection regions 750 and 752 can belong to a second intersection region classification, and so on. In this way, intersection regions can be readily detected as belonging to respective classifications.

[0072] Reference is now made to FIG. 8 The various blocks of the method 800 described herein include computing processes that can be performed using any combination of hardware, firmware, and / or software. For instance, various functions can be carried out by a processor executing instructions stored in memory. The method 800 can also be embodied as computer-usable instructions stored on computer storage media. The method 800 can be provided by a standalone application, a service or hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name a few. FIG. 5 The method 800 is described with respect to the process 500 of FIG. 6 by way of example. However, these methods can additionally or alternatively be performed by any one system or any combination of systems, including but not limited to those described herein.

[0073] FIG. 8 is a flowchart illustrating a method 800 for detecting intersection regions, in accordance with some embodiments of the present disclosure. At block B802, the method 800 includes applying image data representing an intersection in a field of view of an image sensor of a vehicle to a neural network. For example, the sensor data 102 can be applied to the machine learning model 104. The sensor data 102 can represent an intersection in a field of view of an image sensor of the vehicle 900.

[0074] At block B804, the method 800 includes computing, using the neural network and based at least in part on the image data, first data representing a first signed distance function corresponding to the first intersection region class and a second signed distance function corresponding to the second intersection region class. For example, the machine learning model 104 can output signed distance functions 108 for each of the plurality of intersection region classes based on the sensor data 102.

[0075] At block B806, the method 800 includes applying a clustering algorithm to the first signed distance function and the second signed distance function. For example, the mapping 508 can be performed on the signed distance functions 108, including clustering.

[0076] At block B808, the method 800 includes computing, based at least in part on the clustering algorithm, a first instance segmentation map corresponding to the first intersection region class and a second instance segmentation map corresponding to the second intersection region class. For example, the instance segmentation maps 510 can be computed based on the mapping 508 corresponding to each intersection region class detected in the signed distance functions 108.

[0077] At block B810, the method 800 includes determining, based at least in part on the first instance segmentation map and the second instance segmentation map, a world space position corresponding to an instance of the first intersection region class and an instance of the second intersection region class. For example, the 3D world space position corresponding to an instance of an intersection region class can be determined by the coordinate converter 512 based on the instance segmentation mask 510.

[0078] At block B810, the method 800 includes transmitting second data representing the world space position to a planning component of the vehicle. For example, the 3D world space position of an intersection region can be transmitted to the control component 514 of the vehicle 900 for further processing.

[0079] Example autonomous vehicle

[0080] FIG. 9Ais an illustration of an example autonomous vehicle 900 according to some embodiments of the present disclosure. The autonomous vehicle 900 (also referred to herein as “vehicle 900”) can include, but is not limited to, a passenger vehicle such as a car, truck, bus, ambulance, shuttle, electric or motorized bicycle, motorcycle, fire truck, police car, ambulance, boat, engineering vehicle, underwater vehicle, drone, and / or other type of vehicle (e.g., unmanned and / or capable of accommodating one or more passengers). Autonomous vehicles are often described in terms of levels of automation according to the Society of Automotive Engineers (SAE) “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806 published June 15, 2018, Standard No. J3016-201609 published September 30, 2016, and prior and future versions of this standard). The vehicle 900 can be capable of functioning according to one or more of Levels 3-5 of autonomous driving. For example, depending on the embodiment, the vehicle 900 can have conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0081] The vehicle 900 can include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of a vehicle. The vehicle 900 can include a propulsion system 950 such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. The propulsion system 950 can be connected to a drivetrain of the vehicle 900 that can include a transmission in order to effect propulsion of the vehicle 900. The propulsion system 950 can be controlled in response to receiving a signal from a throttle / accelerator 952.

[0082] A steering system 954, which can include a steering wheel, can be used to steer the vehicle 900 (e.g., along a desired path or route) while the propulsion system 950 is operating (e.g., while the vehicle is in motion). The steering system 954 can receive a signal from a steering actuator 956. For full automation (Level 5) functionality, the steering wheel can be optional.

[0083] A braking sensor system 946 can be used to operate vehicle brakes in response to receiving a signal from a brake actuator 948 and / or a brake sensor.

[0084] The vehicle 900 can include one or more system-on-chips (SoCs) 904 FIG. 9C) and / or one or more controllers 936 of one or more GPUs can provide signals (e.g., representative of commands) to one or more components and / or systems of the vehicle 900. For example, the one or more controllers can send signals to operate vehicle brakes via one or more brake actuators 948, to operate a steering system 954 via one or more steering actuators 956, to operate a propulsion system 950 via one or more throttle / accelerator 952. The one or more controllers 936 can include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representative of commands) to enable autonomous driving and / or to assist a human driver in driving the vehicle 900. The one or more controllers 936 can include a first controller 936 for autonomous driving functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functions (e.g., computer vision), a fourth controller 936 for infotainment functions, a fifth controller 936 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 936 can handle two or more of the above functions, two or more controllers 936 can handle a single function, and / or any combination thereof.

[0085] The one or more controllers 936 can provide signals for controlling one or more components and / or systems of the vehicle 900 in response to sensor data (e.g., sensor inputs) received from one or more sensors. The sensor data can be received from, for example and without limitation, a global navigation satellite system sensor 958 (e.g., a global positioning system sensor), a RADAR sensor 960, an ultrasonic sensor 962, a LIDAR sensor 964, an inertial measurement unit (IMU) sensor 966 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 996, a stereo camera 968, a wide-angle camera 970 (e.g., a fisheye camera), an infrared camera 972, a surround camera 974 (e.g., a 360-degree camera), a long-range and / or mid-range camera 998, a speed sensor 944 (e.g., to measure a speed of the vehicle 900), a vibration sensor 942, a steering sensor 940, a brake sensor (e.g., as part of a brake sensor system 946), and / or other sensor types.

[0086] One or more of the controllers 936 can receive input (e.g., represented by input data) from an instrument cluster 932 of the vehicle 900 and provide output (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934, an audible annunciator, a speaker, and / or via other components of the vehicle 900. These outputs can include, for example and without limitation, vehicle speed, velocity, time, map data (e.g., represented by a map display), and / or other information. The one or more controllers 936 can also receive input from and provide output to other components of the vehicle 900, such as the one or more processors 910, the one or more GPUs 920, the one or more memory devices 930, the one or more sensors 940, the one or more actuators 950, and / or other components of the vehicle 900. FIG. 9Cinformation such as information about objects and object states as perceived by the controller 936, and so on. For example, the HMI display 934 can display information about the presence of one or more objects (e.g., a street sign, a warning sign, a traffic light change, and so on) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., change lanes now, exit 34B in two miles, and so on).

[0087] The vehicle 900 further includes a network interface 924 that can communicate over one or more networks using one or more wireless antennas 926 and / or modems. For example, the network interface 924 can be capable of communicating over LTE, WCDMA, UMTS, GSM, CDMA2000, and so on. The one or more wireless antennas 926 can also enable communication between objects (e.g., vehicles, mobile devices, and so on) in the implementation environment using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, and so on and / or one or more low power wide area networks (LPWANs) such as LoRaWAN, SigFox, and so on.

[0088] FIG. 9B FIG. 1 illustrates an example autonomous vehicle 900 for use in accordance with some embodiments of the present disclosure. FIG. 9A FIG. 2 illustrates example camera positions and fields of view of the example autonomous vehicle 900 of FIG. 1. The cameras and respective fields of view are one example embodiment and are not intended to be limiting. For example, additional and / or alternative cameras can be included and / or these cameras can be located at different positions on the vehicle 900.

[0089] Camera types for the cameras can include, but are not limited to, digital cameras that can be suitable for use with components and / or systems of the vehicle 900. The cameras can operate at Automotive Safety Integrity Level (ASIL) B and / or at another ASIL. The camera types can have any image capture rate, such as 60 frames per second (fps), 920 fps, 240 fps, and so on, depending on the embodiment. The cameras can be capable of using a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, a color filter array can include a red clear (RCCC) color filter array, a red clear blue (RCCB) color filter array, a red blue green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras such as cameras with a

[0090] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more of the cameras (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0091] One or more of the cameras can be mounted in mounting assemblies such as custom designed (3-D printed) assemblies to cut off stray light and reflections from within the car that can interfere with the image data capture capabilities of the cameras (e.g., reflections from the dashboard reflected in the windshield mirror). With regard to wing mirror mounting assemblies, the wing mirror assemblies can be custom 3-D printed such that the camera mounting plates match the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0092] Cameras with fields of view that include portions of the environment in front of the vehicle 900 (e.g., front-facing cameras) can be used for surround view to help identify the forward path and obstacles, and to assist in providing information critical to generating an occupancy grid and / or determining a preferred vehicle path with the help of one or more controllers 936 and / or control SoCs. Front-facing cameras can be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. Front-facing cameras can also be used for ADAS functions and systems including lane departure warning ("LDW"), adaptive cruise control ("ACC"), and / or other functions such as traffic sign recognition.

[0093] A wide variety of cameras can be used in a front-facing configuration, including, for example, monocular camera platforms including CMOS (complementary metal-oxide semiconductor) color imagers. Another example can be a wide-angle camera 970, which can be used to perceive objects (e.g., pedestrians, intersection traffic, or bicycles) entering the field of view from the periphery. Although FIG. 9B Although only one wide-angle camera is illustrated in FIG. 9, there can be any number of wide-angle cameras 970 on the vehicle 900. In addition, long-range cameras 998 (e.g., pairs of stereo long-range cameras) can be used for depth-based object detection, especially for objects for which a neural network has not been trained. Long-range cameras 998 can also be used for object detection and classification and basic object tracking.

[0094] One or more stereo cameras 968 can also be included in the front-facing configuration. Stereo cameras 968 can include an integrated control unit that includes a scalable processing unit that can provide a multicore microprocessor with integrated CAN or Ethernet interface and a field programmable logic (FPGA) on a single chip. Such a unit can be used to generate a 3-D map of the vehicle's environment, including distance estimates for all points in the image. Alternative stereo cameras 968 can include a compact stereo vision sensor that can include two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the vehicle to a target object and use the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 968 can be used in addition to or instead of those described herein.

[0095] Cameras with fields of view that include portions of the environment to the side of vehicle 900 (e.g., side-view cameras) can be used for surround view, providing information used to create and update the occupancy grid and to generate side impact collision warnings. For example, surround cameras 974 (e.g., four surround cameras 974 as shown in FIG. 15) can be placed on vehicle 900. Surround cameras 974 can include wide-view cameras 970, fisheye cameras, 360-degree cameras, and / or the like. In one example, four fisheye cameras can be placed on the front, back, and sides of the vehicle. In an alternative arrangement, a vehicle can use three surround cameras 974 (e.g., left, right, and back) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera. FIG. 9B

[0096] Cameras with fields of view that include portions of the environment to the rear of vehicle 900 (e.g., rear-view cameras) can be used for assist parking, surround view, rear collision warnings, and to create and update the occupancy grid. A wide variety of cameras can be used, including but not limited to cameras that are also suitable as front-facing cameras (e.g., long- and / or mid-range cameras 998, stereo cameras 968, infrared cameras 972, etc.) as described herein.

[0097] FIG. 9C For use in a vehicle according to some embodiments of the present disclosure FIG. 9A ​FIG. 1 is a block diagram of an example system architecture of an example autonomous vehicle 900. It should be understood that this arrangement and other arrangements described herein are set forth merely as examples. Arrangements other than those shown and other elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) can be used in addition to or instead of those shown, and some elements can be omitted entirely. Further, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in conjunction with other components, and in any suitable combinations and locations. Various functions described herein as being performed by an entity can be implemented in hardware, firmware, and / or software. For instance, various functions can be implemented by a processor executing instructions stored in memory.

[0098] FIG. 9C Each of the components, features, and systems of vehicle 900 are illustrated as being connected via bus 902. Bus 902 can include a controller area network (CAN) data interface (alternatively referred to herein as a "CAN bus"). The CAN can be a network within vehicle 900 that is used to assist in controlling various features and functions of vehicle 900, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. The CAN bus can be configured to have tens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, ground speed, revolutions per minute (RPM) of the engine, button positions, and / or other vehicle status indicators. The CAN bus can be ASIL B compliant.

[0099] Although bus 902 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet can be used in addition to or instead of a CAN bus. Further, although bus 902 is represented with a single line, this is not intended to be limiting. For example, any number of buses 902 can be present, which can include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses that use different protocols. In some examples, two or more buses 902 can be used to perform different functions, and / or can be used for redundancy. For example, a first bus 902 can be used for collision avoidance functions, and a second bus 902 can be used for drive control. In any example, each bus 902 can communicate with any component of vehicle 900, and two or more buses 902 can communicate with the same components. In some examples, each SoC 904, each controller 936, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from sensors of vehicle 900), and can be connected to a common bus, such as a CAN bus.

[0100] The vehicle 900 can include one or more controllers 936, such as those described herein with respect to FIG. 9A The controllers 936 can be used for a wide variety of functions. The controllers 936 can be coupled to any of the other distinct components and systems of the vehicle 900, and can be used for control of the vehicle 900, artificial intelligence of the vehicle 900, infotainment for the vehicle 900, and / or the like.

[0101] The vehicle 900 can include one or more system on chips (SoCs) 904. The SoCs 904 can include CPUs 906, GPUs 908, processors 910, caches 912, accelerators 914, data stores 916, and / or other components and features not illustrated. The SoCs 904 can be used to control the vehicle 900 in a wide variety of platforms and systems. For example, one or more SoCs 904 can be used in systems, such as systems of the vehicle 900, in conjunction with HD maps 922 that can obtain map refreshes and / or updates from one or more servers (such as the one or more servers 978) via a network interface 924. FIG. 9D

[0102] The CPU 906 can include a CPU cluster or CPU complex (alternatively referred to herein as a “CCPLEX”). The CPU 906 can include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 906 can include eight cores in a coherent multi-processor configuration. In some embodiments, the CPU 906 can include four dual-core clusters with each cluster having a dedicated L2 cache (such as a 2 MB L2 cache). The CPU 906 (e.g., the CCPLEX) can be configured to support simultaneous cluster operation such that any combination of clusters of the CPU 906 can be active at any given time.

[0103] The CPU 906 can implement power management capabilities including one or more of the following features: individual hardware blocks can be automatically clock-gated when idle to save dynamic power; each core clock can be gated when the core is not actively executing instructions due to execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated. The CPU 906 can further implement an enhanced algorithm for managing power states in which the allowed power states and the desired wake-up time are specified, and the hardware / microcode determines the best power state for the core, cluster, and CCPLEX to enter. The processing core can support a simplified power state entry sequence in software, with the work offloaded to microcode. ​

[0104] GPU 908 can include an integrated GPU (alternatively referred to herein as an “iGPU”). GPU 908 can be programmable and efficient for parallel workloads. In some examples, GPU 908 can use an enhanced tensor instruction set. GPU 908 can include one or more streaming microprocessors, where each streaming microprocessor can include an LI cache (e.g., an LI cache having at least 96 KB of storage capacity), and two or more of the streaming microprocessors can share an L2 cache (e.g., an L2 cache having 512 KB of storage capacity). In some embodiments, GPU 908 can include at least eight streaming microprocessors. GPU 908 can use a compute application programming interface (API). In addition, GPU 908 can use one or more parallel computing platforms and / or programming models (e.g., NVIDIA’s CUDA).

[0105] In the case of automotive and embedded uses, GPU 908 can be power-optimized for best performance. For example, GPU 908 can be fabricated on a fin field-effect transistor (FinFET). However, this is not intended to be limiting, and GPU 908 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a thread warp scheduler, a dispatch unit, and / or a 64 KB register file. In addition, the streaming microprocessor can include independent parallel integer and floating point data paths to exploit the mix of computation and addressing computation to provide efficient execution of workloads. The streaming microprocessor can include independent thread scheduling capabilities to allow for more fine-grained synchronization and cooperation between parallel threads. The streaming microprocessor can include a combined LI data cache and shared memory unit to improve performance while simplifying programming.

[0106] GPU 908 can include a high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem that provides approximately 900 GB / s of peak memory bandwidth in some examples. In some examples, in addition to or alternatively from HBM memory, a synchronous graphics random access memory (SGRAM) can be used, such as a fifth generation graphics double data rate synchronous random access memory (GDDR5).

[0107] GPU 908 can include a unified memory technology that includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, improving efficiency of memory ranges shared between processors. In some examples, address translation services (ATS) support can be used to allow GPU 908 to access CPU 906 page tables directly. In such examples, when a GPU 908 memory management unit (MMU) experiences a miss, an address translation request can be transmitted to CPU 906. In response, CPU 906 can look up a virtual-to-physical mapping for the address in its page tables and transmit the translation back to GPU 908. In this way, the unified memory technology can allow a single unified virtual address space for memory of both CPU 906 and GPU 908, simplifying GPU 908 programming and porting applications to GPU 908.

[0108] Further, GPU 908 can include access counters that can track how frequently GPU 908 accesses other processors’ memory. The access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses these pages most frequently.

[0109] SoC 904 can include any number of caches 912, including those described herein. For example, caches 912 can include an L3 cache available to both CPU 906 and GPU 908 (e.g., connected to both CPU 906 and GPU 908). Caches 912 can include a write-back cache that can track the state of a line, for example, by using a cache coherency protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache can include 4 MB or more, although smaller cache sizes can also be used.

[0110] SoC 904 can include an arithmetic logic unit (ALU) that can be used to perform processing for any of a variety of tasks or operations with respect to vehicle 900— e.g., processing a DNN. Further, SoC 904 can include a floating point unit (FPU)— or other mathematical co-processor or digital co-processor type— for performing mathematical operations within the system. For example, SoC 104 can include one or more FPUs integrated as execution units within CPU 906 and / or GPU 908.

[0111] The SoC 904 can include one or more accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC 904 can include a hardware acceleration cluster that can include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to supplement the GPU 908 and offload some of the tasks of the GPU 908 (e.g., freeing up more cycles of the GPU 908 for performing other tasks). As one example, the accelerators 914 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. As used herein, the term “CNN” can include all types of CNNs, including region-based or region with convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0112] The accelerators 914 (e.g., the hardware acceleration cluster) can include a deep learning accelerator (DLA). The DLA can include one or more tensor processing units (TPUs) that can be configured to provide an additional 100 billion operations per second for deep learning applications and inferencing. The TPUs can be accelerators that are configured to perform and optimized for performing image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA can be further optimized for a specific set of neural network types and floating point operations and inferencing. The DLA can be designed to provide higher performance per mm than general purpose GPUs and far beyond the performance of CPUs. The TPUs can perform several functions, including single instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, for example, and post-processor functions.

[0113] The DLA can perform neural networks, especially CNNs, on processed or unprocessed data for any of a wide variety of functions, such as and not limited to: CNNs for object recognition and detection using data from a camera sensor; CNNs for distance estimation using data from a camera sensor; CNNs for emergency vehicle detection and identification and detection using data from a microphone; CNNs for face recognition and vehicle owner identification using data from a camera sensor; and / or CNNs for safety and / or safety related events.

[0114] The DLA can perform any of the functions of the GPU 908, and by using an inferencing accelerator, the designer can target the DLA or the GPU 908 for any function, for example. For example, the designer can focus the processing and floating point operations of the CNNs on the DLA and leave other functions to the GPU 908 and / or other accelerators 914.

[0115] Accelerator 914 (e.g., hardware acceleration cluster) can include a programmable vision accelerator (PVA), which can be alternatively referred to herein as a computer vision accelerator. The PVA can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA can include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0116] The RISC cores can interact with image sensors (e.g., image sensors of any of the cameras described herein), image signal processors, and / or the like. Each of the RISC cores can include any number of memories. Depending on the embodiment, the RISC cores can use any of several protocols. In some examples, the RISC cores can execute a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or a tightly coupled RAM.

[0117] The DMA can enable components of the PVA to access system memory independently of the CPU 906. The DMA can support any number of features to provide optimization to the PVA, including, but not limited to, supporting multi-dimensional addressing and / or circular addressing. In some examples, the DMA can support addressing up to six or more dimensions, which can include block width, block height, block depth, horizontal block stride, vertical block stride, and / or depth stride.

[0118] The vector processors can be programmable processors that can be designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, the PVA can include a PVA core and two vector processing subsystem partitions. The PVA core can include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystems can operate as the main processing engines of the PVA and can include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core can include a digital signal processor, such as, for example, a single instruction multiple data (SIMD), very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and rate.

[0119] Each of the vector processors can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each of the vector processors can be configured to execute independently of the other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelization. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms on the same image simultaneously, or even different algorithms on a sequence of images or portions of an image. Any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each of those PVAs, among other things. Furthermore, the PVAs can include additional error correction code (ECC) memory to enhance overall system security.

[0120] The accelerator 914 (e.g., hardware acceleration cluster) can include an on-chip computer vision network and SRAM to provide high bandwidth, low latency SRAM for the accelerator 914. In some examples, the on-chip memory can include at least 4 MB of SRAM composed of, for example and without limitation, eight field-programmable memory blocks, which can be accessed by both the PVA and the DLA. Each pair of memory blocks can include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory can be used. The PVA and the DLA can access the memory via a backbone that provides high-speed memory access to the PVA and the DLA. The backbone can include an on-chip computer vision network that interconnects the PVA and the DLA to the memory, for example using an APB.

[0121] The on-chip computer vision network can include an interface that determines that both the PVA and the DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst communications for continuous data transmission. This type of interface can comply with ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0122] In some examples, the SoC 904 can include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator can be used to quickly and efficiently determine locations and extents of objects (e.g., within a world model) in order to generate real-time visualizations simulations for RADAR signal interpretation, for sound propagation synthesis and / or analysis, for SONAR system simulation, for general wave propagation simulation, for comparison with LIDAR data for purposes of localization and / or other functions, and / or for other uses. In some embodiments, one or more tree traversal units (TTUs) can be used to perform one or more ray tracing related operations.

[0123] The accelerator 914 (e.g., hardware accelerator cluster) has a wide range of autonomous driving uses. The PVA can be a programmable vision accelerator that can be used for key processing stages in ADAS and autonomous vehicles. The capabilities of the PVA are a good match for algorithm domains that require predictable processing, low power, and low latency. In other words, the PVA performs well on semi-dense or dense regular computations, and even on small data sets that require predictable runtimes with low latency and low power. Thus, in the context of a platform for autonomous vehicles, the PVA is designed to run classical computer vision algorithms because they are effective at object detection and integer math operations.

[0124] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. In some examples, a semi-global matching based algorithm can be used, although this is not intended to be limiting. Many applications for level 3-5 autonomous driving require instant motion estimation / stereo matching (e.g., structure from motion, pedestrian recognition, lane detection, etc.). The PVA can perform computer stereo vision functions on input from two monocular cameras.

[0125] In some examples, the PVA can be used to perform dense optical flow. Raw RADAR data is processed according to a process (e.g., using a 4D fast Fourier transform) to provide processed RADAR. In other examples, the PVA is used for time-of-flight depth processing, such as by processing raw time-of-flight data to provide processed time-of-flight data.

[0126] The DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence metric for each object detection. Such a confidence value can be interpreted as a probability, or as providing a relative "weight" for each detection compared to other detections. The confidence value enables the system to make further decisions about which detections should be considered true positive detections rather than false positive detections. For example, the system can set a threshold for confidence, and only consider detections that exceed the threshold to be true positive detections. In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically perform an emergency brake, which is obviously undesirable. Thus, only the most confident detections should be considered a trigger for AEB. The DLA can run a neural network for regression of a confidence value. The neural network can take as its input at least some subset of parameters, such as a bounding box dimension, a ground plane estimate obtained (e.g., from another subsystem), an inertial measurement unit (IMU) sensor 966 output related to vehicle 900 orientation, distance, a 3D position estimate of an object obtained from the neural network and / or other sensors (e.g., LIDAR sensor 964 or RADAR sensor 960), etc.

[0127] SoC 904 can include one or more data stores 916 (e.g., memory). Data stores 916 can be on-chip memory of SoC 904, which can store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and safety, data stores 916 can be large enough in capacity to store multiple instances of a neural network. Data stores 912 can include L2 or L3 cache 912. References to data stores 916 can include references to memory associated with PVAs, DLAs, and / or other accelerators 914 as described herein.

[0128] The SoC 904 can include one or more processors 910 (e.g., embedded processors). The processors 910 can include a boot and power management processor, which can be a specialized processor and subsystem for handling boot power and management functions and related security implementations. The boot and power management processor can be part of the SoC 904 boot sequence and can provide run-time power management services. The boot power and management processor can provide clock and voltage programming, auxiliary system low power state transitions, SoC 904 thermal and temperature sensor management, and / or SoC 904 power state management. Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC 904 can use the ring oscillator to detect the temperature of the CPU 906, GPU 908, and / or accelerator 914. If it is determined that the temperature exceeds a threshold, the boot and power management processor can enter a temperature fault routine and place the SoC 904 in a lower power state and / or place the vehicle 900 in a driver safe park mode (e.g., safely park the vehicle 900).

[0129] The processors 910 can further include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio over multiple interfaces and a range of widely flexible audio I / O interfaces. In some examples, the audio processing engine is a specialized processor core with a digital signal processor with dedicated RAM.

[0130] The processors 910 can further include an always-on processor engine that can provide the necessary hardware features to support low power sensor management and wake-up use cases. The always-on processor engine can include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0131] The processors 910 can further include a security cluster engine that includes a specialized processor subsystem that handles security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In a secure mode, the two or more cores can operate in a lockstep mode and act as a single core with comparison logic that detects any differences between their operations.

[0132] The processors 910 can further include a real-time camera engine that can include a specialized processor subsystem for handling real-time camera management.

[0133] The processor 910 can further include a high dynamic range signal processor, which can include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0134] The processor 910 can include a video image compositor, which can be a processing block (e.g., implemented on a microprocessor), that implements video post-processing functions needed by the video playback application to produce the final image for the player window. The video image compositor can perform lens distortion correction on the wide-angle camera 970, surround camera 974, and / or on the cab-in monitor camera sensors. The cab-in monitor camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to recognize cab-in events and respond accordingly. The cab-in system can perform lip reading to activate mobile phone services and place a call, dictate an email, change the vehicle destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode, and are disabled otherwise.

[0135] The video image compositor can include enhanced temporal noise reduction for spatial and temporal noise reduction. For example, where motion is present in the video, the noise reduction appropriately weights the spatial information, reducing the weight of information provided by neighboring frames. Where the image or portions of the image do not include motion, the temporal noise reduction performed by the video image compositor can use information from previous images to reduce noise in the current image.

[0136] The video image compositor can also be configured to perform stereo correction on input stereo lens frames. The video image compositor can further be used for user interface composition when the operating system desktop is in use and the GPU 908 does not need to continuously render new surfaces. Even when the GPU 908 is powered on and active, doing 3D rendering, the video image compositor can be used to offload the GPU 908 to improve performance and responsiveness.

[0137] The SoC 904 can further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and input from cameras, a high-speed interface, and / or a video input block that can be used for camera and related pixel input functions. The SoC 904 can further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a particular role.

[0138] The SoC 904 can further include a wide range of peripheral device interfaces to enable communication with peripherals, audio codecs, power management, and / or other devices. The SoC 904 can be used to process data from cameras (connected over Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LIDAR sensor 964, RADAR sensor 960, etc. that can be connected over Ethernet), data from the bus 902 (e.g., speed of the vehicle 900, steering wheel position, etc.), data from GNSS sensor 958 (connected over Ethernet or CAN bus). The SoC 904 can further include a dedicated high-performance mass storage controller, which can include their own DMA engine, and which can be used to free the CPU 906 from routine data management tasks.

[0139] The SoC 904 can be an end-to-end platform with a flexible architecture that spans automation levels 3-5, providing an integrated functional safety architecture for a platform that leverages and efficiently uses computer vision and ADAS technology to achieve diversity and redundancy, along with deep learning tools. The SoC 904 can be faster, more reliable, and even more energy and space efficient than conventional systems. For example, the accelerator 914, when combined with the CPU 906, GPU 908, and data storage 916, can provide a fast and efficient platform for level 3-5 autonomous vehicles.

[0140] The technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on CPUs that can be configured using high-level programming languages such as the C programming language to perform a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to, for example, execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for on-board ADAS applications and a requirement for practical level 3-5 autonomous vehicles.

[0141] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technology described herein allows multiple neural networks to be executed simultaneously and / or sequentially, and the results to be combined together to achieve level 3-5 autonomous driving functionality. For example, a CNN executed on a DLA or dGPU (e.g., GPU 920) can include text and word recognition, allowing a supercomputer to read and understand traffic signs, including signs for which a neural network has not been specifically trained. The DLA can further include a neural network that is able to recognize, interpret, and provide a semantic understanding of the sign, and pass that semantic understanding to a path planning module running on the CPU complex.

[0142] As another example, multiple neural networks can be run simultaneously as required for level 3, 4, or 5 driving. For example, a warning sign consisting of the words "Flashing lights indicate icy conditions" along with a light can be interpreted by several neural networks independently or collectively. The sign itself can be recognized by a first deployed neural network (e.g., a trained neural network) as a traffic sign, the text "Flashing lights indicate icy conditions" can be interpreted by a second deployed neural network that informs the vehicle's path planning software (preferably executing on the CPU complex) that icy conditions exist when flashing lights are detected. The flashing lights can be recognized by operating a third deployed neural network over multiple frames that informs the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks can be run simultaneously, for example, within the DLA and / or on the GPU 908.

[0143] In some examples, a CNN for face recognition and owner recognition can use data from the camera sensor to recognize the presence of an authorized driver and / or owner of the vehicle 900. A processing engine always on the sensor can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in a safe mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 904 provides security against theft and / or carjacking.

[0144] In another example, a CNN for emergency vehicle detection and recognition can use data from the microphone 996 to detect and recognize emergency vehicle sirens. In contrast to conventional systems that detect sirens using a general classifier and manually extract features, the SoC 904 uses a CNN to classify ambient and urban sounds as well as to classify visual data. In a preferred embodiment, a CNN running on the DLA is trained to recognize the relative closing speed of an emergency vehicle (e.g., by using the Doppler effect). The CNN can also be trained to recognize emergency vehicles specific to the local area in which the vehicle is operating as recognized by the GNSS sensor 958. Thus, for example, when operating in Europe, the CNN will seek to detect European sirens, and when in the United States, the CNN will seek to recognize sirens that are only North American. Once an emergency vehicle is detected, a control program can be used to execute an emergency vehicle safety routine, slow the vehicle down, pull over to the side of the road, stop the vehicle, and / or idle the vehicle until the emergency vehicle passes, with the assistance of the ultrasonic sensor 962.

[0145] The vehicle can include a CPU 918 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 904 via a high-speed interconnect (e.g., PCIe). The CPU 918 can include, for example, an X86 processor. The CPU 918 can be used to perform any of a wide variety of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC 904, and / or monitoring the status and health of the controller 936 and / or the infotainment SoC 930.

[0146] The vehicle 900 can include a GPU 920 (e.g., a discrete GPU or dGPU) that can be coupled to the SoC 904 via a high-speed interconnect (e.g., NVIDIA’s NVLINK). The GPU 920 can provide additional artificial intelligence functionality, for example, by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based on input (e.g., sensor data) from sensors of the vehicle 900.

[0147] The vehicle 900 can further include a network interface 924 that can include one or more wireless antennas 926 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). The network interface 924 can be used to enable wireless connections through the Internet with the cloud (e.g., with the server 978 and / or other network devices), with other vehicles, and / or with computing devices (e.g., client devices of passengers). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across a network and through the Internet). The direct link can be provided using a car-to-car communication link. The car-to-car communication link can provide the vehicle 900 with information about vehicles that are approaching the vehicle 900 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 900). This functionality can be part of a cooperative adaptive cruise control functionality of the vehicle 900.

[0148] The network interface 924 can include a SoC that provides modulation and demodulation functionality and enables the controller 936 to communicate over a wireless network. The network interface 924 can include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion can be performed through well-known processes, and / or can be performed using a super-heterodyne process. In some examples, the radio frequency front end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0149] The vehicle 900 can further include a data store 928, which can include off-chip (e.g., off-SoC 904) storage. The data store 928 can include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disks, and / or other components and / or devices that can store data for at least one bit.

[0150] The vehicle 900 can further include a GNSS sensor 958. The GNSS sensor 958 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 958 can be used, including, for example and without limitation, a GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0151] The vehicle 900 can further include a RADAR sensor 960. The RADAR sensor 960 can be used by the vehicle 900 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The RADAR functional safety level can be ASIL B. The RADAR sensor 960 can use the CAN and / or bus 902 (e.g., to transmit data generated by the RADAR sensor 960) for control as well as access to object tracking data, in some examples, Ethernet for access to raw data. A wide variety of RADAR sensor types can be used. For example and without limitation, the RADAR sensor 960 can be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor is used.

[0152] The RADAR sensor 960 can include different configurations, such as long-range with narrow field of view, short-range with wide field of view, short-range side coverage, and so on. In some examples, long-range RADAR can be used for adaptive cruise control functionality. Long-range RADAR systems can provide a wide field of view (e.g., 250 m range) implemented through two or more independent scans. The RADAR sensor 960 can help distinguish between static and moving objects, and can be used by the ADAS system for emergency brake assist and forward collision warning. The long-range RADAR sensor can include a single-station multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas, as well as high-speed CAN and FlexRay interfaces. In examples with six antennas, the central four antennas can create focused beam patterns designed to record the surroundings of the vehicle 900 at higher speed with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 900.

[0153] As one example, a mid-range RADAR system can include a range of up to 960 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 950 degrees (rear). A short-range RADAR system can include, but is not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor the rear and blind spots to the sides of the vehicle.

[0154] A short-range RADAR system can be used in an ADAS system for blind spot detection and / or lane change assist.

[0155] The vehicle 900 can further include ultrasonic sensors 962. The ultrasonic sensors 962, which can be placed on the front, rear, and / or sides of the vehicle 900, can be used for parking assist and / or to create and update an occupancy grid. A wide variety of ultrasonic sensors 962 can be used, and different ultrasonic sensors 962 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensors 962 can operate at an ASIL B functional safety level.

[0156] The vehicle 900 can include LIDAR sensors 964. The LIDAR sensors 964 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensors 964 can be at an ASIL B functional safety level. In some examples, the vehicle 900 can include multiple LIDAR sensors 964 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0157] In some examples, the LIDAR sensors 964 can be capable of providing a list of objects and their distances for a 360-degree field of view. A commercially available LIDAR sensor 964 can have, for example, an advertised range of approximately 900 m, a precision of 2 cm - 3 cm, and support for a 900 Mbps Ethernet connection. In some examples, one or more flush-mounted LIDAR sensors 964 can be used. In such examples, the LIDAR sensors 964 can be implemented as small devices that can be embedded into the front, rear, sides, and / or corners of the vehicle 900. In such examples, the LIDAR sensors 964 can provide a field of view of up to 920 degrees horizontal and 35 degrees vertical with a range of 200 m, even for low reflectivity objects. Front-mounted LIDAR sensors 964 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0158] In some examples, LIDAR technology such as 3D Flash LIDAR can also be used. 3D Flash LIDAR uses a flash of laser light as a source of emission to illuminate the vehicle’s surroundings up to about 200 m. The flash LIDAR unit includes a receptor that records the laser pulse transmission time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LIDAR can allow for the generation of highly accurate and distortion-free images of the surroundings with each laser flash. In some examples, four flash LIDAR sensors can be deployed, one on each side of the vehicle 900. Available 3D flash LIDAR systems include solid-state 3D staring array LIDAR cameras (e.g., non-scanning LIDAR devices) that have no moving parts other than fans. The flash LIDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture the reflected laser light in the form of 3D range point clouds and co-registered intensity data. By using flash LIDAR, and because flash LIDAR is a solid-state device with no moving parts, the LIDAR sensor 964 can be less susceptible to motion blur, vibration, and / or jostling.

[0159] The vehicle can further include an IMU sensor 966. In some examples, the IMU sensor 966 can be located at the center of the rear axle of the vehicle 900. The IMU sensor 966 can include, for example and without limitation, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, the IMU sensor 966 can include an accelerometer and a gyroscope, for example in a six-axis application, while in a nine-axis application the IMU sensor 966 can include an accelerometer, a gyroscope, and a magnetometer.

[0160] In some embodiments, the IMU sensor 966 can be implemented as a microelectromechanical systems (MEMS) inertial navigation system (INS) that combines a microelectromechanical systems (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 966 can enable the vehicle 900 to estimate heading without input from a magnetic sensor by directly observing the change in velocity from GPS to the IMU sensor 966 and correlating it. In some examples, the IMU sensor 966 and the GNSS sensor 958 can be combined into a single integrated unit.

[0161] The vehicle can include a microphone 996 placed in and / or around the vehicle 900. The microphone 996 can be used for emergency vehicle detection and identification, among other things.

[0162] The vehicle can further include any number of camera types, including stereo cameras 968, wide-view cameras 970, infrared cameras 972, surround-view cameras 974, long and / or mid-range cameras 998, and / or other camera types. These cameras can be used to capture image data around the entire periphery of the vehicle 900. The camera types used depend on the embodiment and requirements of the vehicle 900, and any combination of camera types can be used to provide the necessary coverage around the vehicle 900. Further, the number of cameras can vary depending on the embodiment. For example, the vehicle can include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As one example and without limitation, these cameras can support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described in more detail herein with respect to FIG. 9A and FIG. 9B are described in more detail.

[0163] The vehicle 900 can further include vibration sensors 942. The vibration sensors 942 can measure vibrations of components of the vehicle, such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 942 are used, differences between the vibrations can be used to determine the friction or slip of the road surface (e.g., when there is a difference in vibration between a power driven axle and a free spinning axle).

[0164] The vehicle 900 can include an ADAS system 938. In some examples, the ADAS system 938 can include a SoC. The ADAS system 938 can include adaptive / automatic / autonomous cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functionality.

[0165] The ACC system can use RADAR sensors 960, LIDAR sensors 964, and / or cameras. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately ahead of the vehicle 900 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance keeping and, if necessary, suggests a lane change for the vehicle 900. Lateral ACC is relevant to other ADAS applications such as LCA and CWS.

[0166] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link via the network interface 924 and / or wireless antenna 926 or indirectly through a network connection, such as through the Internet. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Generally, V2V communication concepts provide information about the immediately preceding vehicles, such as vehicles immediately ahead of and in the same lane as the vehicle 900, while I2V communication concepts provide information about traffic further ahead. A CACC system can include either or both of I2V and V2V information sources. Given information about vehicles ahead of the vehicle 900, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0167] FCW systems are designed to alert the driver to a hazard so that the driver can take corrective action. FCW systems use a front-facing camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components. FCW systems can provide warnings in the form of, for example, sound, visual warnings, vibrations, and / or quick brake pulses.

[0168] AEB systems detect an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within specified time or distance parameters. AEB systems can use a front-facing camera and / or RADAR sensor 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When an AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes in an effort to prevent or at least mitigate the effects of a predicted collision. AEB systems can include technologies such as dynamic brake support and / or crash imminent braking.

[0169] LDW systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 900 is crossing lane markers. The LDW system is not activated when the driver indicates an intentional lane departure by activating a turn signal. LDW systems can use a front-side facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as displays, speakers, and / or vibrating components.

[0170] An LKA system is a variation of the LDW system. If the vehicle 900 begins to leave the lane, the LKA system provides a steering input or brake to correct the vehicle 900.

[0171] A BSW system detects and warns the driver of vehicles in the car's blind spot. The BSW system can provide visual, audible, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use rear side-facing cameras and / or RADAR sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0172] A RCTW system can provide visual, audible, and / or tactile notifications when objects are detected outside the range of the rear-facing camera while the vehicle 900 is backing up. Some RCTW systems include AEB to ensure vehicle brakes are applied to avoid a collision. The RCTW system can use one or more rear-facing RADAR sensors 960 coupled to a dedicated processor, DSP, FPGA, and / or ASIC that is electrically coupled to driver feedback such as a display, speaker, and / or vibrating component.

[0173] Conventional ADAS systems can be prone to false positive results, which can annoy and distract the driver, but typically are not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether the safety condition is truly present and act accordingly. However, in an autonomous vehicle 900, in the case of conflicting results, the vehicle 900 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 936 or the second controller 936). For example, in some embodiments, the ADAS system 938 can be a secondary and / or auxiliary computer for providing perception information to a backup computer plausibility module. The backup computer plausibility monitor can run redundant diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 938 can be provided to a supervisory MCU. If the outputs from the primary computer and the secondary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0174] In some examples, the host computer can be configured to provide a confidence score to the supervisory MCU indicating the host computer's confidence in the selected result. If the confidence score exceeds a threshold, then the supervisory MCU can follow the host computer's direction, regardless of whether the secondary computer provides conflicting or inconsistent results. In the event that the confidence score does not satisfy the threshold and in the event that the host computer and the secondary computer indicate different results (e.g., a conflict), the supervisory MCU can arbitrate between the computers to determine the appropriate result.

[0175] The supervisory MCU can be configured to run a neural network that is trained and configured to determine conditions under which the secondary computer provides false alarms based on the output from the host computer and the secondary computer. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying metal objects that are not in fact dangerous, such as drain grates or manhole covers that trigger false alarms. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to disregard the LDW when a cyclist or pedestrian is present and lane departure is in fact the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU can include at least one of a DLA or a GPU suitable for running the neural network with associated memory. In preferred embodiments, the supervisory MCU can include and / or be included as a component of the SoC 904.

[0176] In other examples, the ADAS system 938 can include secondary computers that perform ADAS functions using traditional computer vision rules. As such, the secondary computers can use classic computer vision rules (if-then), and the presence of a neural network in the supervisory MCU can improve reliability, safety, and performance. For example, the diverse implementation and intentional non-identity make the overall system more fault-tolerant, especially with respect to faults caused by software (or software-hardware interface) functions. For example, if there is a software bug or error in the software running on the host computer and the non-identical software code running on the secondary computer provides the same overall result, then the supervisory MCU can be more confident that the overall result is correct and that the bug in the software or hardware on the host computer did not cause a substantial error.

[0177] In some examples, the output of the ADAS system 938 can be fed to a perception block of the host computer and / or a dynamic driving task block of the host computer. For example, if the ADAS system 938 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information in identifying the object. In other examples, the secondary computer can have its own neural network that is trained and thus reduces the risk of false positives as described herein.

[0178] The vehicle 900 can further include an infotainment SoC 930 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as a SoC, the infotainment system can not be a SoC and can include two or more discrete components. The infotainment SoC 930 can include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephony (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, park assist, radio data system, vehicle related information such as fuel level, total distance covered, brake fuel level, oil level, doors open / closed, air filter information, etc.) to the vehicle 900. For example, the infotainment SoC 930 can include a radio, a disc player, a navigation system, a video player, USB and Bluetooth connectivity, an in-car computer, in-car entertainment, Wi-Fi, steering wheel audio controls, hands-free voice controls, a heads-up display (HUD), the HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 930 can further be used to provide information (e.g., visual and / or audible) to a user of the vehicle, such as information from the ADAS system 938, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0179] The infotainment SoC 930 can include GPU functionality. The infotainment SoC 930 can communicate with other devices, systems, and / or components of the vehicle 900 over the bus 902 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 930 can be coupled to a supervisory MCU such that, in the event of a failure of the host controller 936 (e.g., a primary and / or backup computer of the vehicle 900), the GPU of the infotainment system can perform some autonomous driving functions. In such examples, the infotainment SoC 930 can place the vehicle 900 in a driver safe park mode as described herein.

[0180] Vehicle 900 may further include instrument cluster 932 (e.g., digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 932 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). Instrument cluster 932 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 930 and instrument cluster 932. In other words, instrument cluster 932 may be included as part of infotainment SoC 930, or vice versa.

[0181] FIG. 9D For cloud-based servers and according to some embodiments of this disclosure FIG. 9A This is a system diagram illustrating communication between example autonomous vehicles 900. System 976 may include server 978, network 990, and vehicles including vehicle 900. Server 978 may include multiple GPUs 984(A)-984(H) (collectively referred to herein as GPU 984), PCIe switches 982(A)-982(H) (collectively referred to herein as PCIe switch 982), and / or CPUs 980(A)-980(B) (collectively referred to herein as CPU 980). GPUs 984, CPUs 980, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 986, such as, but not limited to, NVLink interface 988 developed by NVIDIA. In some examples, GPUs 984 are connected via NVLink and / or NVSwitch SoCs, and GPUs 984 and PCIe switches 982 are connected via PCIe interconnects. Although eight GPUs 984, two CPUs 980, and two PCIe switches are shown in the diagram, this is not intended to be limiting. Depending on the embodiment, each of the servers 978 may include any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, each of the servers 978 may include eight, sixteen, thirty-two, and / or more GPUs 984.

[0182] The server 978 can receive image data from vehicles over the network 990 and representing images showing unexpected or changing road conditions such as a road work that recently started. The server 978 can transmit neural networks 992, updated neural networks 992, and / or map information 994, including information about traffic and road conditions, to vehicles over the network 990. Updates to the map information 994 can include updates to the HD map 922, e.g., information about construction sites, potholes, curves, floods, or other obstacles. In some examples, the neural networks 992, updated neural networks 992, and / or map information 994 can have been generated from experience using training performed at a data center (e.g., using the server 978 and / or other servers) and / or from data received from any number of vehicles in the environment.

[0183] The server 978 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated by vehicles and / or can be generated in simulations (e.g., using game engines). In some examples, the training data is labeled (e.g., in cases where the neural network benefits from supervised learning) and / or undergoes other pre-processing, while in other examples, the training data is not labeled and / or pre-processed (e.g., in cases where the neural network does not require supervised learning). The training can be performed according to any one or more categories of machine learning techniques, including but not limited to: categories such as supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including spare dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning models are trained, the machine learning models can be used by vehicles (e.g., transmitted to vehicles over the network 990) and / or the machine learning models can be used by the server 978 to remotely monitor vehicles.

[0184] In some examples, the server 978 can receive data from vehicles and apply the data to the latest real-time neural networks for real-time intelligent inference. The server 978 can include deep learning supercomputers and / or specialized AI computers powered by GPUs 984, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, the server 978 can include deep learning infrastructure of a data center that is powered using CPUs only.

[0185] The deep learning infrastructure of the server 978 can be capable of fast real-time inference and can use this capability to assess and validate the health of the processors, software, and / or associated hardware in the vehicle 900. For example, the deep learning infrastructure can receive periodic updates from the vehicle 900, such as a sequence of images and / or objects located in the sequence of images that the vehicle 900 has located (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify the objects and compare them to the objects identified by the vehicle 900, and if the results do not match and the infrastructure concludes that the AI in the vehicle 900 is malfunctioning, then the server 978 can transmit a signal to the vehicle 900 instructing the fail-safe computer of the vehicle 900 to take control, notify the passengers, and complete a safe parking operation.

[0186] For inference, the server 978 can include GPUs 984 and one or more programmable inference accelerators (such as NVIDIA’s TensorRT). The combination of GPU-powered servers and inference-accelerated can make real-time responses possible. In other examples, such as where performance is less important, CPU-, FPGA-, and other processor-powered servers can be used for inference.

[0187] Example computing device

[0188] FIG. 10 A block diagram of an example computing device 1000 suitable for implementing some embodiments of the present disclosure. The computing device 1000 can include an interconnection system 1002 that directly or indirectly couples the following devices: memory 1004, one or more central processing units (CPU) 1006, one or more graphics processing units (GPU) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., a display), and one or more logical units 1020.

[0189] Although FIG. 10 Although various blocks of the computing device 1000 are shown as connected to the interconnection system 1002 via the busses, this is not intended to be limiting and is for clarity only. For example, in some embodiments, a presentation component 1018 such as a display can be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, a CPU 1006 and / or GPU 1008 can include memory (e.g., the memory 1004 can represent a storage device in addition to the memory of the GPU 1008, CPU 1006, and / or other components). In other words, the memory 1004 can be a component of the computing device 1000 that is separate from the GPU 1008, CPU 1006, and / or other components, or the memory 1004 can be integrated into the GPU 1008, CPU 1006, and / or other components. FIG. 10The computing device 1000 is intended to represent various computing devices used by the techniques described herein. For example, the computing device 1000 can represent a desktop, a laptop, a tablet, a server, a mobile phone, a gaming console, an electronic control unit (ECU), a virtual reality system, and / or other device or system types. The techniques described herein are not limited to use by any particular type of computing device. FIG. 10 The computing device 1000 is intended to represent various computing devices used by the techniques described herein. For example, the computing device 1000 can represent a desktop, a laptop, a tablet, a server, a mobile phone, a gaming console, an electronic control unit (ECU), a virtual reality system, and / or other device or system types. The techniques described herein are not limited to use by any particular type of computing device.

[0190] The interconnection system 1002 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1002 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standards Association (VESA) bus, a Peripheral Component Interconnect (PCI) bus, a Peripheral Component Interconnect Express (PCIe) bus, and / or another type of bus or link. In some embodiments, there are direct connections between components. For example, the CPU 1006 can be directly connected to the memory 1004. Also, the CPU 1006 can be directly connected to the GPU 1008. Where there are direct or point-to-point connections between components, the interconnection system 1002 can include a PCIe link to perform the connection. In these examples, a PCI bus need not be included in the computing device 1000.

[0191] The memory 1004 can include any of a wide variety of computer-readable media. Computer-readable media can be any available media that can be accessed by the computing device 1000. Computer-readable media can include both volatile and nonvolatile media, and removable and non-removable media. By way of example, and not limitation, computer-readable media can comprise computer storage media and communication media.

[0192] Computer storage media can include volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, the memory 1004 can store computer-readable instructions such as those representing programs and / or program elements, e.g., an operating system.

[0193] Computer storage media can include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1000. Computer storage media, as used herein, does not include signals per se.

[0194] Computer storage media can include computer-readable instructions, data structures, program modules, and / or other data in a modulated data signal, such as a carrier wave or other transport mechanism, and includes any information delivery media. The term "modulated data signal" can refer to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example, and not limitation, computer storage media can include wired media such as a wired network or direct-wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media. Combinations of the above should also be included within the scope of computer readable media.

[0195] The CPUs 1006 can be configured to execute at least some of the computer- readable instructions in order to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. Each of the CPUs 1006 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a large number of software threads concurrently. The CPUs 1006 can include any type of processors and can include different types of processors depending on the type of computing device 1000 being implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1000, the processors can be Advanced RISC Machines (ARM) processors implemented using reduced instruction set computing (RISC) or x86 processors implemented using complex instruction set computing (CISC). The computing device 1000 can include one or more CPUs 1006 in addition to one or more microprocessors or complementary co-processors such as math co-processors.

[0196] In addition to or instead of CPU 1006, one or more GPUs 1008 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. One or more GPUs 1008 can be integrated GPUs (e.g., with one or more CPUs 1006) and / or one or more GPUs 1008 can be discrete GPUs. In embodiments, one or more GPUs 1008 can be co-processors of one or more CPUs 1006. Computing device 1000 can use GPUs 1008 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, one or more GPUs 1008 can be used for general-purpose computing on GPUs (GPGPU). One or more GPUs 1008 can include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 1008 can generate pixel data for output images in response to rendering commands (e.g., rendering commands received from CPU 1006 via a host interface). GPU 1008 can include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory can be included as part of memory 1004. One or more GPUs 1008 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using NVLINK) or through a switch (e.g., using NVSwitch). When combined together, each GPU 1008 can generate pixel data or GPGPU data for a different portion of an output or a different output (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or share memory with other GPUs.

[0197] In addition to, or alternatively, CPU(s) 1006 and / or GPU(s) 1008, logic unit(s) 1020 can be configured to execute at least some of the computer-readable instructions to control one or more components of computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, CPU(s) 1006, GPU(s) 1008, and / or logic unit(s) 1020 can perform any combination of the methods, processes, and / or portions thereof, discretely or jointly. Logic unit(s) 1020 can be part of and / or integrated in one or more of CPU(s) 1006 and / or GPU(s) 1008, and / or logic unit(s) 1020 can be discrete components or otherwise external to CPU(s) 1006 and / or GPU(s) 1008. In embodiments, logic unit(s) 1020 can be a co-processor of one or more CPU(s) 1006 and / or one or more GPU(s) 1008.

[0198] Examples of logic unit(s) 1020 include one or more processing cores and / or components thereof, such as tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), visual processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multi-processors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating point units (FPUs), input / output (I / O) elements, peripheral component interconnects (PCI) or peripheral component interconnect express (PCIe) elements, and the like.

[0199] Communication interface(s) 1010 can include one or more receivers, transmitters, and / or transceivers that enable computing device 1000 to communicate with other computing devices via electronic communication networks, including wired and / or wireless communication. Communication interface(s) 1010 can include components and functionality to enable communication over any of a number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, and the like), wired networks (e.g., communication over Ethernet or InfiniBand), low power wide area networks (e.g., LoRaWAN, SigFox, and the like), and / or the Internet.

[0200] The I / O port 1012 can enable the computing device 1000 to logically couple to other devices, including I / O components 1014, presentation components 1018, and / or other components, some of which can be built into (e.g., integrated with) the computing device 1000. Illustrative I / O components 1014 include a microphone, mouse, keyboard, joystick, game pad, game controller, satellite dish, scanner, printer, wireless device, etc. The I / O components 1014 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some instances, inputs can be transmitted to an appropriate network element for further processing. A NUI can implement any combination of speech recognition, handwriting recognition, facial recognition, biometric recognition, gesture recognition both on screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with a display of the computing device 1000. The computing device 1000 can include depth cameras, infrared cameras, RGB cameras, touch screen technology, and combinations of these, such as a stereoscopic camera system, a infrared camera system, an RGB camera system, touch screen technology, and combinations of these, for gesture detection and recognition.

[0201] The power supply 1016 can include a hard-wired power supply, a battery power supply, or a combination thereof. The power supply 1016 can supply power to the computing device 1000 to enable the components of the computing device 1000 to operate.

[0202] The presentation components 1018 can include a display (e.g., a monitor, a touch screen, a television, a heads-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation components 1018 can receive data from other components (e.g., the GPU 1008, the CPU 1006, etc.) and output the data (e.g., as a

[0203] Example network environment

[0204] A network environment suitable for implementing embodiments of the present disclosure can include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) can be implemented on one or more instances of the computing device 1000— e.g., each device can include similar components, features, and / or functionality of the computing device 1000. FIG. 10

[0205] ​Components of a network environment can communicate with each other over a network, which can be wired, wireless, or both. The network can include multiple networks, or one of multiple networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the public switched telephone network (PSTN), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (among other components) can provide wireless connectivity.

[0206] A compatible network environment can include one or more peer-to-peer network environments (in which case servers can not be included in the network environment) and one or more client-server network environments (in which case one or more servers can be included in the network environment). In a peer-to-peer network environment, functionality described herein with respect to servers can be implemented on any number of client devices.

[0207] In at least one embodiment, the network environment can include one or more cloud-based network environments, distributed computing environments, combinations thereof, and the like. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which can include one or more core network servers and / or edge servers. The framework layer can include a framework for supporting a software layer and / or one or more applications of an application layer. The software or applications can include network-based service software or applications, respectively. In embodiments, one or more client devices can use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer can be, without limitation, a type of free and open-source software web application framework, such as can be used for large-scale data processing (e.g., “big data”) using the distributed file system.

[0208] The cloud-based network environment can provide cloud computing and / or cloud storage that performs any combination of the computing and / or data storage functionality described herein (or one or more portions thereof). Any of these various functionalities can be distributed across multiple locations of central or core servers (e.g., one or more data centers that can be distributed across states, regions, countries, globally, etc.). The core servers can designate at least a portion of the functionality to edge servers if the connection to the user (e.g., client device) is relatively close to the edge servers. The cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0209] A client device can include at least some components, features, and functionality of the example computing device 1000 described herein FIG. 10 by way of example and not limitation, a client device can be embodied as a personal computer (PC), a laptop computer, a mobile device, a smartphone, a tablet computer, a smartwatch, a wearable computer, a personal digital assistant (PDA), an MP3 player, a virtual reality headset, a global positioning system (GPS) or device, a video player, a video camera, a surveillance device or system, a vehicle, a watercraft, an aircraft, a virtual machine, a drone, a robot, a handheld communication device, a hospital device, a gaming device or system, an entertainment system, an in-vehicle computer system, an embedded system controller, a remote control, an appliance, a consumer electronic device, a workstation, an edge device, any combination of these devices described, or any other suitable device.

[0210] The present disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, being executed by a computer or other machine, such as a personal data assistant or other handheld device. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform particular tasks or implement particular abstract data types. The present disclosure can be practiced in a variety of system configurations, including hand-held devices, consumer electronics, general- purpose computers, more specialty computing devices, etc. The present disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network.

[0211] As used herein, the term “and / or,” when used in a list of two or more elements, means that any one of the elements can be present, or combinations of two or more of the elements can be present. For example, “element A, element B, and / or element C” can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Also, “at least one of element A or element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, “at least one of element A and element B” can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0212] The subject matter of the present disclosure is described with specificity herein to meet statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have contemplated that the claimed subject matter might also be embodied in other ways, to include different steps or combinations of steps similar to the ones described in this document, in conjunction with other present or future technologies. Moreover, although the terms "step" and / or "block" might be used herein to connote different elements of methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described.

Claims

1. A method comprising: Applying image data representing the intersection to a neural network; Using the neural network and based at least in part on the image data, first data representing a first signed distance function corresponding to a first intersection area category and a second signed distance function corresponding to a second intersection area category are calculated; The clustering algorithm is applied to the first signed distance function and the second signed distance function; Based at least in part on the clustering algorithm, a first instance segmentation map corresponding to the first intersection region category and a second instance segmentation map corresponding to the second intersection region category are calculated; Based at least in part on the first instance segmentation map and the second instance segmentation map, the world spatial location corresponding to the instance of the first intersection region category and the instance of the second intersection region category is determined; And a planning component that transmits second data representing the world spatial location to the vehicle.

2. The method of claim 1, further comprising: A smoothing operation is applied to the first signed distance function and the second signed distance function to generate a smoothed first signed distance function and a smoothed second signed distance function; and the clustering algorithm is applied to the smoothed first signed distance function and the smoothed second signed distance function.

3. The method of claim 2, wherein the smoothing operation is applied to each pixel of the image representing the image data and corresponds to the values ​​of the first signed distance function and the second signed distance function.

4. The method of claim 2, wherein the smoothing operation is a Gaussian smoothing operation.

5. The method of claim 1, wherein the image data is applied to the neural network at a first spatial resolution, and the first signed distance function and the second signed distance function are output at a second spatial resolution smaller than the first spatial resolution, and the method further comprises: An upsampling operation is applied to the first signed distance function and the second signed distance function to generate an upsampled first signed distance function and an upsampled second signed distance function at the first spatial resolution.

6. The method as described in claim 1, wherein the clustering algorithm is a watershed clustering algorithm.

7. The method of claim 1, further comprising: A zero-value threshold is applied to each symbol of the first signed distance function and the second signed distance function, wherein the clustering algorithm is applied after the zero-value threshold is applied.

8. The method of claim 1, wherein the first instance segmentation map and the second instance segmentation map are represented in image space, and determining the world spatial location comprises: The image spatial location is converted into the world spatial location using one or more of the intrinsic sensor parameters or extrinsic sensor parameters.

9. A system comprising: One or more image sensors in a vehicle; One or more processors; One or more memory devices storing instructions that, when executed by the one or more processors, cause the one or more processors to perform operations, the operations including: applying image data representing an intersection to a deep neural network (DNN); using the DNN and at least partially based on the image data, computing first data representing a signed distance function, each of the signed distance functions corresponding to an intersection region category that the DNN is trained to predict; applying a clustering algorithm to the signed distance functions; computing instance segmentation maps in image space, each instance segmentation map corresponding to an intersection region category of a corresponding signed distance function, at least partially based on the clustering algorithm; determining a world spatial location corresponding to the instance segmentation map; and transmitting second data representing the world spatial location to a vehicle planning component.

10. The system of claim 9, wherein the operation further comprises: A smoothing operation is applied to each of the signed distance functions to generate a corresponding smoothed first signed distance function; Furthermore, the clustering algorithm applied therein is the smoothed first signed distance function.

11. The system of claim 10, wherein the smoothing operation is applied to each pixel of the image representing the image data and corresponds to the value of each signed distance function in the signed distance function.

12. The system of claim 9, wherein the operation further comprises: Apply a zero-value threshold to each symbol of the signed distance function; The clustering algorithm is applied after the zero threshold is applied.

13. The system of claim 9, wherein the image data is applied to the DNN at a first spatial resolution, the signed distance function is output at a second spatial resolution smaller than the first spatial resolution, and the operation further includes: An upsampling operation is applied to each of the signed distance functions to generate a corresponding upsampled signed distance function at the first spatial resolution.

Images