Object Detection Methods for LiDAR Perception

By using multi-view perception deep neural networks to process the perspective and top-down views of LiDAR data, the problem of low accuracy and efficiency in object detection in traditional methods is solved. This enables high recall detection and classification of objects such as pedestrians and bicycles, improving the safety and efficiency of autonomous driving systems.

CN112904370BActive Publication Date: 2024-11-26NVIDIA CORP
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202011272919.8
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Priority Date
2020-06-29
Filing Date
2020-11-13
Publication Date
2024-11-26
Estimated Expiration
2040-11-13

Smart Images

  • Figure CN112904370B_ABST
    Figure CN112904370B_ABST
Patent Text Reader

Abstract

The present application relates to techniques for multi-view deep neural networks for lidar perception. Deep neural networks (DNNs) can be used to detect objects from sensor data of a three-dimensional (3D) environment. For example, a multi-view perception DNN may include multiple component DNNs or stages linked together that sequentially process different views of a 3D environment. An example DNN may include a first stage that performs classification segmentation in a first view (e.g., a perspective view) and a second stage that performs classification segmentation and / or regression instance geometry in a second view (e.g., a top view). The DNN output may be processed to generate 2D and / or 3D bounding boxes and classification labels for objects detected in a 3D environment. Therefore, the techniques described herein can be used to detect and classify active objects and / or parts of an environment, and these detections and classifications can be provided to an autonomous vehicle drive stack to enable safe planning and control of an autonomous vehicle.
Need to check novelty before this filing date? Find Prior Art

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 936,080, filed on November 15, 2019, and U.S. Provisional Application No. 62 / 938,852, filed on November 21, 2019, the entire contents of which are incorporated herein by reference. Background Art

[0003] Designing systems that can safely drive a vehicle without supervision is extremely difficult. An autonomous vehicle should at least be functionally equivalent to a careful driver who, using perception and action systems, is able to identify and respond to moving and static obstacles in a complex environment, thereby avoiding collisions with other objects or structures along the vehicle's path. Therefore, the ability to detect moving objects (e.g., cars, pedestrians, etc.) and other parts of the environment is generally critical for autonomous driving perception systems. Traditional perception methods generally rely on cameras or lidar sensors to detect objects in the environment, and a variety of methods using deep neural networks (DNNs) have been developed to perform lidar and camera perception. The classification of such DNNs includes DNNs that perform panoramic segmentation of camera images in perspective view, and DNNs that perform overhead or "bird's eye view" (BEV) object detection from lidar point clouds. However, these traditional methods have many disadvantages.

[0004] For example, traditional panoptic segmentation DNNs typically perform classification and instance segmentation on images in perspective views (e.g., RBG images from frontal cameras or LiDAR range scans). Figure 1 is an illustration of an exemplary lidar range scan with a perspective view and classifications segmented from panoramic segments of the lidar range scan. Figure 1 , each lidar input (range scan) is shown with a corresponding classification mask showing an exemplary segmentation output (segment classification). Figure 1 Segment instances are omitted. Figure 2 is an illustration of an exemplary panoptic segmentation of a camera image with a perspective view. Figure 2 , the top image is the input image to be segmented, the middle image is the classification mask showing the segmentation classification superimposed on the input image, and the bottom image is the instance mask showing the segmentation instances superimposed on the input image.

[0005] Panoptic segmentation in perspective views often performs well for certain classes, such as pedestrians and cyclists, due to the geometric characteristics of objects in certain classes (e.g., consistent structures). However, panoptic segmentation is often challenged when evaluating features that are not visible from the perspective of the view being analyzed. For example, while panoptic segmentation is able to detect pedestrians in frontal images, panoptic segmentation DNNs often have difficulty accurately predicting 3D bounding boxes or BEV two-dimensional (2D) bounding boxes for detected objects. Similarly, panoptic segmentation DNNs using perspective views often have difficulty accurately detecting objects with features that are not visible from the perspective of the view being analyzed. As a result, conventional panoptic segmentation DNNs have limited accuracy in predicting object classes, object instances, dimensions, and orientations.

[0006] Traditional DNNs perform object detection from BEV (bird’s eye view) projections of LiDAR point clouds, typically detecting a single class including cars, trucks, buses, pedestrians, and / or cyclists when predicting the BEV 2D bounding box. Figure 3 is an illustration of an example object detection performed on a top-down projection of a LiDAR point cloud. DNNs performing BEV object detection typically have difficulty accurately detecting pedestrians or bicycles because the top-down view of these objects is often similar to the top-down views of other objects such as poles, tree trunks, or bushes. As a result, traditional DNNs performing BEV object detection have limited accuracy in predicting object classification, size, and orientation. One possible solution to this problem is to use 3D convolutions on a 3D voxelized volume. However, 3D convolutions are computationally expensive and require processing a large amount of empty voxel space in the volume to be searched, resulting in inefficiencies. Summary of the invention

[0007] Embodiments of the present invention relate to lidar perception for autonomous machines using deep neural networks (DNNs). For example, the systems and methods described herein use object detection techniques to identify or detect instances of obstacles (e.g., cars, trucks, pedestrians, cyclists, etc.) and other objects such as environmental components for autonomous vehicles, semi-autonomous vehicles, robots, and / or other object types. Compared to conventional systems (e.g., the systems described above), the systems of the present invention may include multiple component DNNs or stages linked together that sequentially process different views of a 3D environment. An exemplary multi-view perception DNN may include a first stage that performs classification segmentation in a first view (e.g., a perspective view) and a second stage that performs classification segmentation and / or regresses instance geometry in a second view (e.g., a top-down view).

[0008] For example, the first stage may extract classification data (e.g., confidence maps, fragment masks, etc.) from a lidar range image or an RGB image. The extracted classification data may be converted to a second view of the environment, for example, by marking corresponding 3D locations (e.g., identified by corresponding pixels of the lidar range image) with the extracted classification data and projecting the marked 3D locations to the second view. In some embodiments, geometric data (e.g., height data) of objects in 3D space may be obtained from sensor data (e.g., by projecting a lidar point cloud into one or more height maps in a top-down view) and / or an image of the 3D space (e.g., by projecting the image into world space and projecting it into a top-down view). The extracted classification data and / or geometric data may be stacked and input to a second stage of the DNN, which may extract classification data (e.g., classification confidence maps) and / or regress various types of information about detected objects, such as positions, geometries, and / or orientations in the second view. The DNN output may be processed to generate 2D and / or 3D bounding boxes and classification labels for objects detected in the 3D environment.

[0009] Thus, the techniques described herein can be used to detect and classify active objects and / or parts of the environment, and these detections and classifications can be provided to the autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle. Unlike traditional methods, by sequentially processing multiple views of sensor data using a multi-view-aware DNN, the present technology can retain the advantages of processing each view individually while mitigating potential disadvantages. Using the methods described herein, motorcycles, bicycles, pedestrians, and other vulnerable road users (VRUs) or objects can be detected with high recall rates. In addition, embodiments of the present disclosure can provide a simple and effective method to detect and classify objects, and regress their size and orientation, which is difficult to do with traditional methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The present multi-view lidar perception system and method are described in detail below with reference to the accompanying drawings, wherein:

[0011] Figure 1 is an illustration of an exemplary lidar range scan with a perspective view and classifications segmented from panoramic segments of the lidar range scan;

[0012] Figure 2 is an illustration of an example panoptic segmentation of a camera image under perspective viewing;

[0013] Figure 3 is an illustration of an exemplary object detection performed on a top-down projection of a lidar point cloud;

[0014] Figure 4is a data flow diagram illustrating an example process of an object detection system according to some embodiments of the present disclosure;

[0015] Figure 5 is a data flow diagram illustrating an example process for preprocessing sensor data for a machine learning model in an object detection system according to some embodiments of the present disclosure;

[0016] Figure 6 is a diagram of an exemplary multi-view-aware machine learning model according to some embodiments of the present disclosure;

[0017] Figure 7 is a data flow diagram illustrating an example post-processing process for generating object detections in an object detection system according to some embodiments of the present disclosure;

[0018] Figure 8 is a diagram of an example data flow through an example multi-view-aware machine learning model according to some embodiments of the present disclosure;

[0019] Fig. 9 is a flow chart illustrating a method for multi-view object detection using sensor data according to some embodiments of the present disclosure;

[0020] Fig.10 is a flow chart illustrating a method for perspective and top-down view object detection using LiDAR data according to some embodiments of the present disclosure;

[0021] Fig.11 is a flow chart illustrating a method of multi-view object detection involving projection of tag sensor data according to some embodiments of the present disclosure;

[0022] Fig.12 is an illustration of an exemplary technique for annotating sensor data from different sensors according to some embodiments of the present disclosure;

[0023] Fig.13 is an illustration of example annotations of car and truck classification in camera space according to some embodiments of the present disclosure;

[0024] Fig.14 is an illustration of example annotations for pedestrian classification in camera space according to some embodiments of the present disclosure;

[0025] Fig.15 is an illustration of example annotations of a look-down bounding box in lidar space, according to some embodiments of the present disclosure;

[0026] Fig.16A is an illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure;

[0027] Fig. 16B According to some embodiments of the present disclosure, Fig.16A Examples of camera positions and fields of view for an exemplary autonomous vehicle;

[0028] Fig. 16C According to some embodiments of the present disclosure, Fig.16A A block diagram of an exemplary system architecture of an exemplary autonomous vehicle;

[0029] Fig.16D According to some embodiments of the present disclosure, a method for Fig.16A A system diagram of an exemplary autonomous vehicle communicating with each other; and

[0030] Fig.17 is a block diagram of an example computing device suitable for implementing some embodiments of the invention. DETAILED DESCRIPTION

[0031] Systems and methods are disclosed relating to multi-view lidar perception for autonomous machines using deep neural networks (DNNs). For example, the systems and methods described herein use object detection techniques to identify or detect instances of obstacles (e.g., cars, trucks, pedestrians, cyclists, etc.) and other objects such as parts of the environment for autonomous vehicles, semi-autonomous vehicles, robots, and / or other object types.

[0032] Although the present disclosure may be described with respect to example autonomous vehicle 1600 (also referred to herein as “vehicle 1600 ” or “autonomous vehicle 1600 ”), examples thereof may be described herein with respect to example autonomous vehicle 1600 . Figures 16A-16D For example, the systems and methods described herein may be used in non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), robots, warehouse vehicles, off-road vehicles, flying vessels, boats, and / or other vehicle types. In addition, although the present disclosure may be described with respect to autonomous driving, this is not intended to be limiting. For example, the systems and methods described herein may be used in robotics (e.g., path planning for a robot), aerial systems (e.g., path planning for a drone or other aircraft), marine systems (e.g., path planning for a ship or other watercraft), and / or other technical fields, such as for positioning, path planning, and / or other processes.

[0033] At a high level, a DNN can be used to detect objects from LiDAR data and / or other sensor data that captures a three-dimensional (3D) environment. In some embodiments, the DNN can include multiple component DNNs or stages linked together that sequentially process different views of the 3D environment. An exemplary multi-view aware DNN can include a first stage that performs classification segmentation in a first view (e.g., a perspective view) and a second stage that performs classification segmentation and / or regresses instance geometry in a second view (e.g., a top-down view).

[0034] In some embodiments, the input to the DNN can be formed by lidar data (e.g., a lidar range image, a projection of a lidar point cloud, etc.) and / or data from other sensors (e.g., images from any number of cameras), and a first stage can extract classification data (e.g., classification confidence data, such as confidence maps for any number of classifications) from the input. The confidence map and / or composite segmentation mask can segment (and therefore represent) a first view (e.g., a perspective view) of a 3D space. The confidence map and / or composite segmentation can be projected to a second view (e.g., a top view) to generate transformed classification data for processing by subsequent DNN stages. For example, the extracted classification data can be used to mark corresponding 3D locations (e.g., identified by a lidar range image), and the marked 3D locations (e.g., the marked lidar range image) can be re-projected to a second view of the environment.

[0035] In some embodiments, geometric data (e.g., height data) for objects in 3D space can be obtained from lidar data (e.g., by projecting a lidar point cloud into one or more height maps in a top-down view) and / or an image of the 3D space (e.g., by projecting the image into world space and projecting into a top-down view). The transformed classification data and geometric data can be stacked and input to a second stage of the DNN, which can extract classification data (e.g., classification confidence data, such as confidence maps for any number of classifications) and / or regress various types of information about detected objects, such as position, geometry, and / or orientation. The DNN output can be processed to generate 2D and / or 3D bounding boxes and classification labels for objects detected in the 3D environment.

[0036] Typically, a multi-view perception DNN can accept as input some representation of lidar data and / or other sensor data of a 3D environment. In some embodiments, to form input to the DNN, raw lidar detections from the environment surrounding an autonomous object (e.g., a moving vehicle) can be preprocessed into a format that the DNN understands. In particular, lidar data (e.g., raw lidar detections from ordered or unordered lidar point clouds) can be accumulated, transformed to a single coordinate system (e.g., centered on the ego-actor), compensated for autonomous motion (e.g., to the last known position of the ego-actor), and / or projected to form a lidar range image.

[0037] In some cases, forming a range scan image such as a lidar range image may result in the loss of some sensor data. For example, when detections are accumulated over time as the autonomous object moves, when sensor data is captured from different sensors mounted at different locations of the autonomous object (i.e., from different views of the scene), and / or when sensor data is collapsed into a range image with insufficient resolution to represent adjacent sensor data. In some embodiments, when reflections are blocked together in pixels of a range image, reflections with the closest range can be represented in the range image, and other reflections can be discarded. Additionally or alternatively, the resolution of the range image can be selected in a manner that reduces sensor data loss and / or limits accuracy loss. For example, the height (or vertical resolution) of the range image can be set to correspond to the number of horizontal scan lines of the sensor that captures the sensor data (e.g., a row of pixels in the range image for each scan line of the corresponding lidar sensor). The width (or horizontal resolution) of the range image can be set based on the horizontal resolution of the sensor that captures the sensor data. In general, the horizontal resolution can be a design choice: lower resolutions may have fewer conflicts, but may be easier to handle (and vice versa).

[0038] In some embodiments, the lidar range image can be fed into a multi-view perception DNN (e.g., the first stage of the DNN). In addition or alternatively, the lidar range image and / or other sensor data can be stacked into corresponding channels of the input tensor and fed into the multi-view perception DNN. In any case, the DNN can include multiple stages linked together that sequentially process data from multiple views to predict classification data and / or object instance data for objects detected in a 3D environment. These outputs can be processed into 2D and / or 3D bounding boxes and classification labels for detected objects. In an example application of an autonomous vehicle, the DNN can be used to predict one or more bounding boxes (e.g., 2D bounding boxes, 3D bounding boxes in a top view) for each detected object on a road or sidewalk, a classification label for each detected object, and a 2D mask environment portion (e.g., in a top view) that divides drivable space, sidewalks, buildings, trees, utility poles, and other static objects. In some embodiments, the 2D bounding box in the top view can be adjusted to a 3D bounding box by deriving the height of the box from the predicted object instance data.

[0039] In embodiments where the multi-view-aware DNN includes a chain of multiple stages, different stages can be trained together or separately. In some embodiments, the stages can be trained by using differentiable operations (e.g., differentiable reprojection) to implement the transformation from the output of the first stage (first view) to the input of the second stage (second view). The training data can be obtained by annotating data from multiple sensors in a sensor setting. Since data can be obtained from different sensors at different frequencies, in some embodiments, a specific sensor (e.g., a lidar sensor) can be used as a reference sensor. For each frame of sensor data from a reference sensor (e.g., for each frame of lidar data), a set of sensor data can be managed by identifying a sensor data frame from each other sensor in the sensor setting (the sensor data frame closest in time to the reference sensor). This set of sensor data (e.g., a lidar data frame at timestamp T plus an image taken at timestamp T from each of the multiple cameras in the sensor setting) can be referred to as a set of corrected sensor data at timestamp T. For each set of corrected sensor data, the data from each sensor can be labeled independently of the data of other sensors. In some embodiments, object detection and tracking can be applied to track the movement of annotated objects from frame to frame over time. Thus, annotation tracking can be used to track objects from frame to frame (e.g., using persistent identifiers of annotated objects). In some embodiments, object tracking and / or detection from sensor data of a particular sensor can be linked with corresponding object tracking and / or detection for the same object from sensor data of a different sensor. Annotations and / or links between different types of sensor data for the same object can be generated manually and / or automatically and can be used to generate training data for a multi-view aware DNN.

[0040] Thus, the techniques described herein can be used to detect and classify animated objects and / or parts of the environment, and these detections and classifications can be provided to the autonomous vehicle drive stack to enable safe planning and control of the autonomous vehicle. Unlike traditional methods, by using a multi-view-aware DNN to sequentially process multiple views of sensor data, the present technology can retain the advantages of processing each view individually while mitigating potential disadvantages. Using the methods described herein, motorcycles, bicycles, pedestrians, and other vulnerable road users (VRU) objects can be detected with high recall rates. In addition, embodiments of the present disclosure can provide a simple and effective method to detect and classify objects, and regress their size and orientation, which is difficult to do with traditional methods.

[0041] Example Object Detection System

[0042] Reference Figure 4 , Figure 41 is a data flow diagram showing an example process of an object detection system according to some embodiments of the present disclosure. It should be understood that this arrangement and other arrangements described herein are only set forth as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used to supplement or replace the content shown, and some elements may be omitted entirely. In addition, many elements described herein are functional entities that may be implemented as discrete or distributed components, or together with other components and in any appropriate combination and position. The various functions performed by the entity described herein may be performed by hardware, firmware, and / or software. For example, the various functions may be implemented by a processor executing instructions stored in a memory.

[0043] At a high level, process 400 may include a machine learning model 408 configured to detect objects, such as instances of understandable objects and / or portions of an environment, based on sensor data 402 of a three-dimensional (3D) environment. Sensor data 402 may be preprocessed 404 into input data 406 in a format understood by machine learning model 408, and input data 406 may be fed into machine learning model 408 to detect objects 416 in a 3D environment. In some embodiments, machine learning model 408 may include multiple component machine learning models or stages linked together that sequentially process different views of the 3D environment. Machine learning model 408 may predict a representation of a classification confidence of a detected object (e.g., classification confidence data 410) and / or a representation of object instance data for the detected object (e.g., instance regression data 412), which may be post-processed 414 in object detection 416 including a bounding box, closed polyline or other bounding shape to identify the location, size and / or orientation of the detected object. Object detection 416 may correspond to obstacles, static environmental components, and / or other objects around the autonomous vehicle and may be detected by control elements of the autonomous vehicle (e.g., controller 1636, ADAS system 1638, SOC 1604, software stack 422, and / or Figures 16A-16D The system may be used in conjunction with other components of the autonomous vehicle 1600 in order to help the autonomous vehicle perform one or more operations (e.g., obstacle avoidance, path planning, mapping, etc.) in an environment.

[0044] In general, object detection may be performed using sensor data 402 from any number and type of sensors, such as, but not limited to, lidar sensors, radar sensors, cameras, and / or sensors such as described below with respect to Figures 16A-16D For example, sensor 401 may include one or more sensors 401 of an autonomous object or ego character, such as Figures 16A-16DLidar sensor 1664 of autonomous vehicle 1600 is shown—and sensor 401 can be used to generate sensor data 402 representative of objects in the 3D environment surrounding the autonomous object.

[0045] Taking lidar data as an example, lidar data (e.g., sensor data 402) from one or more lidar sensors (e.g., sensor 401) can be used to perform object detection. In general, a lidar system can include a transmitter that emits laser pulses. The emitted light waves reflect from certain objects and materials, and one of the lidar sensors can detect these reflections and reflection characteristics, such as azimuth, azimuth, elevation, range (e.g., beam flight time), intensity, reflectivity, signal-to-noise ratio (SNR), etc. The reflection and reflection characteristics may depend on objects in the environment, speed, material, sensor mounting location and direction, etc. Firmware associated with the lidar sensor can be used to control the lidar sensor to capture and / or process sensor data 402, such as reflection data from the sensor's field of view.

[0046] In general, sensor data 402 may include raw sensor data, lidar point cloud data, and / or reflection data processed into some other format. For example, reflection data may be combined with position and orientation data (e.g., from GNSS and IMU sensors) to form a point cloud representing reflections detected from the environment. Each detection in the point cloud may include the three-dimensional position of the detection and metadata about the detection, such as one or more reflection features. Some non-limiting examples of lidar sensors include Velodyne HDL / VLS series and Ouster OS1 / OS2 series lidar sensors, and non-limiting example operating (e.g., scanning) frequencies may be >= 5 Hz. Although these embodiments describe sensor data 402 as lidar data, sensor data 402 may additionally or optionally include sensor data from other sensors, such as radar data (e.g., radar point clouds), image data (e.g., RBG images from one or more cameras mounted around the self-character), and / or other types.

[0047] The sensor data 402 may be preprocessed 404 into a format that the machine learning model 408 understands. For example, in embodiments where the sensor data 402 includes lidar data (and / or other radar data such as radar data), the lidar data (and / or other data) may be accumulated, converted to a single coordinate system (e.g., centered on the ego character / vehicle), autonomous motion compensated (e.g., the latest known position of the ego character / vehicle), and / or formed into a projected image of a desired size (e.g., spatial dimensions) by a projector. For example, a (accumulated, autonomous motion compensated) lidar point cloud may be projected into a range lidar range image with a perspective view. Any suitable perspective projection (e.g., spherical, cylindrical, pinhole, etc.) may be used. In some cases, the type of projection may depend on the type of sensor. As a non-limiting example, for a rotating sensor, a spherical or cylindrical projection may be used. In some embodiments, for a time-of-flight camera (e.g., a flash lidar), a pinhole projection may be used. In another instance, the (accumulated, autonomous motion compensated) radar point cloud may be orthogonally projected to form a top-down view image with a desired ground sampling distance. In any case, the projected image (e.g., a lidar range image) and / or other reflection data may be stored and / or encoded into a suitable representation (e.g., input data 406 ), which may serve as input to the machine learning model 408 .

[0048] Figure 5 4 is a data flow diagram illustrating an example process for preprocessing 404 sensor data 402 for a machine learning model 408 in an object detection system according to some embodiments of the present disclosure. The sensor data 402 may be accumulated 510 (which may include transformation to a single coordinate system), autonomous motion compensated 520, and / or encoded 530 into a suitable representation such as a projected image (e.g., a lidar range image) and / or a tensor, for example, with multiple channels storing different reflectance features.

[0049] More specifically, sensor data 402, such as lidar data, may be accumulated 510 from multiple sensors (e.g., some or all of multiple surrounding lidar sensors 1664) from different locations of the autonomous vehicle 1600 and may be transformed to a single vehicle coordinate system (e.g., centered on the vehicle). Additionally or alternatively, sensor data 402 may be accumulated 510 over time to increase the density of the accumulated sensor data. Sensor detections may be accumulated over any desired time window (e.g., 0.5 seconds, 1 second, 2 seconds, etc.). The size of the window may be selected based on the sensor and / or application (e.g., a smaller window may be selected for noisy applications such as highway scenarios). In this way, each input in the machine learning model 408 may be generated from the accumulated detections of each time window of the rolling window (e.g., spanning from the t window size to the current duration). Each window to be evaluated may be incremented by any suitable step size, which may but need not correspond to the window size. Thus, each consecutive input to the machine learning model 408 may be based on consecutive windows, which may but need not overlap.

[0050] In some embodiments, autonomous motion compensation 520 can be applied to sensor data 402. For example, accumulated detections can be autonomous motion compensated to the latest known vehicle position. More specifically, the position of old detections can be propagated to the latest known position of the moving vehicle, using the known motion of the vehicle to estimate the position of the old detections at a desired point in time (e.g., relative to the current position of the vehicle). The result can be a set of accumulated, autonomous motion compensated sensor data 402 (e.g., a lidar point cloud) for a particular time slice.

[0051] In some embodiments, the (accumulated, autonomous motion compensated) sensor data 402 may be encoded into an appropriate representation, such as a projected image, which may include multiple channels storing different features such as reflective features. More specifically, the accumulated autonomous motion compensated detections may be projected to form a projected image of a desired size (e.g., spatial dimension). Any desired view of the environment may be selected for the projected image, such as a top view, a front view, a perspective view, and / or other views. In one example, a lidar point cloud (e.g., spherical, cylindrical, pinhole) may be projected to form a lidar range image with an environmental perspective view, and the lidar range image may be used as input data 406 to a machine learning model 408. In some embodiments, images with the same or different views may be generated, with each image being input into a separate channel of the machine learning model 408. As a non-limiting example, different sensors 401 (whether of the same type or different sensors) may be used to generate image data (e.g., lidar range images, camera images, etc.) with the same (e.g., perspective) environmental view in a common image space, and image data from different sensors 401 or sensor modes may be stored in separate channels of a tensor. These are meant only as examples, and other variations may be implemented within the scope of the present disclosure.

[0052] Because image data may be evaluated as input to machine learning model 408, there may be a tradeoff between prediction accuracy and computational requirements. As such, a desired spatial size of the projected image may be selected as a design choice. Additionally or alternatively, to reduce data loss due to lower image resolution, the size of the projected image may be based on characteristics of the corresponding sensor 401 that captured sensor data 402. As a non-limiting example, the height (or vertical resolution) of the lidar range image may be set to correspond to the number of horizontal scan lines of the sensor that captured sensor data 402 (e.g., a row of pixels in the range image for each scan line of the corresponding lidar sensor), and the width (or horizontal resolution) of the lidar range image may be set based on the horizontal resolution of sensor 401 that captured sensor data 402.

[0053] In some embodiments, the projected image may include multiple layers, and the pixel values ​​of different layers store different reflection characteristics. In some embodiments, for each pixel storing sensor data representing multiple reflections, a set of features can be calculated, determined, or otherwise selected from the reflection characteristics of the reflections (e.g., azimuth, azimuth, elevation, range, intensity, reflectivity, SNR, etc.). In some cases, when sensor data representing multiple reflections are combined in a pixel of a projected image (e.g., a range image), sensor data representing one of the reflections (e.g., a reflection with the closest range) can be represented in the projected image, and sensor data representing other reflections can be discarded. For example, in a range image with a pixel storing multiple reflections together, the pixel can store a range value corresponding to the reflection with the closest range. Additionally or alternatively, when multiple reflections are combined together in a pixel to form a point tower, a specific feature of the pixel can be calculated by clustering the corresponding reflection features of the multiple overlapping reflections (e.g., using standard deviation, average, etc.). In general, any given pixel can have multiple associated feature values, which can be stored in corresponding channels of a tensor. In any case, the sensor data 402 can be encoded 530 into various types of input data 406 (e.g., such as a projected image of a lidar range image, a tensor encoding the projected image, and corresponding reflectance features), and the input data 406 can be used as input into a machine learning model 408.

[0054] At a high level, machine learning model 408 can detect objects such as obstacle instances, static portions of the environment, and / or other objects represented in input data 406 (e.g., lidar range images, camera images, and / or other sensor data stacked into corresponding channels of the input tensor). For example, machine learning model 408 can extract classification data (e.g., classification confidence data 410) and / or object instance data, such as location, geometry, and / or orientation data (e.g., instance regression data 412) representing objects detected in a 3D environment. The classification data and object instance data can be post-processed 414 to generate classification labels and 2D and / or 3D bounding boxes, closed polylines, or other bounding shapes to identify the location, geometry, and / or orientation of the detected object instances.

[0055] In some embodiments, the machine learning model 408 can be implemented using a DNN, such as a convolutional neural network (CNN). Although certain embodiments are described with a machine learning model 408 implemented using a neural network (and specifically a CNN), this is not intended to be limiting. For example, and without limitation, the machine learning model 408 may include any type of machine learning model, such as a machine learning model using linear regression, logistic regression, decision tree, support vector machine (SVM), naive Bayes, k-nearest neighbor (Knn), K-means clustering, random forest, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolution, recursion, perceptrons, long / short term memory (LSTM), Hopfield, Boltzmann, deep belief, deconvolution, generative adversarial, liquid machines, etc.) and / or other types of machine learning models.

[0056] In some embodiments, the machine learning model 408 may include a common backbone (or laminar stream) having multiple heads (or at least partially discrete laminar streams) for predicting different outputs based on the input data 406. For example, the machine learning model 408 may include, but is not limited to, a feature extractor (e.g., a DNN, an encoder / decoder, etc.), including convolutional layers, pooling layers, and / or other layer types, wherein the output of the feature extractor is provided as input to a first head for predicting classification data, and a second head for predicting the position, geometry, and / or orientation of a detected object. In some examples, the first head and the second head may receive parallel inputs and thus may produce different outputs from similar input data.

[0057] In general, the machine learning model 408 may include a feature extractor configured to evaluate images with different views of a 3D environment. For example, the machine learning model 408 may include separate feature extractors in multiple stages linked together to sequentially process data from multiple views of the 3D environment. For example, the machine learning model 408 may include a first stage having a first feature extractor configured to extract classification data from an image with a first view of the environment (e.g., a perspective view), and the output of the first feature extractor may be converted to a second view of the environment (e.g., a top-down view) and input into a second feature extractor, which may include a common backbone and multiple heads that extract different outputs for detected objects, such as classification data and object instance data. Additionally or alternatively, multiple images with different views may be generated, each of which may be fed into a separate side-by-side feature extractor, and the latent space tensors output by the separate feature extractors may be combined to form classification data and / or object instance data. These architectures are meant as examples, and other architectures are contemplated within the scope of the present disclosure.

[0058] Now go to Figure 6 , Figure 6 is an exemplary multi-view-aware machine learning model according to some embodiments of the present disclosure (e.g., Figure 4 408 of the machine learning model 408). Figure 6 In the example of , the machine learning model 408 is illustrated with an example architecture including multiple stages linked together for sequential processing. In the first stage, the encoder / decoder 605 can extract class confidence data 610 (e.g., one or more confidence maps in a first view, such as a perspective view) from the input data 406, and can transform the classification confidence data 610 into a second view to form transformed classification confidence data 630 (e.g., one or more confidence maps in a second view such as a top view). This data can be supplemented by geometric data 640 of the object represented in the sensor data 402 (e.g., representing the geometric dimensions of the object in a direction orthogonal to two dimensions of the second view, such as height). The converted classification confidence data 630 (e.g., one or more confidence maps in the overhead view) and / or geometry data 640 (e.g., one or more height maps) can be encoded 645 (e.g., stacked into corresponding channels of a tensor) and fed to a second stage, which includes an encoder / decoder backbone 650 connected to a classification confidence head 655 and an instance regression head 660, which can extract the classification confidence data 610 (e.g., one or more confidence maps in the overhead view) and instance regression data 412 (e.g., object instance data, such as position, geometry, and / or orientation).

[0059] The encoder / decoder 605 can be implemented using encoder and decoder components with skip connections (e.g., similar to feature pyramid networks, U-Net, etc.). For example, the encoder / decoder 605 can accept input data 406, such as a lidar range image and / or an RBG image, and can apply various convolutions, pooling, and / or other types of operations to extract classification confidence data 610 for any number of supported classifications. In an example implementation, the encoder / decoder 605 can include an encoding (contraction) path and a decoding (expansion) path. Along the contraction path, each resolution can include any number of layers (e.g., convolutions, expansion convolutions, initiation blocks, etc.) and downsampling operations (e.g., max pooling). Along the expansion path, each resolution can include any number of layers (e.g., deconvolution, upsampling followed by convolution, and / or other types of operations). In the expansion path, each resolution of the feature map can be upsampled and connected to a feature map of the same resolution from the contraction path (e.g., in the depth dimension). In this example, the corresponding resolutions of the contraction path and the expansion path can be connected with a skip connection, which can be used to add or connect feature maps from the corresponding resolution.

[0060] The output of the encoder / decoder 605 can be classification confidence data 610 for any number of supported classifications (e.g., one channel for each classification). Examples of supported classifications can include vehicles (e.g., cars, buses, trucks, etc.), vulnerable road users (e.g., motorcycles, bicycles, pedestrians, etc.), environmental parts (e.g., drivable spaces, sidewalks, buildings, trees, utility poles, etc.), subclasses thereof (e.g., walking pedestrians), some combination thereof, and / or others. For example, the classification confidence data 610 can include representations of one or more confidence maps (e.g., one for each classification). As a non-limiting example, the encoder / decoder 605 can output a tensor having N channels corresponding to N classifications (e.g., one confidence map for each channel). Thus, each pixel in the tensor can store a depth-direction pixel value representing the probability, score, or logic that the pixel is part of the corresponding classification for each channel. In some embodiments, the sum of the depth-direction pixel values ​​can be normalized to a certain value (e.g., 1). In some embodiments, the predictions can be used to generate different classification segmentation masks for each class (channel), and / or can be collapsed into a single composite segmentation mask where each pixel contains a classification label (e.g., represented by a different integer). Figure 6 In the illustrated embodiment, the encoder / decoder 605 outputs classification confidence data 610 (e.g., one or more confidence maps), however, in other embodiments, the encoder / decoder 605 may additionally or alternatively output other types of classification data (e.g., N classification segmentation masks storing a binary value for each pixel, a composite segmentation mask storing the most likely classification label for each pixel).

[0061] In general, the output of the encoder / decoder 605 (e.g., classification confidence data 610 or other classification data) can be converted from a first view of the environment to a second view. For example, the classification data extracted by the encoder / decoder 605 can be used to mark 620 corresponding 3D locations in the environment, and the marked 3D locations can be projected 625 into the second view.

[0062] Taking a confidence map for a particular classification as an example, the confidence map can have spatial dimensions corresponding to the input to the encoder / decoder 605 (e.g., a lidar range image), and the confidence map can include a classification value (e.g., probability, score, or logic) for each pixel. In some cases, the classification values ​​can be mapped to known 3D locations identified by the corresponding sensor data 402 and / or input data 406. For example, the corresponding input lidar range image can have a known correspondence between range scan pixels and corresponding points in a lidar point cloud (lidar detection), which can have a known 3D location. Therefore, the classification value from the predicted confidence map can be associated with the 3D location of the lidar detection represented by the corresponding range scan pixel in the input lidar range image.

[0063] In another example, assume that the input to the encoder / decoder 605 includes a representation of an RGB image generated by a camera, and the encoder / decoder 605 classifies each pixel of the RGB image by generating one or more classification values ​​for each pixel. The classification value can be associated with a 3D position identified from other sensor data such as lidar or radar detection, or associated with a 3D position from a 3D representation of the environment (e.g., a 3D map of the environment). For example, lidar or radar data captured in the same time slice as the input RGB image can be projected to form a range image with the same view as the input RGB image. In this case, by identifying the 3D position of the sensor detection represented by the corresponding range scan pixel in the range image (e.g., a point in a point cloud), the classification value from the prediction confidence map can be associated with the 3D position in a manner similar to the previous example. In another example, image data generated by a sensor with a known direction and position in a 3D representation of the environment (e.g., an RGB image generated by a camera) may not be projected into the world space to identify the 3D position of the object corresponding to each pixel in the world space. These are just a few examples, and other variations may be implemented within the scope of the present disclosure.

[0064] Thus, 3D positions from sensor data (e.g., sensor data 402) of the corresponding time slice or from the corresponding portion of the 3D representation of the environment can be labeled 620 with classification data (e.g., classification value, label) extracted by the encoder / decoder 605. The labeled 3D positions can be projected 625 into a second view of the environment, for example, by orthogonally projecting the labeled 3D positions to form a projected image with a desired spatial size and ground sampling distance (e.g., a top-down view image with a top-down view). In one example implementation, the semantically labeled range image can be converted to a top-down representation.

[0065] In some cases, projecting 625 the marked 3D position may simply include flattening the marked 3D position into a plane, or otherwise (e.g., orthogonally) projecting the marked 3D position into the projection image (e.g., flattening the point cloud into a plane by discarding the z-value of the marked point). In some cases, projecting 625 the marked 3D position may include grouping together multiple points in a pixel of the projection image. Thus, when grouping together multiple points in a pixel of the projection image (e.g., a top-down image), any technique may be used to select or otherwise represent one or more points (e.g., selecting one point such as the highest point or the lowest point). In some cases, filtering may be employed (e.g., omitting points above 3 or 4 meters). Each pixel of the generated projection image where the projected 3D position is located may store the extracted classification data, where the 3D position is labeled (e.g., a classification value or label).

[0066] Thus, the result can be transformed classification data (e.g., a transformed confidence map or segmentation mask) representing the classification data extracted in the second view of the environment. In general, any number of confidence maps (e.g., storing N confidences for the classification value of each pixel in the corresponding class), single classification segmentation masks (e.g., storing N classification segmentation masks indicating binary values ​​for each pixel in the corresponding class), and / or composite segmentation masks (e.g., a single channel storing the label of the most likely classification for each pixel) can be transformed 615. If the transformed classification data has N channels, the transformed classification data can have N corresponding channels. Figure 6 In the illustrated embodiment, classification confidence data 610 (e.g., one or more confidence maps in a perspective view) may be transformed to form transformed classification confidence data 630 (e.g., one or more confidence maps in a top view). However, in other embodiments, other types of classification data (e.g., N-class segmentation masks storing binary values ​​for each pixel, composite segmentation masks storing the most likely classification label for each pixel) may additionally or alternatively be transformed 615.

[0067] Typically, the converted classified data 630 can represent a second view of the environment, and thus the converted classified data can reveal object features (e.g., position, geometry, orientation) in two dimensions represented by the second view. In some embodiments, geometric data 640 representing object features (e.g., orthogonal to the second view) can be generated 635 from sensor data (e.g., sensor data 402) corresponding to a time slice. For example, sensor data representing the 3D position of an object detected in the environment can be sampled or otherwise processed to represent features of the detected object in a particular dimension (e.g., an orthogonal dimension), for example, by taking one or more slices of the sensor data in the particular dimension. Taking a lidar or radar point cloud where the second view is a top-down view as an example, one or more slices of the point cloud can be taken in a high dimension to generate geometric data 640 having a planar representation of the detected object, such as a minimum and maximum height map. In this example, each pixel of such a map can represent a column in the top-down view. If sensor data representing multiple objects in a 3D environment are grouped together in a single pixel (e.g., multiple points in a point cloud are located in a column represented by a pixel), any type of sampling or statistical metric can be used to represent the collection of data (e.g., points in a column). For example, each pixel can store the minimum height of all points in the column (e.g., a minimum height map), the maximum height of all points in the column (e.g., a maximum height map), the median height of all points in the column, the average height of all points in the column, the variance of the heights of all points in the column, and / or the like. In this way, one or more slices 635 of geometric data 640 (e.g., height data) having a planar representation of objects detected in the environment can be generated.

[0068] In another example, geometric data 640 for objects in an environment can be generated from image data (e.g., RGB images) 635 generated by a sensor (e.g., a camera). For example, the known orientation and position of the sensor that captured the image data can be used to back-project the image data into a 3D representation of the environment (e.g., a 3D map or some other world space) and identify the 3D position of the object in the world space corresponding to each pixel. In this case, one or more slices of the identified 3D positions can be used to generate 635 geometric data 640 (one or more height maps).

[0069] In general, the transformed classification data (e.g., transformed classification confidence data 630) and / or geometric data (e.g., geometry data 640) can be encoded 645 or otherwise organized into some suitable representation for the encoder / decoder backbone 650. For example, in an embodiment where the transformed classification confidence data 630 includes N transformed confidence maps and the geometry data 640 includes M height maps, for the N confidence maps and the M height maps, the transformed classification confidence data 630 and the geometry data 640 can be encoded 645 into a tensor having N channels. This is just one example, and any suitable representation of the transformed classification data and / or geometry data can be implemented.

[0070] Figure 6 The second stage of the machine learning model 408 includes an encoder / decoder backbone 650, a classification confidence head 655, and an instance regression head 660. The second stage can extract features from the transformed representation of classification data and / or geometry data (e.g., a tensor with M+N channels) and can perform classification segmentation and / or regress instance geometry in the second view.

[0071] The encoder / decoder backbone 650 may be implemented using encoder and decoder components with skip connections (e.g., similar to a feature pyramid network, U-Net, etc.). For example, the encoder / decoder backbone 650 may accept transformed representations of classification confidence data 630 and / or geometry data 640 and apply various convolutions, pooling, and / or other types of operations to extract features into some latent space. Figure 6 , an encoder / decoder backbone 650 is illustrated using an example implementation involving an encoding (contracting) path and an example decoding (dilation) path. Along the contracting path, each resolution may include any number of layers (e.g., convolutions, dilated convolutions, inception blocks, etc.) and downsampling operations (e.g., max pooling). Along the dilation path, each resolution may include any number of layers (e.g., deconvolutions, upsampling followed by convolutions, and / or other types of operations). In the dilation path, each resolution of the feature map may be upsampled and concatenated with a feature map of the same resolution from the contracting path (e.g., in the depth dimension). In this example, the corresponding resolutions of the contracting and dilation paths may be connected with skip connections, which may be used to add or concatenate feature maps from corresponding resolutions. In this way, the encoder / decoder backbone 650 may extract features into some latent space tensors, which may be input into the classification confidence head 655 and the instance regression head 660.

[0072] The classification confidence head 655 may include any number of layers 655A, 655B, 655C (e.g., convolutions, pooling, classifiers such as softmax transfer functions, and / or other types of operations, etc.) that predict classification data from the output of the encoder / decoder backbone 650. For example, the classification confidence head 655 may include a channel (e.g., a layer flow plus a classifier) ​​for each class of objects to be detected (e.g., vehicles, cars, trucks, vulnerable road users, pedestrians, bicycles, motorcycles, drivable spaces, sidewalks, buildings, trees, utility poles, subclasses thereof, some combination thereof, etc.), so that the classification confidence head 655 extracts the classification data (e.g., classification confidence data 410) in any suitable form. For example, the classification confidence head 655 may predict a confidence map representing the inferred confidence of whether a particular object is present (regardless of the class), a separate confidence map for each class, etc. In some embodiments, the classification confidence data 410 predicted by the classification confidence head 655 can take the form of a multi-channel tensor, where each channel can be considered as a heat map storing the classification value (e.g., probability, score, or logic) that each pixel belongs to the classification corresponding to the channel.

[0073] The instance regression head 660 may include any number of layers 660A, 660B, 660C (e.g., convolutions, pooling, classifiers such as softmax transfer functions, and / or other types of operations, etc.) that predict object instance data (e.g., the location, geometry, and / or orientation of detected objects) from the output of the encoder / decoder backbone 650. The instance regression head 660 may include N channels (e.g., layers plus classifiers), where each channel regresses a specific type of information about the detected object instance, such as the location of the object (e.g., a dx / dy vector pointing to the center of the object), the object height, the object width, the object orientation (e.g., a rotation angle such as sine and / or cosine), some statistical measure thereof (e.g., minimum, maximum, mean, median, variance, etc.), and / or the like. As a non-limiting example, the instance regression head 660 may include separate dimensions that identify the x-dimension of the center of the detected object, the y-dimension of the center of the detected object, the width of the detected object, the height of the detected object (e.g., displacement from the ground), the sine of the direction of the detected object (e.g., the rotation angle in the 2D image space), the cosine of the direction of the detected object, and / or other types of information. These types of object instance data are merely examples, and other types of object information may be regressed in addition or alternatively. The instance regression head 660 may include a separate regression channel for each classification, or a set of channels for all classifications. In some embodiments, the instance regression data 412 predicted by the instance regression head 660 may be in the form of a multi-channel tensor, where each channel may include a floating point number that regresses a specific type of object information (e.g., a specific object dimension).

[0074] Thus, the machine learning model 408 can predict multi-channel classification data (e.g., classification confidence data 410) and / or multi-channel object instance data (e.g., instance regression data 412) from a particular input (e.g., input data 406). Some possible training methods are described in more detail below. In operation, the output of the machine learning model 408 can be post-processed (e.g., decoded) to generate bounding boxes, closed polylines, or other bounding shapes to identify the location, geometry, and / or orientation of detected object instances. For example, when the machine learning model 408 predicts classification confidence data 410 and / or instance regression data 412 for a particular view of the environment (e.g., a top-down view), a bounding box, closed polyline, or other bounding shape can be identified relative to that view (e.g., in the same image space as the input to the second stage of the machine learning model 408). In some embodiments, because the object instance data may be noisy and / or may produce multiple options, non-maximum suppression, density-based spatial clustering of noisy applications (DBSCAN), and / or another function can be used to generate the bounding shape.

[0075] Figure 7 4 is a data flow diagram illustrating an example post-processing process 414 for generating object detections 416 in an object detection system according to some embodiments of the present disclosure. In this example, the post-processing process 414 includes instance decoding 710 and filtering and / or clustering 720. In general, instance decoding 710 can identify 2D and / or 3D candidate bounding boxes (or other bounding shapes) based on object instance data (e.g., position, geometry, and / or orientation data) from corresponding channels of instance regression data 412 and / or confidence maps or masks from corresponding channels of classification data (e.g., class confidence data 410) of the classification. More specifically, the predicted confidence map and the predicted object instance data can specify information about the detected object instance, such as the position of the object, the height of the object, the width of the object, the orientation of the object, etc. This information can be used to identify candidate object detections (e.g., candidate objects with unique center points, object heights, object widths, object orientations, etc.). The result can be a set of two-dimensional and / or three-dimensional candidate bounding boxes (or other bounding shapes) for each object classification.

[0076] Various types of filtering and / or clustering 720 can be used to remove duplication and / or noise from the candidate bounding boxes (or other bounding shapes) for each object classification. For example, in some embodiments, non-maximum suppression can be used to remove duplication. Non-maximum suppression can be used if two or more candidate bounding boxes have associated confidence values ​​indicating that the candidate bounding boxes may correspond to the same object instance. In such an example, the confidence value that is the highest for the object instance can be used to determine which candidate bounding box to use for the object instance, and non-maximum suppression can be used to remove or suppress other candidates.

[0077] For example, each candidate bounding box (or other bounding shape) can be associated with a corresponding confidence / probability value associated with one or more corresponding pixels of the corresponding channel of the classification confidence data 410 from the evaluated classification (e.g., using the confidence / probability value of a representative pixel such as a center pixel, using an average or some other composite value calculated over the candidate region, etc.). Thus, candidate bounding shapes whose confidence / probability of being a member of an object classification is less than a certain threshold (e.g., 50%) can be filtered out. Additionally or alternatively, the candidate bounding box (or other shape) with the highest confidence / probability score for a particular classification can be assigned an instance ID, a metric such as intersection over union (IoU) can be calculated for each of the other candidates in the classification, and candidates with an IoU above a certain threshold can be filtered out to remove duplicates. The process can be repeated, assigning an instance ID to the candidate with the next highest confidence / probability score, removing duplicates, and repeating until no more candidates remain. The process can be repeated for each other classification to remove duplicate candidates.

[0078] In some embodiments, a clustering method such as density-based spatial clustering of noisy applications (DBSCAN) can be used to remove duplicate candidate boundary shapes. For example, the candidate boundary shapes can be clustered (e.g., the centers of the candidate boundary shapes can be clustered), it can be determined that the candidate objects in each cluster correspond to the same object instance, and duplicate candidate objects can be removed from each cluster.

[0079] Thus, the extracted classification data and / or object instance data may be decoded (e.g., via instance decoding 710), filtered, and / or clustered (e.g., via filtering and / or clustering 720) to identify bounding boxes, closed polylines, or other bounding shapes for objects detected in each particular classification (e.g., based on data from corresponding channels of classification confidence data 410 and instance regression data 412). A classification label may be applied to each identified bounding shape based on the particular classification being evaluated (e.g., based on a known mapping between channels and classification labels). In some cases, 2D bounding shapes may be determined first, and a third dimension (e.g., height) of each 2D bounding shape may be inferred from the extracted object instance data (e.g., based on a particular regression channel for the object's height, a separate regression channel based on minimum and maximum values ​​in the height dimension, etc.). Thus, 2D and / or 3D bounding shapes and classification labels may be identified for objects detected in the environment.

[0080] In summary, the machine learning model 408 can accept a representation of sensor data having a first view, such as a lidar range image, perform segmentation on the representation of the sensor data to extract classification data, convert the classification data to a representation having a second view, and perform segmentation and instance regression on the second view to extract classification data and / or object instance data. The extracted classification data and / or object instance data can be post-processed to generate classification labels and 2D and / or 3D bounding boxes, closed polylines, or other bounding shapes to identify the location, size, and / or orientation of object instances detected in the projection image.

[0081] Figure 8 is a diagram of an example data flow through an example multi-view-aware machine learning model according to some embodiments of the present disclosure. Figure 8 , the lidar range image 810 is input to the first stage of the neural network (e.g., Figure 6 The neural network segments the lidar range image to generate a segmented lidar range image 820. The segmented lidar range image 820 is converted to a top-down view 830, stacked with the height data, and passed through a second stage of the neural network (e.g., Figure 6 The decoder is fed into the encoder / decoder backbone 650, the classification confidence head 655, and the instance regression head 660 of the segmented lidar range image 820. Note that the classification regions (e.g., drivable space 825) of the segmented lidar range image 820 have been transformed into corresponding regions in the overhead view 830 (e.g., transformed drivable space 835). The second stage of the neural network extracts the classification data and object instance data and post-processes them to generate bounding boxes for the detected objects.

[0082] Once the location, geometry, orientation, and / or classification label of the object instance is determined, the 2D pixel coordinates defining the object instance can be converted to 3D world coordinates for use with the corresponding classification label by the autonomous vehicle when performing one or more operations (e.g., obstacle avoidance, lane keeping, lane changing, path planning, mapping, etc.). In some embodiments, a low-level lidar perception stack that does not use a DNN can process sensor data to detect objects in parallel with the machine learning model 408 (e.g., for redundancy). Therefore, returning to Figure 4 , object detection 416 (e.g., bounding box, closed polyline, or other bounding shape) can be performed by Figures 16A-16D The control components of the illustrated autonomous vehicle 1600 use, for example, an autonomous driving software stack 422 executing on one or more components (e.g., SoC 1604, CPU 1618, GPU 1620, etc.) of the vehicle 1600. For example, the vehicle 1600 may use the information (e.g., obstacle instances) to navigate, plan, or otherwise perform one or more maneuvers (e.g., obstacle avoidance, lane keeping, lane changing, merging, splitting, etc.) in the environment.

[0083] In some embodiments, object detection 416 can be used by one or more layers of an autonomous driver software stack 422 (also referred to herein as "driver stack 422"). Driver stack 422 can include a sensor manager (not shown), a perception component (e.g., corresponding to a perception layer of driver stack 422), a world model manager 426, a planning component 428 (e.g., corresponding to a planning layer of driver stack 422), a control component 430 (e.g., corresponding to a control layer of driver stack 422), an obstacle avoidance component 432 (e.g., corresponding to an obstacle avoidance or collision avoidance layer of driver stack 422), a driver component 434 (e.g., corresponding to a driver layer of driver stack 422), and / or other components, corresponding to additional and / or alternative layers of driver stack 422. In some examples, process 400 can be performed by a perception component, which can feed layers of driver stack 422 to a world model manager, as described in more detail herein.

[0084] The sensor manager can manage and / or extract sensor data 402 from sensors of the vehicle 1600. For example, and with reference to Fig. 16C, sensor data 402 may be generated (e.g., continuously, at intervals, based on certain conditions) by radar sensor 1660. The sensor manager may receive sensor data 402 from sensors in different formats (e.g., sensors of the same type may output sensor data in different formats), and may be configured to convert the different formats to a unified format (e.g., for each sensor of the same type). As a result, other components, features, and / or functions of autonomous vehicle 1600 may use the unified format, thereby simplifying the processing of sensor data 402. In some examples, the sensor manager may apply controls back to the sensors of vehicle 1600 using the unified format, such as setting a frame rate or performing gain control. The sensor manager may also update sensor packets or communications corresponding to the sensor data with a timestamp to help inform the processing of the sensor data by various components, features, and functions of the autonomous vehicle control system.

[0085] The world model manager 426 can be used to generate, update and / or define a world model. The world model manager 426 can use information generated and received by the perception components of the drive stack 422 (e.g., the location of detected obstacles). The perception components may include obstacle sensors, path sensors, waiting sensors, map sensors, and / or other perception components. For example, the world model can be defined at least in part based on the supply of obstacles, paths, and waiting conditions perceived by obstacle sensors, path sensors, waiting sensors, and / or map sensors in real time or near real time. The world model manager 426 can continuously update the world model based on newly generated and / or received inputs (e.g., data) from obstacle receivers, path receivers, waiting receivers, map receivers, and / or other components of the autonomous vehicle control system.

[0086] The world model can be used to help inform the planning component 428, control component 430, obstacle avoidance component 432, and / or actuation component 434 of the actuation stack 422. The obstacle perception can perform obstacle perception, which can be based on the locations where the vehicle 1600 is allowed to drive or is able to drive (e.g., based on the locations of a drivable path defined by avoiding detected obstacles), and the speed at which the vehicle 1600 can travel without colliding with obstacles (e.g., objects such as structures, entities, vehicles, etc.) sensed by sensors of the vehicle 1600 and / or the machine learning model 408.

[0087] The path receiver can perform path perception, such as by sensing a nominal path available in a particular situation. In some examples, the path sensor can further take into account changes in lanes to perceive the path. The lane map can represent one or more paths available to the vehicle 1600 and can be as simple as a single path on a highway entrance ramp. In some examples, the lane map can include a path to a desired lane and / or can indicate available changes along a highway (or other road type), or can include nearby lanes, lane changes, forks, turns, grade separations, merges, and / or other information.

[0088] The waiting receiver may be responsible for determining constraints on the vehicle 1600 based on rules, conventions, and / or practical considerations. For example, the rules, conventions, and / or practical considerations may relate to traffic lights, multi-way stops, yields, merges, toll booths, gates, emergency personnel, road workers, parked buses or other vehicles, one-way bridge arbitration, ferries, etc. Thus, the waiting sensor may be utilized to identify potential obstacles and implement one or more controls (e.g., slow down, stop, etc.) that may not be possible relying solely on the obstacle receiver.

[0089] The map receiver may include mechanisms to recognize behavior, and in some examples, to determine a particular example of what conventions apply in a particular area. For example, the map sensor may determine based on data representing previous driving or travel that at a particular intersection, there are no U-turns between certain times, that electronic signs indicating lane directions change over time, that two traffic lights that are very close together (e.g., with little offset from each other) are associated with different roads, that in Rhode Island, the first car to turn left at a traffic light is violating the law because it would be turning in front of oncoming vehicles when the light is green, and / or other information. The map receiver may notify the vehicle 1600 of static or stationary infrastructure objects and obstacles. The map receiver may also generate information for a waiting receiver and / or a path receiver, for example, to determine which light at an intersection must be green in order for the vehicle 1600 to take a particular path.

[0090] In some examples, information from a map receiver may be sent, transmitted, and / or provided to a server (e.g., Fig.16DThe map manager may include a map receiver and / or a location manager of the vehicle 1600, and may send, transmit, and / or provide information from the server to a map receiver and / or a location manager of the vehicle 1600. The map manager may include a cloud map application that is remotely located from the vehicle 1600 and can be accessed by the vehicle 1600 via one or more networks. For example, the map receiver and / or location manager of the vehicle 1600 may communicate with one or more other components or features of the map manager and / or server to inform the map sensor and / or local manager of the 1600 vehicle's past and present drives or trips, as well as the past and present drives or trips of other vehicles. The map manager may provide map outputs (e.g., map data), which may be localized by the localization manager based on the vehicle's specific location 1600, and the localized map outputs may be used by the world model manager 426 to generate and / or update the world model.

[0091] The planning component 428 may include components, features, and / or functions such as a route planner, a lane planner, a behavior planner, and a behavior selector. The route planner may use information from a map receiver, a map manager, and / or a location manager, among other information, to generate a planned path that may include GNSS waypoints (e.g., GPS waypoints), 3D world coordinates (e.g., Cartesian coordinates, polar coordinates, etc.), coordinates relative to an origin 1600 on the vehicle, and the like. These waypoints may represent a specific distance in the future of the vehicle 1600, such as city blocks, kilometers, feet, inches, miles, etc., which may be the subject of a lane planner.

[0092] The lane planner may use as input a lane map (e.g., a lane map from a path sensor), object poses in the lane map (e.g., according to a localization manager), and / or object points and directions to future distances from a path planner. The object point and direction may be mapped to the best matching driving point and direction in the lane map (e.g., based on GNSS and / or compass directions). A graph search algorithm may then be performed on the lane map from the current edge in the lane map to find the shortest path to the object point.

[0093] The behavior planner can determine the feasibility of basic behaviors of the vehicle 1600, such as staying in the lane or changing lanes to the left or right, so that these feasible behaviors can be matched with the most desired behavior output by the lane planner. For example, if it is determined that the desired behavior is unsafe and / or unavailable, a default behavior can be selected instead (e.g., the default behavior may be to stay in the lane when the desired behavior or changing lanes is unsafe).

[0094] The control component 430 can follow as closely as possible the trajectory or path (lateral and longitudinal) received from the behavior selector of the planning component 428 (e.g., based on the object detection 416) within the capabilities of the vehicle 1600. The control component 430 can use tight feedback to handle unexpected events or behaviors that are not modeled and / or anything that causes a deviation from the ideal (e.g., unexpected delays). In some examples, the control component 430 can use a forward prediction model that takes controls as input variables and generates predictions that can be compared to a desired state (e.g., compared to a desired lateral and longitudinal path requested by the planning component 428). Control measures can be determined to minimize the difference.

[0095] Although planning component 428 and control component 430 are shown separately, this is not intended to be limiting. For example, in some embodiments, the delineation between planning component 428 and control component 430 may not be precisely defined. Therefore, at least some components, features, and / or functions attributed to planning component 428 may be associated with control component 430, and vice versa. This may also be true for any individually illustrated component of driver stack 422.

[0096] The obstacle avoidance component 432 can help the autonomous vehicle 1600 avoid collisions with objects (e.g., moving and stationary objects). The obstacle avoidance component 432 may include a computational mechanism at a "primitive level" of obstacle avoidance and may act as a "survival brain" or "reptilian brain" for the vehicle 1600. In some examples, the obstacle avoidance component 432 may be used independently of the components, features, and / or functions of the vehicle 1600, which require compliance with traffic laws and courteous driving. In these examples, in order to ensure that no collision occurs between the vehicle 1600 and any object, the obstacle avoidance component may ignore traffic laws, road rules, and courteous driving norms. Therefore, the obstacle avoidance layer may be an independent layer outside of the road layer rules, and the obstacle avoidance layer may ensure that the vehicle 1600 performs safe operations only from an obstacle avoidance perspective. On the other hand, the rules of the road layer may ensure that the vehicle complies with traffic laws and practices and abides by legal and conventional rights of way (as described herein).

[0097] In some examples, obstacle avoidance component 432 can use drivable path and / or object detection 416 to determine the control or action to take. For example, drivable path can provide an indication to obstacle avoidance component 432 that vehicle 1600 can maneuver without striking any objects, structures, and / or the like, or at least without the presence of static structures.

[0098] In a non-limiting embodiment, the obstacle avoidance component 432 can be implemented as an independent, discrete feature of the vehicle 1600. For example, the obstacle avoidance component 432 can operate independently (e.g., in parallel with, before, and / or after the planning layer, control layer, drive layer, and / or other layers of the drive stack 422).

[0099] Thus, the vehicle 1600 can use this information (e.g., as edges or tracks of a path) to navigate, plan, or otherwise perform one or more maneuvers (e.g., lane keeping, lane changing, merging, splitting, etc.) in the environment.

[0100] Now refer to Figure 9-11 , each block of methods 900, 1000 and 1100 described herein includes a computing process that can be performed using any combination of hardware, firmware and / or software. For example, instructions stored in a memory can be executed by a processor to perform various functions. The method can also be implemented as a computer-usable instruction stored on a computer storage medium. These methods can be provided by a stand-alone application, a service or a hosted service (standalone or in conjunction with another hosted service), or a plug-in for another product. In addition, by way of example, methods 900, 1000 and 1100 are described with respect to the object detection system described herein. However, these methods can be performed additionally or alternatively by any one system or any combination of systems, including but not limited to the systems described herein.

[0101] Fig. 9 9 is a flow chart illustrating a method 900 for multi-view object detection using sensor data according to some embodiments of the present invention. At block B902, the method 900 includes determining first data representing a first view of an environment based on sensor data from at least one sensor in the environment. For example, Figure 4 Sensor 401 – For example Figures 16A-16D The lidar sensor 1664 of the autonomous vehicle 1600 in FIG. 401 can be used to generate sensor data 402 representing objects in the 3D environment surrounding the sensor 401. The sensor data 402 can be pre-processed 404 to form a projected image of a desired size (e.g., spatial dimension). For example, the (accumulated, autonomous motion compensated) lidar point cloud can be projected to form a lidar range image with perspective. The projected image (e.g., lidar range image) and / or other reflection data can be stored and / or encoded as an appropriate representation (e.g., input data 406) of the machine learning model 408.

[0102] At block B904, method 900 includes extracting, using one or more neural networks (NNs), classification data representing one or more classifications based at least on the first view of the first data. Figure 4 The input data 406 may be used as input to a machine learning model 408, which may include multiple component machine learning models or stages linked together that sequentially process different views of the 3D environment. In an exemplary first stage, Figure 6The encoder / decoder 605 can extract classification data (e.g., one or more confidence maps, one or more segmentation masks) from the input data 406 in a first view, such as a perspective view. Figure 6 In the illustrated embodiment, classification confidence data 610 may represent one or more confidence maps that store pixel values ​​representing the probability, score, or logic that each pixel is part of a corresponding classification for each map.

[0103] At block B906, method 900 includes generating transformed classification data representing one or more classifications in the second view of the environment based at least on projecting the one or more classifications from the first view to the second view. Figure 6 In the illustrated embodiment, the classification data may correspond to classification confidence data 610 (which may represent one or more confidence maps in a first view, such as a perspective view), and the classification confidence data 610 may be transformed into a second view to form transformed classification confidence data 630 (e.g., one or more confidence maps in a second view, such as a top-down view).

[0104] At block B908, method 900 includes using the one or more NNs to generate second data representing one or more boundary shapes of one or more objects detected in the environment based at least on the converted classification data. Figure 6 In the illustrated embodiment, the transformed classification data may correspond to transformed classification confidence data 630. The transformed classification confidence data 630 (e.g., one or more confidence maps in the overhead view) and / or geometry data 640 (e.g., one or more height maps) may be encoded 645 (e.g., stacked into corresponding channels of a tensor) and fed into a second stage of the machine learning model 408, which may include an encoder / decoder backbone 650 connected to a classification confidence head 655 and an instance regression head 660. The classification confidence head 655 and the instance regression head 660 may extract the classification confidence data 610 (e.g., one or more confidence maps in the overhead view) and the instance regression data 412 (e.g., object instance data, such as location, geometry, and / or orientation), which may be post-processed (e.g., decoded) to generate a bounding box, closed polyline, or other bounding shape identifying the location, size, and / or orientation of the detected object instance.

[0105] Fig.10 1 is a flow chart illustrating a method 1000 for perspective and top-down view object detection using lidar data according to some embodiments of the present invention. At block B1002, method 1000 includes receiving lidar data from one or more lidar sensors in an environment. For example, with one or more lidar sensors (e.g., Figure 4 Firmware associated with the sensor 401) can be used to control the lidar sensor to capture and / or process lidar data, such as one or more lidar point clouds.

[0106] At block B1004, method 1000 includes generating first data representing a perspective view of the environment from the laser radar data. Figure 4 In embodiments where sensor data 402 includes lidar data, the lidar data may be accumulated, transformed to a single coordinate system (e.g., centered around the ego character / vehicle associated with the lidar sensor), autonomous motion compensated (e.g., to the last known position of the ego character / vehicle), and / or projected to form a projected image of a desired size (e.g., spatial size). For example, a (accumulated, autonomous motion compensated) lidar point cloud may be projected from a lidar range image with perspective. The projected image (e.g., lidar range image) and / or other reflectance data may be stored and / or encoded as an appropriate representation (e.g., input data 406) for a machine learning model 408.

[0107] At block B1006, method 1000 includes generating classification data from the first data using one or more neural networks (NNs), the classification data representing one or more classifications in the perspective. For example, Figure 4 The input data 406 may be used as input to a machine learning model 408, which may include multiple component machine learning models or stages linked together that sequentially process different views of the 3D environment. In an exemplary first stage, Figure 6 The encoder / decoder 605 can extract classification data (e.g., one or more confidence maps, one or more segmentation masks) from the input data 406 in a first view, such as a perspective view.

[0108] At block B1008, method 1000 includes generating converted classification data representing one or more classifications in the overhead view of the environment by projecting one or more classifications in the perspective view into the overhead view using the lidar data. Figure 6In the illustrated embodiment, the classification data may correspond to classification confidence data 610 (which may represent one or more confidence maps in a first view, such as a perspective view). In some cases, the classification value represented by the classification confidence data 610 may be associated with a corresponding 3D position detected by a lidar represented by a corresponding range scan pixel in a lidar range image to generate a marked 3D position. The marked 3D position may be projected 625 into a second view of the environment, for example, by orthogonally projecting the marked 3D position to form a projected image having a desired spatial size and ground sampling distance (e.g., a top-down view image with a top-down view). Each pixel of the resulting projected image at which the projected 3D position is located may store the extracted classification data, wherein the 3D position is labeled (e.g., a classification value or label).

[0109] At block B1010, method 1000 includes generating, using one or more NNs, second data representing one or more bounding shapes of one or more objects detected in the environment based at least on the converted classification data in the top view. For example, the converted classification data (e.g., the converted classification confidence data 630) and / or geometric data (e.g., the geometric data 640) may be encoded 645 and fed to Figure 6 The second stage of the machine learning model 408. The output of the second stage (e.g., classification confidence data 410 and instance regression data 412) can be post-processed (e.g., decoded) to generate bounding boxes, closed polylines, or other bounding shapes to identify the location, geometry, and / or orientation of the detected object instances.

[0110] Fig.11 is a flow chart illustrating a method 1100 for multi-view object detection involving projection of labeled sensor data according to some embodiments of the present disclosure. At block B1102, the method 1100 includes generating classification data representing one or more classifications from image data representing an image of a first view of an environment using one or more neural networks (NNs). For example, Figure 4 The input data 406 (e.g., a representation of a LiDAR range image) may be used as input to a machine learning model 408, which may include multiple component machine learning models or stages linked together that sequentially process different views of the 3D environment. In an exemplary first stage, Figure 6 The encoder / decoder 605 can extract classification data (e.g., one or more confidence maps, one or more segmentation masks) from the input data 406 in a first view, such as a perspective view.

[0111] At block B1104, method 1100 includes associating the classification data with corresponding three-dimensional (3D) locations identified from the corresponding sensor data to generate labeled sensor data. For example, the 3D locations of sensor data (e.g., sensor data 402) for the corresponding time slice may be labeled using the classification data (e.g., classification value, label) extracted by encoder / decoder 605. In some embodiments, the corresponding input lidar range image may have a known correspondence between range scan pixels and corresponding points in the lidar point cloud (lidar detection) that may have known 3D locations. Thus, for example, a classification value from a predicted confidence map may be associated with a 3D location of a lidar detection represented by a corresponding range scan pixel in the input lidar range image.

[0112] At block B1106, method 1100 includes projecting the tagged sensor data into a second view of the environment to generate transformed classification data representing one or more classifications in the second view. For example, the tagged 3D location can be projected 625 into the second view of the environment, for example, by orthogonally projecting the tagged 3D location to form a projected image having a desired spatial dimension and ground sampling distance (e.g., an overhead image of a top-down view). Each pixel of the resulting projected image where the 3D location is projected can store the extracted classification data, where the 3D location is labeled (e.g., a classification value or label).

[0113] At block B1108, method 1100 includes generating, using the one or more neural networks (NNs), second data representing one or more bounding shapes of one or more objects detected in the environment based at least on the converted classification data. For example, the converted classification data (e.g., the converted classification confidence data 630) and / or geometric data (e.g., the geometric data 640) may be encoded 645 and fed to Figure 6 The second stage of the machine learning model 408. The output of the second stage (e.g., classification confidence data 410 and instance regression data 412) can be post-processed (e.g., decoded) to generate bounding boxes, closed polylines, or other bounding shapes to identify the location, geometry, and / or orientation of the detected object instances.

[0114] Training a machine learning model for an object detection system

[0115] To train a machine learning model for an object detection system (e.g. Figure 4 In the case of a machine learning model 408 of the present invention, input training data can be generated from sensor data using techniques for operating the machine learning model 408 described herein. Ground truth training data can be obtained by annotating data from multiple sensors in a sensor setting.

[0116] For example, a lidar sensor and one or more cameras (e.g., a lidar sensor 1664, a stereo camera 1668, a wide-angle camera 1670 (e.g., a fisheye camera), an infrared camera 1672, a surround camera 1674 (e.g., a 360-degree camera), Figures 16A-16D The scene is observed by long-range and / or mid-range cameras 1698 of the autonomous vehicle 1600, etc.) to collect lidar data and image data. Since data can be obtained from different sensors at different frequencies, in some embodiments, a particular sensor (e.g., a lidar sensor) can be used as a reference sensor. For each frame of sensor data from the reference sensor (e.g., for each frame of lidar data), a set of sensor data can be managed by identifying the sensor data frame from each other sensor in the sensor setup (the sensor data frame closest in time to the reference sensor). This set of sensor data (e.g., the lidar data frame at timestamp T plus the image closest in time to T from each of the multiple cameras in the sensor setup) can be referred to as a set of corrected sensor data at timestamp T.

[0117] Typically, as described in more detail above, each set of corrected sensor data can be used to generate input training data for the machine learning model 408 (e.g., Figure 6 406 and / or geometric data 640). In order to generate ground truth training data for each set of accurate sensor data, the data from each sensor can be annotated and / or labeled independently of the data from other sensors. For example, a frame of lidar data can be projected to form a projected image, and the projected image can be annotated with 2D or 3D boundary shapes and supported classification labels. Additionally or alternatively, each image (or a composite image formed from constituent images from a selected set of sensor data) can be annotated with 2D or 3D boundary shapes and supported classification labels. The annotations can be generated manually and / or automatically, and used to derive corresponding ground truth outputs (e.g., classification data, such as confidence maps or segmentation masks, object instance data, etc.) of the machine learning model 408.

[0118] Fig.12 is an illustration of an exemplary technique for annotating sensor data from different sensors according to some embodiments of the present invention. Fig.12 In the example shown, sensor data may be captured from a lidar sensor and one or more cameras. In general, sensor data may be captured and managed to generate training data at any desired frequency. Fig.12The top timeline shows that in this example, lidar data frames are generated more frequently than camera images. In this case, the lidar sensor can be used as a reference sensor, and the camera image closest in time to each lidar frame can be identified to form a set of corrected sensor data (e.g., set of corrected sensor data 1210). Fig.12 In the example shown, the corrected sensor data may be generated at a frequency of one frame per second, although the corrected sensor data may be generated with any desired frequency.

[0119] exist Fig.12 In , the rectified sensor data 1210 includes a composite RBG image 1220 and frames of lidar data 1230. The triangles in the frames of lidar data 830 represent the correspondence with three images generated by three different cameras, and the numbered triangle edges represent the fields of view of the three corresponding camera stations. In this example, the composite RBG image 1220 is formed by stitching together three different RGB images. The composite RBG image 1220 and frames of lidar data 1230 can be independently annotated with 2D or 3D boundary shapes and supported classification labels. In Fig.12 , labeled image 1240 and labeled lidar frame 1250 show example annotations (classification labels are omitted for simplicity).

[0120] In general, sensor data (e.g., lidar data, RBG images) can be annotated (e.g., manually, automatically, etc.) with labels or other markers to identify the location, geometry, orientation, and / or classification of relevant object instances in the sensor data. In some examples, the labels can be generated in a 2D or 3D drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, other types of suitable programs, and / or can be drawn by hand. In any example, the labels can be artificially synthesized (e.g., generated from a computer model or rendering), truly generated (e.g., designed and generated based on real-world data), machine-automated (e.g., using feature analysis and learning to extract features from the data and then generate labels), manually annotated (e.g., an annotator or annotation expert defines the location of the labels), and / or a combination thereof (e.g., a human identifies the vertices of a polyline and a machine generates polygons using a polygon rasterizer). Typically, the labels can include 2D and / or 3D bounding boxes, closed polylines, or other bounding shapes drawn, annotated, overlaid, and / or associated with the sensor data.

[0121] Fig.131 is an illustration of example annotations for car and truck classification in camera space according to some embodiments of the present invention. For example, a boundary 1310 has been identified around a truck, and the boundary 1310 has been labeled with a corresponding classification label 1320 (e.g., truck) and one or more sub-classification labels 1330 (e.g., occluded: front view, truncated: front view). Fig.14 1 is an illustration of example annotations for pedestrian classification in camera space according to some embodiments of the present invention. For example, a boundary 1410 has been identified around a pedestrian, and the boundary 1410 has been labeled with a corresponding classification label 1420 (e.g., person) and one or more sub-classification labels 1430 (e.g., occlusion: front view, truncation: bottom truncation). Fig.15 is an illustration of example annotations of a top-down bounding box in lidar space according to some embodiments of the present invention. For example, boundaries may be identified around cars and trucks (e.g., Fig.15 Each boundary can be labeled with the corresponding class label. For simplicity, Fig.15 Classification labels are omitted.

[0122] In some embodiments, object detection and tracking may be applied to track the movement of annotation objects from frame to frame over time. Thus, annotation tracking may be used to track objects from frame to frame (e.g., using persistent identifiers for annotation objects), and the tracked annotations may be updated to fine-tune the annotations in a single frame.

[0123] In some embodiments, object tracking and / or detection from sensor data of a particular sensor may be linked with corresponding object tracking and / or detection for the same object from sensor data of a different sensor. Fig.12 , the third column shows an example in which labeled images 1240 and labeled lidar frames 1250 can be linked. For example, annotations from different types of sensor data can be projected into a common space (e.g., annotations from labeled images 1240 can be projected into the image space of labeled lidar frames 1250, and vice versa, annotations from labeled images 1240 and labeled lidar frames 1250 can be projected into some other space, etc.). Fig.12In the example shown, annotations from both labeled images 1240 and labeled lidar frames 1250 can be projected into the 3D space shown by labeled frames 1260. In some embodiments, annotations from different types of sensor data and / or different classification labels can be presented in a manner that illustrates different classifications or sensor data (e.g., presented in different colors, labels, etc.). Corresponding annotations from different types of sensor data (e.g., overlapping annotations with a threshold IoU) can be linked (e.g., manually and / or automatically). In some embodiments, linked annotations or object tracks from different types of sensor data can be used as input to an object detection and tracking process to track the movement of annotated objects from frame to frame over time, and with improved accuracy.

[0124] Additionally or alternatively, the linked sensor data can facilitate identifying and / or editing annotations. For example, the annotation software can present any view combination of sensor data, annotations, and / or labels, and can accept new annotations and / or edits to existing annotations. As a non-limiting example, the labeled lidar frame 1270 uses 3D software to display a 2D top-down bounding box expanded to 3D from the labeled lidar frame 1250 based on the correspondence with the annotations from the labeled image 1240 indicated by the labeled frame 1260. Although the focus of the preceding discussion is on the linking of lidar to camera, any type of sensor data can be linked to any other type of sensor data (including links between two different types of sensor data from the same sensor, such as camera-to-camera links). These are meant only as examples, and other variations can be implemented within the scope of the present disclosure.

[0125] Typically, each set of corrected sensor data can be used to generate input training data for machine learning model 408 (e.g., Figure 6 The input data 406 and / or geometric data 640), and the annotations (e.g., boundaries, closed regions, classification labels) can be used to generate ground truth data for the machine learning model 408 (e.g., Figure 6In order to generate ground truth data from the annotations, the location, geometry, orientation, and / or classification of each annotation can be used to generate a confidence map and / or segmentation mask that matches the view, size, and dimensions of the output of the machine learning model 408 (e.g., classification confidence data 610 and classification confidence data 410). As a non-limiting example, for a given classification and the corresponding dimension of a tensor storing different confidence maps in each channel, the pixel value of each pixel within a marked boundary shape belonging to the class can be set to a value indicating a positive classification (e.g., 1). This process can be repeated and stored in the corresponding channel of the ground truth classification confidence tensor. Additionally or alternatively, the location, geometry, orientation, and / or classification of each annotation can be used to generate object instance data that matches the view, size, and dimensions of the instance regression data 412. For example, for each pixel containing an annotation, the annotation can be used to calculate corresponding position, geometry and / or orientation information (e.g., rotation angle relative to the orientation of the projected image), and / or the like, relative to each pixel, object height, object width, object orientation (e.g., where the object is located - e.g., the center of the object), the calculated object instance data can be stored in a corresponding channel of the ground truth instance regression tensor. Thus, the annotation can be used to generate ground truth classification segmentation and / or instance regression data, which can be used together with corresponding input training data (e.g., input image, input tensor) as part of a training dataset to train the machine learning model 408.

[0126] In embodiments where the machine learning model 408 includes a chain of multiple stages, the different stages may be trained together or separately. In some embodiments, the transformation from the output of a first stage (first view) to the input of a second stage (second view) may be achieved by using differentiable operations (e.g., differentiable reprojection). Figure 6 For example, the training input data can be input into the first stage of the machine learning model 408, the output of the first stage can be transformed 615 using a differentiable transformation, decoded (e.g., using ground truth set data 640), and fed into the second stage of the machine learning model 408.

[0127] In any case, the ground truth classification segmentation and / or instance regression data can be used to train the machine learning model 408. For example, one or more loss functions (e.g., a single loss function, a loss function for each output type, etc.) can be used to compare the output accuracy of the machine learning model 408 to the ground truth value, and the parameters of the machine learning model 408 can be updated (e.g., using backward pass, backpropagation, forward pass, etc.) until the accuracy reaches an optimal or acceptable level. In some embodiments where the machine learning model 408 includes multiple heads, the multiple heads can be trained together with a common backbone, or can be trained separately. In embodiments involving multi-view scenarios with separate DNN feature extractors, the DNN feature extractors can be trained together or individually. These and other variations are also considered within the present disclosure.

[0128] In some embodiments, a classification loss function such as a multi-class cross entropy loss can be used for the classification task. The multi-class cross entropy loss can include a pixel-level cross entropy loss summed over all classes. For example, the classification loss (e.g., for the first and / or second stage of the machine learning model 408) can be expressed as H(p, y) = -∑ i y i log(p i ), where y is the ground truth data (e.g., 1 where the pixel is in the class and 0 otherwise), p is the predicted depth pixel vector with each dimension corresponding to a specific class, p is the predicted classification data for each pixel (e.g., the probability, score, or logistic that the pixel is in a given class i), and the summation can be performed for each pixel over all classes (e.g., over all depth channels). The classification loss for the full frame can be calculated by calculating the classification loss for each pixel and taking its average.

[0129] In some embodiments, a regression loss function (e.g., L1 or L2 loss) can be used for the regression task. For example, the regression loss (e.g., for the second stage of the machine learning model 408) can be expressed as R(v, t) = |vt|, where v is the predicted depth-direction pixel vector to be regressed, t is the object / ground truth depth-direction pixel vector, and the regression loss per pixel can be summed over all depth channels. In some embodiments, the regression loss can be used only for those pixels that belong to the corresponding classification mask (e.g., by Figure 6 Thus, for each pixel (in the mask), the regression loss may be summed over each depth channel and averaged over all object pixels, e.g., by normalizing (e.g., dividing by) the number of pixels that fall within the object mask (e.g., for all classes).

[0130] In some embodiments, the total loss can be calculated as the sum of the classification loss (e.g., from the first and / or second stage of the machine learning model 408) and the regression loss (e.g., from the second stage of the machine learning model 408). In some embodiments, the contributions to the losses from different tasks can be weighted using fixed weights and / or automatic weights. Additionally or alternatively, the classification loss can be weighted to offset the classification imbalance present in the training data set. These and other variations can be implemented within the scope of the present invention.

[0131] Example Autonomous Vehicle

[0132] Fig.16A 1 is a diagram of an exemplary autonomous vehicle 1600 according to some embodiments of the present disclosure. Autonomous vehicle 1600 (also referred to herein as "vehicle 1600") may include, but is not limited to, a passenger vehicle, such as a car, a truck, a bus, a first response vehicle, a shuttle, an electric or motorized bicycle, a motorcycle, a fire truck, a police car, an ambulance, a boat, a construction vehicle, an underwater vehicle, a drone, and / or other types of aircraft (e.g., unmanned and / or capable of accommodating one or more passengers). Autonomous vehicles are generally described by automation level, as defined by the National Highway Traffic Safety Administration (NHTSA) of the U.S. Department of Transportation and the Society of Automotive Engineers (SAE) "Classification and Definition of Terms Related to Driving Automation Systems for Road Motor Vehicles" (Standard No. J3016-201806, issued on June 15, 2018, Standard No. J3016-201609, issued on September 30, 2016, and previous and future versions of this standard). Vehicle 1600 may be capable of implementing one or more functions according to levels 3 to 5 of autonomous driving levels. For example, depending on the embodiment, vehicle 1600 may have conditional automation (level 3), high automation (level 4), and / or full automation (level 5).

[0133] The vehicle 1600 may include components such as a chassis, a body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components of the vehicle. The vehicle 1600 may include a drive system 1650, such as an internal combustion engine, a hybrid generator, a fully electric engine, and / or other drive system types. The drive system 1650 may be connected to a transmission of the vehicle 1600, which may include a transmission to enable driving of the vehicle 1600. The drive system 1650 may be controlled in response to receiving a signal from a throttle / accelerator 1652.

[0134] When the drive system 1650 is operating (e.g., when the vehicle is in motion), a steering system 1654 including a steering wheel can be used to guide the vehicle 1600 (e.g., along a desired path or route). The steering system 1654 can receive signals from a steering actuator 1656. The steering wheel can be optionally fully automated (level 5) functionality.

[0135] Brake sensor system 1646 may be used to operate vehicle brakes in response to signals from brake actuator 1648 and / or brake sensors.

[0136] The controller 1636 may include one or more system on chip (SoC) 1604 ( Fig. 16C ) and / or GPU, can provide signals (e.g., representing commands) to one or more components and / or systems of the vehicle 1600. For example, the controller can send signals through one or more brake actuators 1648 to operate vehicle brakes, through one or more steering actuators 1656 to operate the steering system 1654, and through one or more throttle / accelerators 1652 to operate the drive system 1650. The controller 1636 may include one or more on-board (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to achieve autonomous driving and / or assist a human driver in driving the vehicle 1600. The controller 1636 may include a first controller 1636 for autonomous driving functions, a second controller 1636 for functional safety functions, a third controller 1636 for artificial intelligence functions (e.g., computer vision), a fourth controller 1636 for infotainment functions, a fifth controller 1636 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1636 may handle two or more of the above functions, two or more controllers 1636 may handle a single function, and / or any combination thereof.

[0137] The controller 1636 may provide signals for controlling one or more components and / or systems of the vehicle 1600 in response to sensor data (e.g., sensor input) received from one or more sensors. The sensor data may be received from, for example and without limitation, a global navigation satellite system sensor 1658 (e.g., a global positioning system sensor), a radar sensor 1660, an ultrasonic sensor 1662, a lidar sensor 1664, an inertial measurement unit (IMU) sensor 1666 (e.g., an accelerometer, a gyroscope, a magnetic compass, a magnetometer, etc.), a microphone 1696, a stereo camera 1668, a wide angle camera 1670 (e.g., a fisheye camera), an infrared camera 1672, an omnidirectional body camera 1674 (e.g., a 360 degree camera), a long-range and / or mid-range camera 1698, a speed sensor 1644 (e.g., for measuring the speed of the vehicle 1600), a vibration sensor 1642, a steering sensor 1640, a brake sensor (e.g., as part of a brake sensor system 1646), and / or other sensor types.

[0138] One or more of the controllers 1636 may receive input (e.g., represented by input data) from the instrument panel 1632 of the vehicle 1600 and provide output (e.g., represented by output data, display data, etc.) through a human machine interface (HMI) display 1634, an audible alarm, a speaker, and / or through other components of the vehicle 1600. The output may include information such as vehicle speed, velocity, time, map data (e.g., Fig. 16C The HMI display 1634 may include information about the presence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.), and / or information about driving maneuvers that the vehicle has taken, is taking, or is about to take (e.g., changing lanes now, taking exit 34B in two miles, etc.).

[0139] The vehicle 1600 also includes a network interface 1624 that can communicate via one or more networks using one or more wireless antennas 1626 and / or a modem. For example, the network interface 1624 can communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. The wireless antenna 1626 can also use a local area network (such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc.), and / or a low power wide area network (LPWAN) such as LoRaWAN, SigFox, etc. to enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.).

[0140] Fig. 16B According to some embodiments of the present disclosure Fig.16A 1600. The cameras and respective fields of view are an exemplary embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included, and / or the cameras may be located at different locations on vehicle 1600.

[0141] The camera type of the camera may include, but is not limited to, a digital camera that can be used with the components and / or systems of the vehicle 1600. The camera may operate at an automotive safety integrity level (ASIL) B and / or other ASILs. According to an embodiment, the camera type may have any image capture rate, such as 60 frames per second (fps), 1620fps, 240fps, etc. The camera may use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red transparent transparent (RCCC) color filter array, a red transparent blue (RCCB) color filter array, a red blue green transparent (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or other types of color filter arrays. In some embodiments, a clear pixel camera such as one with an RCCC, RCCB, and / or RBGC color filter array may be used to strive to improve light sensitivity.

[0142] In some examples, one or more cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function single camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more cameras (e.g., all cameras) can record and provide image data (e.g., video) simultaneously.

[0143] One or more cameras can be mounted in a mounting assembly, such as a custom designed (3-D printed) assembly, so as to cut out stray light and reflections from inside the vehicle (e.g., reflections of the dashboard reflecting off the windshield mirror) that may interfere with the camera's ability to capture image data. With respect to the rearview mirror mounting assembly, the rearview mirror assembly can be custom 3D printed so that the camera mounting plate matches the shape of the rearview mirror. In some examples, the camera can be integrated into the rearview mirror. For side view cameras, the camera can also be integrated into the four pillars at each corner of the cab.

[0144] A camera (e.g., a forward-facing camera) whose field of view includes a portion of the environment in front of the vehicle 1600 can be used for all-around vision to help identify the forward path and obstacles, and with the help of one or more controllers 1636 and / or the controlling SOC, to help identify the forward path and obstacles, providing key information for generating an occupancy grid and / or determining a preferred vehicle path. A forward-facing camera can be used to perform many of the same ADAS functions as a lidar, including emergency braking, pedestrian detection, and collision avoidance. A forward-facing camera can also be used for ADAS functions and systems, including other functions such as lane departure warning ("LDW"), automatic cruise control ("ACC"), and / or traffic sign recognition.

[0145] A variety of cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform including a CMOS (complementary metal oxide semiconductor) color imager. Another example may be a wide-angle camera 1670, which may be used to sense objects that enter the field of view from the periphery (e.g., pedestrians, intersection traffic, or bicycles). Although in Fig. 16B Only one wide-angle camera is shown, but there may be any number of wide-angle cameras 1670 on vehicle 1600. In addition, remote cameras 1698 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. Remote cameras 1698 may also be used for object detection and classification, as well as basic object tracking.

[0146] One or more stereo cameras 1668 may also be included in the front configuration. Stereo cameras 1668 may include an integrated control unit including a scalable processing unit that may provide programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet interface on a single chip. Such a unit may be used to generate a three-dimensional map of the vehicle environment, including distance estimates for all points in the image. Another stereo camera 1668 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip that may measure the distance from the vehicle to the object and use the generated information (e.g., metadata) to activate automatic emergency braking and lane departure warning functions. Other types of stereo cameras 1668 may also be used to supplement or replace those described herein.

[0147] Cameras with a field of view that includes portions of the surroundings to the sides of the vehicle 1600 (e.g., side view cameras) can be used for all-around vision, providing for creating and updating occupancy grids and generating side collision warning information. For example, a surround camera 1674 (e.g., Fig. 16BThe four surround cameras 1674 shown in FIG. 16 can be placed on the vehicle 1600. The surround cameras 1674 can include a wide-angle camera 1670, a fisheye camera, a 360-degree camera, etc. For example, the four fisheye cameras can be placed on the front, rear, and sides of the vehicle. In another arrangement, the vehicle can use three surround cameras 1674 (e.g., left, right, and rear), and can utilize one or more other cameras (e.g., a forward-looking camera) as a fourth surround view camera.

[0148] A camera having a partial view of the environment including the rear of the vehicle 1600 (e.g., a rear view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. As described herein, a variety of cameras can be used, including but not limited to cameras that are also suitable as forward-facing cameras (e.g., long-range and / or mid-range cameras 1698, stereo cameras 1668, infrared cameras 1672, etc.).

[0149] Fig. 16C According to some embodiments of the present disclosure, Fig.16A Block diagram of an exemplary system architecture of an exemplary autonomous vehicle 1600. It should be understood that this and other arrangements described herein are set forth only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, functional groupings, etc.) may be used to supplement or replace the content shown, and some elements may be omitted entirely. In addition, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or together with other components, and in any appropriate combination and location. The various functions performed by the entities described herein may be performed by hardware, firmware, and / or software. For example, the various functions may be performed by a processor executing instructions stored in a memory.

[0150] Fig. 16C Each component, feature, and system of the vehicle 1600 in the example is connected via a bus 1602. The bus 1602 may include a controller area network (CAN) data interface (also referred to herein as a "CAN bus"). The CAN can be a network within the vehicle 1600 that helps control various features and functions of the vehicle 1600, such as starting the brakes, accelerating, braking, steering, windshield wipers, etc. The CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). The CAN bus can be read to find steering wheel angle, driving speed, engine revolutions per minute (RPM), button position, and / or other vehicle status indicators. The CAN bus can be compatible with ASIL B.

[0151] Although bus 1602 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may also be used to supplement or replace the CAN bus. In addition, although a single line is used to represent bus 1602, this is not intended to be limiting. For example, there may be any number of buses 1602, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1602 may be used to perform different functions, and / or may be used for redundancy. For example, a first bus 1602 may be used for collision avoidance functions, and a second bus 1602 may be used for drive control. In any example, each bus 1602 may communicate with any component of vehicle 1600, and two or more buses 1602 may communicate with the same component. In some examples, each SoC 1604, each controller 1636, and / or each computer in the vehicle may access the same input data (e.g., input from sensors of vehicle 1600), and may be connected to a common bus, such as a CAN bus.

[0152] The vehicle 1600 may include one or more controllers 1636, such as those described herein. Fig.16A Controller 1636 described. Controller 1636 can be used for various functions. Controller 1636 can be coupled to any of the various other components and systems of vehicle 1600, and can be used for control of vehicle 1600, artificial intelligence of vehicle 1600, infotainment of vehicle 1600, etc.

[0153] The vehicle 1600 may include a system on a chip (SoC) 1604. The SoC 1604 may include a CPU 1606, a GPU 1608, a processor 1610, a cache 1612, an accelerator 1614, a data memory 1616, and / or other components and features not shown. The SoC 1604 may be used to control the vehicle 1600 in various platforms and systems. For example, the SoC 1604 may be combined with an HD map 1622 in a system (e.g., a system of the vehicle 1600), and the HD map 1622 may be obtained from one or more servers (e.g., a server) via a network interface 1624. Fig.16D Server 1678) obtains map refreshes and / or updates.

[0154] CPU 1606 may include a CPU cluster or CPU complex (also referred to herein as a "CCPLEX"). CPU 1606 may include multiple cores and / or L2 caches. For example, in some embodiments, CPU 1606 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1606 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1606 (e.g., CCPLEX) may be configured to support synchronous cluster operation enablement, such that any combination of CPU 1606 clusters is active at any given time.

[0155] CPU 1606 may implement power management capabilities including one or more of the following features: individual hardware blocks may be automatically clock gated when idle to save dynamic power; each core clock may be gated when the core is not actively executing instructions due to executing WFI / WFE instructions; each core may be independently power gated; each core cluster may be independently clock gated when all cores are clock gated or power gated; and / or each core cluster may be independently power gated when all cores are power gated. CPU 1606 may further implement an enhanced algorithm for managing power states, where allowed power states and expected wakeup times are specified, and hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core may support a simplified power state entry sequence in software, with work offloaded to microcode.

[0156] GPU 1608 may include an integrated GPU (also referred to herein as an "iGPU"). GPU 1608 may be programmable and may be efficient for parallel workloads. In some examples, GPU 1608 may use an enhanced tensor instruction set. GPU 1608 may include one or more streaming microprocessors, each of which may include a level 1 cache (e.g., an L1 cache with at least 96KB storage capacity), and two or more streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB storage capacity). In some embodiments, GPU 1608 may include at least eight streaming microprocessors. GPU 1608 may use a computing application programming interface (API). In addition, GPU 1608 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0157] GPU 1608 can be power optimized for optimal performance in automotive and embedded use cases. For example, GPU 1608 can be manufactured with fin field effect transistors (FinFETs). However, this is not intended to be limiting, and GPU 1608 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can include multiple mixed precision processing cores divided into multiple blocks. For example, but not limited to, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed precision NVIDIA tensor cores for deep learning matrix algorithms, L0 instruction cache, thread warp scheduler, scheduling unit and / or 64KB register file can be allocated to each processing block. In addition, the streaming microprocessor may include independent parallel integer and floating point data paths to provide efficient execution of workloads through mixed calculations and addressing calculations. The streaming microprocessor may include independent thread scheduling capabilities to achieve finer-grained synchronization and collaboration between parallel threads. A streaming microprocessor may include a combined L1 data cache and shared memory unit to increase performance while simplifying programming.

[0158] GPU 1608 may include high bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / sec in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as graphics double data rate type five synchronous random access memory (GDDR5), may be used in addition to or in lieu of HBM memory.

[0159] The GPU 1608 may include unified memory technology including access counters to allow more accurate migration of memory pages to the processors that access them most frequently, thereby improving the efficiency of memory ranges shared between processors. In some examples, address translation service (ATS) support may be used to allow the GPU 1608 to directly access the CPU 1606 page table. In such an example, when the GPU 1608 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU 1606. In response, the CPU 1606 may look up the virtual-to-physical mapping of the address in its page table and send the translation back to the GPU 1608. Thus, the unified memory technology may allow a single unified virtual address space for the memory of the CPU 1606 and the GPU 1608, thereby simplifying the programming of the GPU 1608 and the porting of applications to the GPU 1608.

[0160] Additionally, GPU 1608 may include access counters that can keep track of how often GPU 1608 accesses other processors' memory. The access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses the page most frequently.

[0161] SoC 1604 may include any number of caches 1612, including those described herein. For example, cache 1612 may include an L3 cache available to both CPU 1606 and GPU 1608 (e.g., CPU 1606 and GPU 1608 are connected). Cache 1612 may include a write-back cache that may keep track of the state of a line, such as by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). According to an embodiment, the L3 cache may include 4MB or more, although smaller cache sizes may be used.

[0162] The SoC 1604 may include an arithmetic logic unit (ALU) that may be used to perform processing for various tasks or operations related to the vehicle 1600, such as processing a DNN. In addition, the SoC 1604 may include a floating point unit (FPU) or other math coprocessor or digital coprocessor type for performing mathematical operations within the system. For example, the SoC 104 may include one or more FPUs that are integrated as execution units within the CPU 1606 and / or GPU 1608.

[0163] SoC 1604 may include one or more accelerators 1614 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, SoC 1604 may include a hardware acceleration cluster, which may include optimized hardware accelerators and / or large on-chip memory. Large on-chip memory (e.g., 4MB of SRAM) enables the hardware acceleration cluster to accelerate neural networks and other calculations. The hardware acceleration cluster can be used to supplement GPU 1608 and offload some tasks of GPU 1608 (e.g., freeing up more cycles of GPU 1608 to perform other tasks). As an example, accelerator 1614 can be used to stabilize object workloads (e.g., perception, convolutional neural network (CNN)), etc. that are suitable for acceleration. The term "CNN" as used herein may include all types of CNNs, including region-based or regional convolutional neural networks (RCNN) and fast RCNN (e.g., for object detection).

[0164] Accelerator 1614 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that may be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured and optimized to perform image processing functions (e.g., for CNN, RCNN, etc.). The DLA may be further optimized for a specific set of neural network types and floating point operations and inference. The design of the DLA may provide higher performance per millimeter than a general-purpose GPU and greatly exceed the performance of the CPU. The TPU may perform multiple functions, including single instance convolution functions, support, for example, INT8, INT16, and FP16 data types for features and weights, and post-processor functions.

[0165] The DLA can quickly and efficiently execute neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, for example, but not limited to: a CNN for object recognition and detection using data from a camera sensor; a CNN for distance estimation using data from a camera sensor; a CNN for emergency vehicle detection and identification and detection using data from a microphone; a CNN for facial recognition and identification of a vehicle owner using data from a camera sensor; and / or a CNN for safety and / or security-related events.

[0166] The DLA can perform any function of the GPU 1608, and by using, for example, an inference accelerator, the designer can target either the DLA or the GPU 1608 for any function. For example, the designer can focus the processing of CNNs and floating point operations on the DLA and leave other functions to the GPU 1608 and / or other accelerators 1614.

[0167] Accelerator 1614 (e.g., a hardware acceleration cluster) may include a programmable vision accelerator (PVA), which may also be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA may provide a balance between performance and flexibility. For example, each PVA may include, for example and without limitation, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0168] The RISC core can interact with an image sensor (e.g., an image sensor of any camera described herein), an image signal processor, etc. Each RISC core can include any number of memories. Depending on the number of RISCs, the RISC can use any protocol. In some examples, the RISC core can execute a real-time operating system (RTOS). The RISC core can be implemented using one or more integrated circuit devices, application specific integrated circuits (ASICs), and / or memory devices. For example, the RISC core can include an instruction cache and / or a tightly coupled RAM.

[0169] The DMA may enable components of the PVA to access system memory independently of the CPU 1606. The DMA may support any number of features for providing optimizations to the PVA, including, but not limited to, support for multi-dimensional addressing and / or circular addressing. In some examples, the DMA may support up to six or more addressing dimensions, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0170] A vector processor may be a programmable processor that may be designed to efficiently and flexibly execute programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or a vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single instruction, multiple data (SIMD), a very long instruction word (VLIW) digital signal processor. The combination of SIMD and VLIW may improve throughput and speed.

[0171] Each vector processor can include an instruction cache and can be coupled to a dedicated memory. As a result, in some examples, each vector processor can be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA can be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can execute the same computer vision algorithm, but on different areas of an image. In other examples, the vector processors included in a particular PVA can execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms on sequence images or portions of images. Among other aspects, any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each PVA. In addition, the PVA can include additional error correction code (ECC) memory to enhance the security of the entire system.

[0172] The accelerator 1614 (e.g., a hardware acceleration cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1614. In some examples, the on-chip memory may include at least 4MB of SRAM, which is composed of, for example and without limitation, eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuits, controllers, and multiplexers. Any type of memory may be used. The PVA and DLA may access the memory through a backbone network that provides high-speed access to the memory for the PVA and DLA. The backbone network may include an on-chip computer vision network that interconnects the PVA and DLA to the memory (e.g., using APB).

[0173] The on-chip computer vision network may include an interface that determines that both the PVA and DLA provide ready valid signals before transmitting any control signals / addresses / data. Such an interface may provide independent phases and independent channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface may comply with ISO 26262 or IEC 61508 standards, but other standards and protocols may also be used.

[0174] In some examples, SoC 1604 may include a real-time ray tracing hardware accelerator, such as described in U.S. Patent Application No. 16 / 101,232 filed on August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the position and range of objects (e.g., within a world model) to generate real-time visualization simulations, for radar signal interpretation, for sound propagation synthesis and / or analysis, for simulation of sonar systems, for general wave propagation simulations, for comparison with lidar data to achieve positioning and / or other functions, and / or for other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing related operations.

[0175] The accelerator 1614 (e.g., a cluster of hardware accelerators) has a wide range of uses in autonomous driving. The PVA can be a programmable vision accelerator that can be used in key processing stages of ADAS and autonomous vehicles. The capabilities of the PVA are well suited for algorithmic domains that require predictable processing at low power and low latency. In other words, the PVA performs well on semi-intensive or intensive regular computations, even on small data sets with predictable runtimes that require low latency and low power. Therefore, in the context of autonomous vehicle platforms, the PVA is designed to run classic computer vision algorithms because they are efficient at object detection and integer operations.

[0176] For example, according to one embodiment of the technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching based algorithms can be used, although this is not intended to be limiting. Many Level 3-5 autonomous driving applications require dynamic motion estimation / stereo matching (e.g., motion structure, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0177] In some examples, PVA can be used to perform dense optical flow. Processed radar is provided based on processing raw radar data (e.g., using a 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, e.g., by processing raw time-of-flight data to provide processed time-of-flight data.

[0178] DLA can be used to run any type of network to enhance control and driving safety, including, for example, a neural network that outputs a confidence level for each object detection. Such a confidence value can be interpreted as a probability, or provide a relative "weight" of each detection relative to other detections. This confidence value enables the system to further decide which detections should be considered true correct detections rather than false correct detections. For example, the system can set a threshold for the confidence and only consider detections that exceed the threshold as true detections. In an automatic emergency braking (AEB) system, a false correct detection causes the vehicle to automatically perform emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run a neural network to regress confidence values. The neural network can take as its input at least some subset of parameters, such as the bounding box size associated with the direction, distance, 3D position estimate of the object of the vehicle 1600 obtained from the neural network and / or other sensors (e.g., lidar sensor 1664 or radar sensor 1660), the obtained ground plane estimate (e.g., obtained from another subsystem), and the output of the inertial measurement unit (IMU) sensor 1666.

[0179] SoC 1604 may include data store 1616 (e.g., memory). Data store 1616 may be on-chip memory of SoC 1604 that may store neural networks to be executed on the GPU and / or DLA. In some instances, the capacity of data store 1616 may be large enough to store multiple instances of the neural network for redundancy and safety. Data store 1612 may include an L2 or L3 cache 1612. As described herein, references to data store 1616 may include references to memory associated with the PVA, DLA, and / or other accelerators 1614.

[0180] SoC 1604 may include one or more processors 1610 (e.g., embedded processors). Processor 1610 may include a boot and power management processor, which may be a dedicated processor and subsystem that handles boot power and management functions and related safety enforcement. The boot and power management processor may be part of the SoC 1604 boot sequence and may provide runtime power management services. The boot power and management processor may provide clock and voltage programming, assistance with system low power state transitions, management of SoC 1604 thermal and temperature sensors, and / or management of SoC 1604 power states. Each temperature sensor may be implemented as a ring oscillator with an output frequency proportional to temperature, and SoC 1604 may use the ring oscillator to detect the temperature of CPU 1606, GPU 1608, and / or accelerator 1614. If it is determined that the temperature exceeds a threshold, the boot and power management processor may enter a temperature fault routine and place SoC 1604 in a low power state and / or place vehicle 1600 in a driver safety parking mode (e.g., bringing vehicle 1600 to a safe parking point).

[0181] Processor 1610 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine may be an audio subsystem that enables multi-channel audio on multiple interfaces, as well as a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0182] The processor 1610 may further include an always-on processor engine that may provide the necessary hardware features to support low-power sensor management and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (e.g., timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0183] Processor 1610 may also include a safety cluster engine that includes a dedicated processor subsystem for handling safety management of automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, two or more cores may operate in lockstep mode and as a single core with comparison logic to detect any differences between their operations.

[0184] Processor 1610 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0185] Processor 1610 may also include a high dynamic range signal processor, which may include an image signal processor that is a hardware engine that is part of the camera processing pipeline.

[0186] The processor 1610 may include a video image synthesizer, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to generate a final image for a player window. The video image synthesizer may perform lens distortion correction for the wide-angle camera 1670, the omnidirectional body camera 1674, and / or the in-cabin surveillance camera sensor. Preferably, the neural network is configured to monitor events of another SoC running in the cabin through another SoC instance running in the cabin, and respond accordingly to events of another SoC running. The in-cabin system may perform lip reading to activate cellular service, as well as make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain features are only available to the driver when the vehicle is operating in automatic mode and are otherwise disabled.

[0187] The video image synthesizer may include enhanced temporal noise suppression for both spatial and temporal noise reduction. For example, in the case of motion occurring in the video, the noise suppression appropriately weights the spatial information, reducing the weight of information provided by adjacent frames. In the case where an image or portion of an image does not include motion, the temporal noise suppression performed by the video image synthesizer may use information from previous images to reduce noise in the current image.

[0188] The video image compositor can also be configured to perform stereoscopic rectification on the input stereoscopic lens frames. The video image compositor can also be used for user interface composition when the operating system desktop is in use, and does not require the GPU 1608 to continuously render new surfaces. Even when the GPU 1608 is powered on and 3D rendering is in progress, the video image compositor can be used to offload the GPU 1608 to improve performance and responsiveness.

[0189] SoC 1604 may also include a mobile industry processor interface (MIPI) camera serial interface for receiving video and input from a camera, a high-speed interface, and / or a video input block that may be used for a camera and related pixel input functions. SoC 1604 may also include a software-controllable input / output controller that may be used to receive I / O signals that are not committed to a specific role.

[0190] SoC 1604 may also include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. SoC 1604 may be used to process data from cameras (e.g., via Gigabit multimedia serial links and Ethernet connections), sensors (e.g., LiDAR sensor 1664, radar sensor 1660, etc., which may be connected via Ethernet), data from bus 1602 (e.g., speed of vehicle 1600, steering wheel position, etc.), data from GNSS sensor 1658 (e.g., via Ethernet or CAN bus connections). SoC 1604 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free CPU 1606 from routine data management tasks.

[0191] SoC 1604 can be an end-to-end platform with a flexible architecture spanning automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and effectively uses computer vision and ADAS technologies to achieve diversity and redundancy, providing a flexible platform, reliable driver software stack, and deep learning tools. SoC 1604 can be faster, more reliable, and even more energy-efficient and space-efficient than traditional systems. For example, when accelerator 1614 is combined with CPU 1606, GPU 1608, and data storage 1616, a fast and efficient platform can be provided for level 3-5 autonomous vehicles.

[0192] As a result, the technology provides capabilities and functionality that are not possible with traditional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured to perform a wide variety of processing algorithms on a wide variety of visual data using a high-level programming language (e.g., the C programming language). However, CPUs often fail to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and a requirement for practical Level 3-5 autonomous vehicles.

[0193] Compared to traditional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the techniques described herein allow multiple neural networks to be executed simultaneously and / or sequentially, and allow the results to be combined together to achieve Level 3-5 autonomous driving functions. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1620) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including signs that the neural network has not been specifically trained. The DLA can further include a neural network that is capable of recognizing, interpreting, and providing semantic understanding of the symbols and passing the semantic understanding to a path planning module running on the CPU complex.

[0194] As another example, multiple neural networks may be run simultaneously, as may be required for Level 3, 4, or 5 driving. For example, a warning sign consisting of “WARNING: FLASHING LIGHTS INDICATES ICE” and electric lights may be interpreted by several neural networks independently or collectively. The sign itself may be recognized as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text “FLASHING LIGHTS INDICATES ICE” may be interpreted by a second deployed neural network, which notifies the vehicle's path planning software (preferably executing on a CPU complex) when flashing lights are detected and an icy condition exists. The flashing lights may be recognized by operating a third deployed neural network over multiple frames, notifying the vehicle's path planning software of the presence (or absence) of flashing lights. All three neural networks may be run simultaneously, such as within a DLA and / or on GPU 1608.

[0195] In some examples, the CNN for facial recognition and owner recognition can use data from the camera sensor to identify the presence of an authorized driver and / or owner of the vehicle 1600. The always-on sensor processing engine can be used to unlock the vehicle when the owner approaches the driver's door and turns on the lights, and in security mode, disable the vehicle when the owner leaves the vehicle. In this way, the SoC 1604 provides security against theft and / or carjacking.

[0196] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 1696 to detect and identify emergency vehicle alarms. Unlike traditional systems that use general classifiers to detect alarms and manually extract features, SoC 1604 uses CNN to classify environmental and urban sounds, as well as classify visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative closing speed of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the area in which the vehicle is operating, as identified by the GNSS sensor 1658. Thus, for example, when operating in Europe, the CNN will seek to detect European alarms, while when in the United States, the CNN will only seek to identify North American alarms. Once an emergency vehicle is detected, the control program can be used to execute an emergency vehicle safety routine, slowing the vehicle, pulling the vehicle over, and / or idling the vehicle with the help of the ultrasonic sensor 1662 until the emergency vehicle passes.

[0197] The vehicle may include a CPU 1618 (e.g., a discrete CPU or dCPU) that may be coupled to the SoC 1604 via a high-speed interconnect (e.g., PCIe). For example, the CPU 1618 may include an X86 processor. The CPU 1618 may be used to perform any of a variety of functions, including arbitrating potentially inconsistent results between the ADAS sensors and the SoC 1604, and / or monitoring the status and health of the controller 1636 and / or the infotainment SoC 1630.

[0198] The vehicle 1600 may include a GPU 1620 (e.g., a discrete GPU or dGPU) that may be coupled to the SoC 1604 via a high-speed interconnect (e.g., NVIDIA's NVLINK). The GPU 1620 may provide additional artificial intelligence functionality, such as by executing redundant and / or different neural networks, and may be used to train and / or update the neural network based on input from sensors of the vehicle 1600 (e.g., sensor data).

[0199] The vehicle 1600 may further include a network interface 1624, which may include one or more wireless antennas 1626 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 1624 can be used to implement wireless connections with the cloud (e.g., with a server 1678 and / or other network devices), with other vehicles, and / or with a computing device (e.g., a passenger's client device) over the Internet. In order to communicate with other vehicles, a direct link can be established between the two vehicles and / or an indirect link can be established (e.g., through a network and the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide the vehicle 1600 with information about vehicles near the vehicle 1600 (e.g., vehicles in front of, to the side of, and / or behind the vehicle 1600). This functionality can be part of the collaborative adaptive cruise control functionality of the vehicle 1600.

[0200] The network interface 1624 may include a SoC that provides modulation and demodulation functions and enables the controller 1636 to communicate over a wireless network. The network interface 1624 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. The frequency conversion may be performed by well-known processes and / or may be performed using superheterodyne processing. In some examples, the radio frequency front end function may be provided by a separate chip. The network interface may include a wireless function for communicating via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0201] The vehicle 1600 may further include data storage 1628, which may include off-chip (e.g., outside of the SoC 1604) memory. The data storage 1628 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices that can store at least one bit of data.

[0202] The vehicle 1600 may also include a GNSS sensor 1658. The GNSS sensor 1658 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used to assist in mapping, perception, generating occupancy grids, and / or path planning functions. Any number of GNSS sensors 1658 may be used, such as, but not limited to, GPS using a USB connector with an Ethernet to serial (RS-232) bridge.

[0203] The vehicle 1600 may further include a radar sensor 1660. The vehicle 1600 may use the radar sensor 1660 for remote vehicle detection even in darkness and / or inclement weather conditions. The radar functional safety level may be ASIL B. The radar sensor 1660 may use CAN and / or bus 1602 (e.g., to transmit data generated by the radar sensor 1660) for control and access to object tracking data, and in some examples, may access Ethernet to access raw data. A variety of radar sensor types may be used. For example, and without limitation, the radar sensor 1660 may be suitable for front, rear, and side radar use. In some examples, a pulse Doppler radar sensor is used.

[0204] The radar sensor 1660 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range radar can be used for adaptive cruise control functions. The long-range radar system can provide a wide field of view achieved by two or more independent scans, for example within a range of 250 meters. The radar sensor 1660 can help distinguish between static and moving objects and can be used by ADAS systems for emergency brake assistance and forward collision warning. The long-range radar sensor can include a single-station multi-mode radar with multiple (e.g., six or more) fixed radar antennas and high-speed CAN and FlexRay interfaces. In the example with six antennas, the central four antennas can produce a focused beam pattern designed to record the surroundings of the vehicle 1600 at higher speeds with minimal interference from traffic in adjacent lanes. The additional two antennas can expand the field of view, making it possible to quickly detect vehicles entering or leaving the lane of the vehicle 1600.

[0205] For example, a mid-range radar system may include a range of up to 1,660 meters (front) or 80 meters (rear), and a field of view of up to 42 degrees (front) or 1,650 degrees (rear). A short-range radar system may include, but is not limited to, a radar sensor designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a radar sensor system may generate two beams to continuously monitor the blind spots behind and beside the vehicle.

[0206] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0207] The vehicle 1600 may also include an ultrasonic sensor 1662. The ultrasonic sensor 1662 may be located at the front, rear, and / or side of the vehicle 1600 and may be used for parking assistance and / or creating and updating an occupancy grid. A variety of ultrasonic sensors 1662 may be used, and different ultrasonic sensors 1662 may be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor 1662 may operate at a functional safety level of ASILB.

[0208] Vehicle 1600 may include a lidar sensor 1664. Lidar sensor 1664 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. Lidar sensor 1664 may be functional safety level ASIL B. In some examples, vehicle 1600 may include multiple lidar sensors 1664 (e.g., two, four, six, etc.), which may use Ethernet (e.g., provide data to a Gigabit Ethernet switch).

[0209] In some examples, the lidar sensor 1664 can provide a list of objects and their distances in a 360-degree field of view. For example, a commercial lidar sensor 1664 can have an advertised range of approximately 1600m, an accuracy of 2cm-3cm, and support for a 1600Mbps Ethernet connection. In some examples, one or more non-obtrusive lidar sensors 1664 can be used. In these examples, the lidar sensor 1664 can be implemented as a small device that can be embedded in the front, rear, side, and / or corner of the vehicle 1600. In such an example, the lidar sensor 1664 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, with a range of 200 meters even for objects with low reflectivity. The front-mounted lidar sensor 1664 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0210] In some examples, lidar technology, such as 3D flash lidar, can also be used. 3D flash lidar uses laser pulses as a propagation source to illuminate up to 200 meters around the vehicle. The flash lidar includes a receiver that records the laser pulse propagation time and the reflected light at each pixel, which in turn corresponds to the distance from the vehicle to the object. The flash lidar can produce a high-precision and distortion-free image of the environment for each laser flash. In some examples, four flash lidar sensors can be deployed, one on each side of the vehicle 1600. Available 3D flash lidar systems include solid-state 3D staring array lidar cameras with no moving components except the fan (e.g., non-scanning lidar devices). The flash lidar device can use a Class I (eye-safe) laser pulse of 5 nanoseconds per frame and can capture reflected laser light in the form of a three-dimensional distance point cloud and co-registered intensity data. By using flash lidar, and because the flash lidar is a solid-state device with no moving components, the lidar sensor 1664 can be less susceptible to motion blur, vibration and / or shock.

[0211] The vehicle may also include an IMU sensor 1666. In some examples, the IMU sensor 1666 may be located at the center of the rear axle of the vehicle 1600. The IMU sensor 1666 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1666 may include an accelerometer and a gyroscope, and in a nine-axis application, the IMU sensor 1666 may include an accelerometer, a gyroscope, and a magnetometer.

[0212] In some embodiments, the IMU sensor 1666 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a micro-electromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1666 can enable the vehicle 1600 to estimate heading without the need for input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1666. In some examples, the IMU sensor 1666 and the GNSS sensor 1658 can be combined in a single integrated unit.

[0213] The vehicle may include microphones 1696 placed in and / or around the vehicle 1600. The microphones 1696 may be used for emergency vehicle detection and identification, among other things.

[0214] The vehicle may further include any number of camera types, including stereo cameras 1668, wide-angle cameras 1670, infrared cameras 1672, omnidirectional body cameras 1674, long-range and / or mid-range cameras 1698, and / or other camera types. The cameras may be used to capture image data of the entire periphery of the vehicle 1600. The type of camera used depends on the embodiment and requirements of the vehicle 1600, and any combination of camera types may be used to provide the necessary coverage around the vehicle 1600. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or other numbers of cameras. As an example, the cameras may support (but are not limited to) Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Reference is made herein to Fig.16A and Fig. 16B Describe each camera in more detail.

[0215] The vehicle 1600 may also include a vibration sensor 1642. The vibration sensor 1642 may measure vibrations of a vehicle component, such as an axle. For example, changes in vibration may indicate changes in the road surface. In another example, when two or more vibration sensors 1642 are used, the difference between the vibrations may be used to determine friction or slippage of the road surface (e.g., when the difference in vibration is between a powered drive shaft and a free-spinning shaft).

[0216] The vehicle 1600 may include an ADAS system 1638. In some examples, the ADAS system 1638 may include a SoC. The ADAS system 1638 may include automatic / adaptive / auto cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0217] The ACC system may use a radar sensor 1660, a lidar sensor 1664, and / or a camera. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle in front of the vehicle 1600 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle in front. Lateral ACC performs distance keeping and suggests that the vehicle 1600 change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0218] CACC uses information from other vehicles, which can be received from other vehicles via a wireless link through the network interface 1624 and / or the wireless antenna 1626, or indirectly received through a network connection (e.g., via the Internet). The direct link can be provided by a vehicle-to-vehicle (V2V) communication link, while the indirect link can be an infrastructure-to-vehicle (I2V) communication link. In general, the V2V communication concept provides information about the previous vehicle (e.g., the vehicle in front of the same lane as vehicle 1600), while the I2V communication concept provides information about the traffic ahead. The CACC system can include one or both of the I2V and V2V information sources. Taking into account the information of the vehicle in front of vehicle 1600, CACC may be more reliable, and it has the potential to improve the smoothness of traffic flow and reduce congestion on the road.

[0219] The FCW system is intended to alert the driver to hazards so that the driver can take corrective action. The FCW system uses a front camera and / or radar sensor 1660 coupled to a dedicated processor, DSP, FPGA and / or ASIC electrically coupled to driver feedback (e.g., a display, speaker and / or vibration component). The FCW system can provide warnings such as sound, visual warnings, vibrations and / or rapid brake pulses.

[0220] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply brakes if the driver does not take corrective action within specified time or distance parameters. The AEB system can use a forward-facing camera and / or radar sensor 1660 coupled to a dedicated processor, DSP, FPGA and / or ASIC. When the AEB system detects a hazard, it will typically first alert the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system can automatically apply brakes to prevent or at least mitigate the effects of the expected collision. The AEB system can include technologies such as dynamic brake support and / or emergency braking.

[0221] The LDW system provides visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle 1600 crosses lane markings. The LDW system does not activate when the driver indicates an intention to leave the lane by activating a turn signal. The LDW system may use a forward-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration assembly.

[0222] The LKA system is a variation of the LDW system. If the vehicle 1600 begins to leave the lane, the LKA system provides steering input or braking to correct the vehicle 1600.

[0223] The BSW system detects and warns the driver of vehicles in the blind spot of the car. The BSW system can provide visual, audible and / or tactile alerts to indicate that it is unsafe to merge or change lanes. The system can provide additional warnings when the driver uses a turn signal. The BSW system can use a rear-facing camera and / or radar sensor 1660 coupled to a dedicated processor, DSP, FPGA and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker and / or vibration component.

[0224] When the vehicle 1600 is in reverse, the RCTW system can provide visual, audible and / or tactile notifications when an object outside the range of the rear-facing camera is detected. Some RCTW systems include AEB to ensure that vehicle brakes are applied to avoid collisions. The RCTW system can use one or more rear-facing radar sensors 1660 coupled to a dedicated processor, DSP, FPGA and / or ASIC, which are electrically coupled to driver feedback such as a display, speaker and / or vibration component.

[0225] Conventional ADAS systems may be prone to pseudo-correct results, which may be annoying and distracting to the driver, but are generally not catastrophic because the ADAS system alerts the driver and allows the driver to decide whether a safe condition exists and act accordingly. However, in the autonomous vehicle 1600, the vehicle 1600 itself must decide whether to pay attention to the results from the main computer or the auxiliary computer (e.g., the first controller 1636 or the second controller 1636) in the event of conflicting results. For example, in some embodiments, the ADAS system 1638 can be a backup and / or auxiliary computer for providing perception information to the backup computer rationality module. The backup computer rationality monitor can run redundant diversity software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1638 can be provided to the monitoring MCU. If the outputs of the main computer and the auxiliary computer conflict, the monitoring MCU must determine how to coordinate the conflict to ensure safe operation.

[0226] In some examples, the main computer can be configured to provide a confidence score to the monitoring MCU, indicating the main computer's confidence in the selected result. If the confidence score exceeds a threshold, the monitoring MCU can follow the direction of the main computer, regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence does not meet the threshold, and the main computer and the auxiliary computer display different results (e.g., conflict), the monitoring MCU can arbitrate between the computers to determine the appropriate result.

[0227] The monitoring MCU may be configured to run a neural network that is trained and configured to determine under what circumstances the secondary computer provides a false alarm based on outputs from the primary computer and the secondary computer. Thus, the neural network in the monitoring MCU may learn when the output of the secondary computer can be trusted and when it cannot. For example, when the secondary computer is a radar-based FCW system, the neural network in the monitoring MCU may learn when the FCW system identifies a metal object that is not actually dangerous, such as a drain grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based LDW system, the neural network in the monitoring MCU may learn to override the LDW in the presence of a cyclist or pedestrian and that actually deviating from the lane is the safest action. In an embodiment that includes a neural network running on the monitoring MCU, the monitoring MCU may include at least one of a DLA or a GPU suitable for running a neural network with an associated memory. In a preferred embodiment, the monitoring MCU may include and / or be included as a component of the SoC 1604.

[0228] In other examples, the ADAS system 1638 may include an auxiliary computer that uses traditional computer vision rules to perform ADAS functions. Thus, the auxiliary computer can use classic computer vision rules (if-then), and the presence of a neural network in the monitoring MCU can improve reliability, safety, and performance. For example, different implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functions. For example, if there is a software defect or error in the software running on the main computer, and the non-identical software code running on the auxiliary computer provides the same overall result, the monitoring MCU can have greater confidence that the overall result is correct, and errors in the software or hardware on the main computer will not cause significant errors.

[0229] In some examples, the output of the ADAS system 1638 can be fed into the perception block of the main computer and / or the dynamic driving task block of the main computer. For example, if the ADAS system 1638 indicates a forward collision warning due to an object directly ahead, the perception block can use this information in identifying the object. In other examples, as described herein, the auxiliary computer can have its own neural network that is trained to reduce the risk of false positives.

[0230] The vehicle 1600 may also include an infotainment SoC 1630 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1630 may include a combination of hardware and software that can be used to provide audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), phone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, vehicle-related information such as fuel level, total distance traveled, brake fuel level, engine oil level, door open / closed, air filter information, etc.) to the vehicle 1600. For example, the infotainment SoC 1630 may be a radio, a disk player, a navigation system, a video player, USB and Bluetooth connectivity, an onboard computer, onboard entertainment, Wi-Fi, steering wheel audio controls, hands-free voice control, a head-up display (HUD), a human-machine interface (HMI) display 1634, a remote communication device, a control panel (e.g., for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1630 may also be used to provide information (e.g., visual and / or auditory) to a user of the vehicle, such as information from an ADAS system 1638, autonomous driving information, such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0231] The infotainment SoC 1630 may include GPU functionality. The infotainment SoC 1630 may communicate with other devices, systems, and / or components of the vehicle 1600 via a bus 1602 (e.g., a CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1630 may be coupled to a monitoring MCU so that when a main controller 1636 (e.g., a main computer and / or backup computer of the vehicle 1600) fails, the GPU of the infotainment system may perform some self-driving functions. In this example, the infotainment SoC 1630 may place the vehicle 1600 in a driver-to-safety parking mode as described herein.

[0232] The vehicle 1600 may also include an instrument panel 1632 (e.g., a digital instrument panel, an electronic instrument panel, a digital instrument panel, etc.). The instrument panel 1632 may include a controller and / or a supercomputer (e.g., a discrete controller or a supercomputer). The instrument panel 1632 may include a set of instruments such as a speedometer, a fuel level, an oil pressure, a tachometer, an odometer, a turn indicator light, a shift position indicator light, a seat belt warning light, a parking brake warning light, an engine fault indicator light, an airbag (SRS) system information, lighting control, safety system control, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 1630 and the instrument panel 1632. In other words, the instrument cluster 1632 may be included as part of the infotainment SoC 1630, and vice versa.

[0233] Fig.16D According to some embodiments of the present disclosure, a method for Fig.16A 16. System diagram of an exemplary autonomous vehicle 1600 communicating with each other. System 1676 may include a server 1678, a network 1690, and a vehicle including vehicle 1600. Server 1678 may include multiple GPUs 1684(A)-1684(H) (collectively referred to herein as GPUs 1684), PCIe switches 1682(A)-1682(H) (collectively referred to herein as PCIe switches 1682), and / or CPUs 1680(A)-1680(B) (collectively referred to herein as CPUs 1680). GPUs 1684, CPUs 1680, and PCIe switches may be interconnected with a high-speed interconnect, such as, but not limited to, NVLink interface 1688 and / or PCIe connection 1686 developed by NVIDIA. In some examples, GPUs 1684 are connected via NVLink and / or NVSwitch SoCs, and GPUs 1684 and PCIe switches 1682 are connected via PCIe interconnects. Although eight GPUs 1684, two CPUs 1680, and two PCIe switches are illustrated, this is not intended to be limiting. Depending on the embodiment, each of the servers 1678 may include any number of GPUs 1684, CPUs 1680, and / or PCIe switches. For example, the servers 1678 may each include eight, sixteen, thirty-two, and / or more GPUs 1684.

[0234] Server 1678 may receive image data representing images from a vehicle via network 1690 that represent unexpected or changed road conditions, such as recently begun road work. Server 1678 may send neural network 1692, updated neural network 1692, and / or map information 1694 to the vehicle via network 1690, including information about traffic and road conditions. Updates to map information 1694 may include updates to HD map 1622, such as information about construction sites, potholes, detours, flooding, and / or other obstacles. In some examples, neural network 1692, updated neural network 1692, and / or map information 1694 may be generated as a result of new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed at a data center (e.g., using server 1678 and / or other servers).

[0235] Server 1678 can be used to train a machine learning model (e.g., a neural network) based on training data. The training data can be generated by the vehicle, and / or can be generated in simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., neural networks benefit from supervised learning) and / or otherwise preprocessed, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classifications of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternate dictionary learning), rule-based machine learning, anomaly detection, and any variants or combinations thereof. Once the machine learning model has completed training, the machine learning model can be used by the vehicle (e.g., transmitted to the vehicle via network 1690, and / or the machine learning model can be used by server 1678 to remotely monitor the vehicle.

[0236] In some examples, server 1678 may receive data from the vehicle and apply the data to the latest real-time neural network for real-time intelligent inference. Server 1678 may include a deep learning supercomputer and / or a dedicated AI computer powered by GPU 1684, such as DGX and DGX Station developed by NVIDIA. However, in some examples, server 1678 may include a deep learning infrastructure using only a CPU-driven data center.

[0237] The deep learning infrastructure of server 1678 may have the capability to make rapid, real-time inferences, and may use that capability to assess and validate the health of the processors, software, and / or associated hardware in vehicle 1600. For example, the deep learning infrastructure may receive periodic updates from vehicle 1600, such as a sequence of images and / or objects that vehicle 1600 has located in that sequence of images (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them to the objects identified by vehicle 1600, and if the results do not match and the infrastructure concludes that the AI ​​in vehicle 1600 has failed, server 1678 may send a signal to vehicle 1600 instructing the fail-safe computer of vehicle 1600 to take control, notify passengers, and complete a safe parking maneuver.

[0238] For inference, the server 1678 may include a GPU 1684 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-driven servers and inference acceleration may enable real-time responses. In other examples, such as where performance is less critical, servers driven by CPUs, FPGAs, and other processors may be used for inference.

[0239] Computing device examples

[0240] Fig.17 17 is a block diagram of an example computing device 1700 suitable for implementing some embodiments of the present invention. The computing device 1700 may include an interconnect system 1702 that directly or indirectly couples the following devices: memory 1704, one or more central processing units (CPUs) 1706, one or more graphics processing units (GPUs) 1708, communication interfaces 1710, input / output (I / O) ports 1712, input / output components 1714, power supplies 1716, one or more presentation components 1718 (e.g., displays), and one or more logic units 1720.

[0241] although Fig.17 The various blocks of are shown as being connected with lines via interconnect system 1702, but this is not intended to be limiting and is for clarity only. For example, in some embodiments, presentation components 1718 such as a display device may be considered I / O components 1714 (e.g., if the display is a touch screen). As another example, CPU 1706 and / or GPU 1708 may include memory (e.g., memory 1704 may represent a storage device in addition to the memory of GPU 1708, CPU 1706, and / or other components). In other words, Fig.17The computing devices described herein are illustrative only. No distinction is made between “workstations,” “servers,” “laptops,” “desktops,” “tablets,” “client devices,” “mobile devices,” “handheld devices,” “gaming consoles,” “electronic control units (ECUs),” “virtual reality systems,” and / or other device or system types, all of which are contemplated as being used in the context of Fig.17 within the range of computing devices.

[0242] The interconnection system 1702 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1702 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a video electronics standard association (VESA) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, there is a direct connection between components. For example, CPU 1706 may be directly connected to memory 1704. In addition, CPU 1706 may be directly connected to GPU 1708. In the case where there is a direct or point-to-point connection between components, the interconnection system 1702 may include a PCIe link for performing the connection. In these examples, the PCI bus does not need to be included in the computing device 1700.

[0243] Memory 1704 may include any of a variety of computer-readable media. Computer-readable media may be any available media that can be accessed by computing device 1700. Computer-readable media may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, computer-readable media may include computer storage media and communication media.

[0244] Computer storage media may include volatile and nonvolatile media and / or removable and non-removable media implemented in any method or technology for storage of computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1704 may store computer-readable instructions (e.g., representing programs and / or program elements, such as operating systems). Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tapes, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and can be accessed by computing device 1700. As used herein, computer storage media itself does not include signals.

[0245] Computer storage media may embody computer readable instructions, data structures, program modules, and / or other data types in a modulated data signal such as a carrier wave or other transport mechanism, and include any information transmission media. The term "modulated data signal" may refer to a signal having one or more of its characteristics set or changed in such a manner as to encode information in the signal. By way of example and not limitation, computer storage media may include wired media (such as a wired network or a direct wired connection) and wireless media (such as acoustic, RF, infrared, and other wireless media). Combinations of any of the above should also be included within the scope of computer readable media.

[0246] CPU 1706 can be configured to execute at least a portion of the computer-readable instructions to control one or more components of computing device 1700 to perform one or more methods and / or processes described herein. Each CPU 1706 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of processing multiple software threads simultaneously. CPU 1706 may include any type of processor, and may include different types of processors (e.g., processors with fewer cores for mobile devices and processors with more cores for servers) depending on the type of computing device 1700 implemented. For example, depending on the type of computing device 1700, the processor may be an Advanced RISC Machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or auxiliary coprocessors (e.g., math coprocessors), computing device 1700 may also include one or more CPU 1706.

[0247] In addition to or in lieu of CPU 1706, GPU 1708 may be configured to execute at least some computer-readable instructions to control one or more components of computing device 1700 to perform one or more methods and / or processes described herein. One or more of GPU 1708 may be an integrated GPU (e.g., with one or more of CPU 1706) and / or one or more of GPU 1708 may be a discrete GPU. In an embodiment, one or more of GPU 1708 may be a coprocessor of one or more of CPU 1706. GPU 1708 may be used by computing device 1700 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPU 1708 may be used for general-purpose computing on GPU (GPGPU). GPU 1708 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. GPU 1708 may generate pixel data for outputting an image in response to a rendering command (e.g., a rendering command received from CPU 1706 via a host interface). GPU 1708 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1704. GPU 1708 may include two or more GPUs operating in parallel (e.g., via a link). The link may connect the GPUs directly (e.g., using NVLINK) or through a switch (e.g., using NVSwitch). When combined together, each GPU 1708 may generate pixel data or GPGPU data for different parts of the output or different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory, or may share memory with other GPUs.

[0248] In addition to the CPU 1706 and / or the GPU 1708, the logic unit 1720 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1700 to perform at least some of the computer-readable instructions in one or more methods and / or processes described herein. In an embodiment, the CPU 1706, the GPU 1708, and / or the logic unit 1720 may perform any combination of methods, processes, and / or parts thereof discretely or jointly. One or more of the logic units 1720 may be a part of one or more of the CPU 1706 and / or the GPU 1708 and / or integrated in the CPU 1706 and / or the GPU 1708. The logic unit 1720 may be a discrete component or external to the CPU 1706 and / or the GPU 1708. In an embodiment, one or more of the logic units 1720 may be a coprocessor of one or more of the CPU 1706 and / or the GPU 1708.

[0249] Examples of logic unit 1720 include one or more processing cores and / or components thereof, such as a tensor core (TC), a tensor processing unit (TPU), a pixel visual core (PVC), a visual processing unit (VPU), a graphics processing cluster (GPC), a texture processing cluster (TPC), a streaming multiprocessor (SM), a tree traversal unit (TTU), an artificial intelligence accelerator (AIA), a deep learning accelerator (DLA), an arithmetic logic unit (ALU), an application specific integrated circuit (ASIC), a floating point unit (FPU), an input / output (I / O) element, a peripheral component interconnect (PCI) or a peripheral component interconnect express (PCIe) element, and the like.

[0250] The communication interface 1710 may include one or more receivers, transmitters, and / or transceivers to enable the computing device 1700 to communicate with other computing devices via an electronic communication network including wired and / or wireless communications. The communication interface 1710 may include components and functions to enable communication via, for example, a wireless network (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), a wired network (e.g., communication via Ethernet or InfiniBand technology), a low power wide area network (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0251] I / O ports 1712 can logically couple computing device 1700 to other devices including I / O components 1714, presentation components 1718, and / or other components, some of which can be built-in (e.g., integrated in) computing device 1700. Exemplary I / O components 1714 include microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dishes, scanners, printers, wireless devices, and the like. I / O components 1714 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by a user. In some cases, the input can be transmitted to an appropriate network element for further processing. NUI can implement any combination of voice recognition, stylus recognition, facial recognition, biometrics, gesture recognition on and near the screen, air gestures, head and eye tracking, and touch recognition (described in more detail below) associated with the display of computing device 1700. Computing device 1700 may include a depth camera for gesture detection and recognition, such as a stereo camera system, an infrared camera system, an RGB camera system, touch screen technology, and combinations thereof. Additionally, computing device 1700 may include an accelerometer or gyroscope capable of detecting motion (e.g., as part of an inertial measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by computing device 1700 to render immersive augmented reality or virtual reality.

[0252] The power supply 1716 may include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1716 may provide power to the computing device 1700 to enable the components of the computing device 1700 to operate.

[0253] Presentation component 1718 may include a display (e.g., a monitor, a touch screen, a television screen, a head-up display (HUD), other display types, or combinations thereof), speakers, and / or other rendering components. Presentation component 1718 may receive data from other components (e.g., GPU 1708, CPU 1706, etc.) and output data (e.g., as images, video, sound, etc.).

[0254] The present disclosure may be described in the general context of computer code or machine-usable instructions, including computer-executable instructions such as program modules executed by a computer or other machine (e.g., a personal data assistant or other handheld device). Typically, program modules include routines, programs, objects, components, data structures, etc., which refer to code that performs a specific task or implements a specific abstract data type. The present disclosure may be implemented in a variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. The present disclosure may also be implemented in a distributed computing environment, where tasks are performed by remote processing devices linked through a communication network.

[0255] As used herein, the statement of "and / or" with respect to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and element B. In addition, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and element B.

[0256] In order to satisfy statutory requirements, the subject matter of the present disclosure is described in detail herein. However, the description itself is not intended to limit the scope of the present disclosure. On the contrary, the inventors have contemplated that the claimed subject matter may also be implemented in other ways to include different steps or combinations of steps similar to the steps described in this document, as well as other existing or future technologies. In addition, although the terms "step" and / or "block" may be used to refer to different elements of the method employed, these terms should not be interpreted as implying any specific order between the various steps disclosed herein unless the order of the various steps is explicitly described.

Claims

1. An object detection method, comprising: converting the accumulated sensor data into motion-compensated sensor data corresponding to the position of the ego character at a particular time; projecting the motion compensated sensor data into a two-dimensional image space to generate first data representing a first two-dimensional view of an environment; extracting, based at least on the first data, using one or more neural networks, classification data representing one or more classifications of objects or scenes depicted in the first two-dimensional view; generating transformed classification data representing one or more classifications in a second two-dimensional view of the environment based at least on projecting the one or more classifications from the first two-dimensional view to the second two-dimensional view; as well as Based at least on the converted classification data, second data representing one or more bounding shapes of one or more objects detected in the environment is generated using the one or more neural networks. 2 . The method of claim 1 , wherein the first two-dimensional view is a perspective view and the second two-dimensional view is a top view.

3. The method of claim 1 , wherein the first data representing the first two-dimensional view of the environment comprises a projection of a lidar point cloud, the projection representing a perspective view of the environment, and wherein projecting the one or more classifications from the first two-dimensional view to the second two-dimensional view comprises: The one or more classifications are projected from the perspective view to a top-down view of the environment using the lidar point cloud.

4. The method of claim 1 , wherein the first data represents a lidar range image of the first two-dimensional view, and determining the first data comprises: Projecting a lidar point cloud into the lidar range image.

5. The method of claim 1 , wherein the first two-dimensional view is a lidar range image having pixel heights corresponding to a plurality of horizontal scan lines of a lidar sensor that captured the accumulated sensor data.

6. The method of claim 1, wherein the accumulated sensor data comprises sensor data accumulated over a period of time from one or more lidar sensors of the ego character, and the first two-dimensional view is a lidar range image of the environment.

7. The method of claim 1 , wherein projecting the one or more classifications from the first two-dimensional view to the second two-dimensional view comprises: A differentiable transformation is applied to the three-dimensional location associated with the classification data.

8. The method of claim 1, wherein the accumulated sensor data represents a lidar point cloud, wherein the converted classification data represents one or more confidence maps in the second two-dimensional view, and wherein the method further comprises: generating third data representing one or more height maps based at least on projecting the lidar point cloud into the second two-dimensional view; forming a tensor comprising a first set of one or more channels storing the converted classification data representing the one or more confidence maps and a second set of one or more channels storing the third data representing the one or more height maps; as well as Using the one or more neural networks, second classification data representing one or more second classifications in the second two-dimensional view and fourth data representing object instance geometry of the one or more objects are extracted from the tensor.

9. The method according to claim 1, further comprising: decoding outputs of the one or more neural networks to generate candidate boundary shapes for the one or more objects; identifying the second data representing the one or more bounding shapes of the one or more objects based on performing at least one of filtering or clustering on the candidate bounding boxes to remove duplicate candidate objects from the candidate bounding boxes; as well as A classification label is assigned to each of the one or more bounding shapes based on the outputs of the one or more neural networks.

10. The method of claim 1, wherein determining the second data representing the one or more boundary shapes comprises: decoding outputs of the one or more neural networks to generate candidate boundary shapes for the one or more objects; as well as The second data representing the one or more bounding shapes of the one or more objects is identified based on performing at least one of non-maximum suppression or density-based spatial clustering for a noisy application to remove duplicate candidate objects from the candidate bounding shapes.

11. The method of claim 1 , wherein the output of the one or more neural networks comprises a tensor storing regressed geometric data for each detected object, wherein determining the second data representing the one or more boundary shapes comprises: One or more three-dimensional boundary shapes are generated for the one or more objects from the regressed geometric data.

12. The method according to claim 1, further comprising: The one or more neural networks are trained using training data generated using annotation tracking for tracking an annotation object between two or more frames of corresponding sensor data.

13. The method according to claim 1, further comprising: The one or more neural networks are trained using training data generated using links between object trajectories generated for a particular object from corresponding sensor data from two or more sensors.

14. A method for object detection, comprising: receiving lidar data from one or more lidar sensors in an environment; generating first data representing a perspective view of the environment from the lidar data; generating classification data from the first data using one or more neural networks, the classification data representing one or more classifications in the perspective view; generating transformed classification data representing the one or more classifications in the overhead view by projecting the one or more classifications in the perspective view into the overhead view of the environment using the lidar data; as well as Based at least on the converted classification data in the top-down view, second data representing one or more bounding shapes of one or more objects detected in the environment is generated using the one or more neural networks.

15. The method of claim 14, wherein said generating first data representing a perspective view of said environment comprises: accessing cumulative sensor data accumulated over a period of time from the one or more lidar sensors of the ego character; converting the accumulated sensor data into motion compensated sensor data corresponding to the position of the ego character at a particular moment in time; as well as The motion compensated sensor data is projected into a two-dimensional image space to generate the first data representing a lidar range image of the perspective view of the environment.

16. The method of claim 14, wherein the one or more neural networks include a first stage configured to evaluate the first data representing the perspective view, and a second stage configured to evaluate the converted classified data representing the top-down view.

17. The method of claim 14, wherein the second data further represents a classification label for each of the one or more bounding shapes of the one or more objects.

18. A method for object detection, comprising: generating, using one or more neural networks, classification data representing one or more classifications of first two-dimensional points in the first two-dimensional view of the environment; associating the one or more classifications of the first two-dimensional point with corresponding three-dimensional locations from corresponding sensor data; projecting the one or more classifications of the first two-dimensional point from respective three-dimensional locations to second two-dimensional points in a second two-dimensional view of the environment to generate transformed classification data representing the one or more classifications in the second two-dimensional view; as well as Based at least on the converted classification data, second data representing one or more bounding shapes of one or more objects detected in the environment is generated using the one or more neural networks. The method of claim 18 , wherein the second data further represents one or more classification labels of the one or more objects.

20. The method according to claim 18, wherein: The generating of second data representing the one or more bounding shapes comprises generating the one or more bounding shapes and associated classification labels for the one or more objects based on second classification data representing one or more second classifications in the second two-dimensional view and third data representing object instance geometry, wherein the second classification data and the third data are extracted by the one or more neural networks.