Ground Truth Data Generation for Deep Neural Network Perception in Autonomous Driving Applications

Abstract

To provide a system and a method for an annotation pipeline.SOLUTION: In an annotation pipeline 100 used to produce 2D and / or 3D ground truth data for deep neural networks (DNNs), initially sensor data are captured with different types of sensors and synchronized to align frames of sensor data that represent a similar world state. The aligned frames are sampled and packaged into a sequence of annotation scenes to be annotated. An annotation project may be decomposed into modular tasks and encoded into a labeling tool. During the tasks, each type of sensor data in an annotation scene may be simultaneously presented. Information may be projected across sensor modalities to provide useful contextual information. After all annotation tasks have been completed, the resulting ground truth data may be exported in any suitable format.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to ground truth data generation for deep neural network perception in autonomous driving applications.

Background Art

[0002] Designing a system for autonomously and safely driving a vehicle without supervision is extremely difficult. An autonomous vehicle has a remarkable ability to move in a careful - complex environment, identify and react to static obstacles, and avoid collisions with other objects or structures along the vehicle's path - using a perception and action system that should at least have the ability to act as a functional equivalent of a driver. Thus, the ability to detect instances of environmental organisms (e.g., passenger cars, pedestrians, etc.) and other parts is often essential for an autonomous driving perception system. Conventional perception methods often rely on cameras or LiDAR sensors to detect objects in the environment, and various techniques for performing LiDAR and camera perception using a deep neural network (DNN) have been developed. Such classes of DNNs include DNNs that perform panoramic segmentation of camera images in perspective views and DNNs that perform top - down or “bird's eye view” (BEV) object detection from LiDAR point clouds.

[0003] To train a DNN to perform perception with appropriate accuracy, the DNN needs to be trained with accurate ground truth data. Real - time DNN perception is usually performed in two dimensions (2D) due to computational constraints, and thus, the ground truth for these networks is usually given in 2D. However, as technology progresses, three - dimensional (3D) perception is starting to become practical, and there is an unmet need for high - quality 3D ground truth data.

[0004] Conventional techniques for generating ground truth for DNN perception in autonomous driving applications have various drawbacks. Consider the aforementioned example where the first DNN performs panoramic segmentation of a camera image in perspective, and the second DNN performs object detection from a top-down projection of a LiDAR point cloud. In this case, the first DNN requires a camera image with ground truth annotations, and the second DNN requires a top-down LiDAR projection with ground truth annotations. Traditionally, these types of ground truth annotations are generated in a separate labeling process. However, in certain situations, generating accurate labels is difficult or even impossible. Taking LiDAR and RADAR labeling as examples, these modalities generate sparse data and sometimes lack the granularity and context required to apply accurate labels. In a top-down view, the top-down view of these objects often resembles the top-down view of other objects such as poles, tree trunks, or bushes, making it difficult or even impossible to distinguish between pedestrians and cyclists. As a result, conventional labeling techniques can yield erroneous ground truth data. Ideally, these errors would be discovered during quality checks, but they still negatively impact throughput and efficiency, as well as wasting computational resources. [Overview of the project] [Means for solving the problem]

[0005] Embodiments of this disclosure relate to annotation pipelines that generate 2D and / or 3D ground truth data for deep neural networks (DNNs), such as those that perform perception in autonomous or semi-autonomous vehicles, robots, or other types of objects.

[0006] In general, the annotation pipeline described herein is an improved workflow and software interface that streamlines the production of high-quality ground truth data. Initially, sensor data may be captured by different types of sensors (sensor modalities) during a capture session. Data from different sensors may be synchronized to align frames of sensor data representing similar world conditions. In one example involving LiDAR and cameras, as a LiDAR spin travels through and views different parts of the environment, the temporally closest camera frame for any given LiDAR spin may be selected based on the camera's field of view relative to the LiDAR spin initiation angle and how long it takes for the LiDAR spin to align with the camera's field of view. In some embodiments, a camera-specific time or index offset relative to the LiDAR spin initiation may be determined and / or applied to align the LiDAR frame with the camera frame of each camera. In general, frames of different types of sensor data may be aligned, sampled, and packaged into a sequence of annotation scenes to be annotated.

[0007] In some embodiments, an annotation project can be broken down into modular tasks that can be assigned to different labelers. In non-limiting embodiments including cameras and LiDAR, first some or all camera images in a sequence can be annotated, then some or all LiDAR frames in the sequence can be annotated (e.g., a first top-down 2D bounding box, then a 3D bounding box), objects appearing in a sequence of annotation scenes can be linked, and then objects appearing in both sensor modalities (LiDAR and camera frames) can be linked together. An annotation task can be encoded into a labeling tool that arranges the order of inputs using a wizard that assigns the task to a labeler and guides the labeler through the task. During the annotation task, each type of sensor data in the annotation scene can be presented to the labeler (e.g., side-by-side), and / or the information can be projected across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data. In some embodiments, the labeling tool can guide the labeler through an object-by-object annotation procedure for each annotation scene in the sequence.

[0008] After some or all annotation tasks within an annotation project are completed, the resulting ground truth data can be exported in any suitable format. Thus, one or more machine learning models can be trained using the exported ground truth data.

[0009] The system and method for generating ground truth data for deep neural network perception are described in detail below with reference to the attached diagrams. [Brief explanation of the drawing]

[0010] [Figure 1] This is a data flow diagram illustrating an exemplary annotation pipeline in some embodiments of the present disclosure. [Figure 2] This figure shows exemplary alignment between LiDAR spin and two cameras according to some embodiments of the present disclosure. [Figure 3] This is a table illustrating exemplary alignment of sensor data indices according to some embodiments of the present disclosure. [Figure 4] This figure shows an example of a data capture session, an annotation scene, and a section of the annotation scene according to some embodiments of the present disclosure. [Figure 5] This is an illustration illustrating an exemplary user interface for image labeling according to some embodiments of the present disclosure. [Figure 6] This illustration shows an exemplary user interface for camera-assisted LiDAR labeling according to some embodiments of the present disclosure. [Figure 7] This illustration shows an exemplary user interface for camera-assisted LiDAR labeling with orientation vectors, according to some embodiments of the present disclosure. [Figure 8] Illustrations illustrating exemplary user interfaces for camera-assisted LiDAR labeling with a three-dimensional (3D) bounding box, according to some embodiments of the present disclosure. [Figure 9] Illustrations of exemplary user interfaces for LiDAR tracking according to some embodiments of the present disclosure. [Figure 10] Illustrations of exemplary user interfaces for camera-LiDAR link configuration according to some embodiments of the present disclosure. [Figure 11] Illustrations of exemplary ground truth annotations, as illustrated in some embodiments of the present disclosure. [Figure 12] This flowchart illustrates a method for generating ground truth annotations for sensor data from different types of sensors, according to several embodiments of the present disclosure. [Figure 13]This flowchart illustrates a method for generating ground truth annotations for LiDAR and camera frames according to some embodiments of the present disclosure. [Figure 14A] Illustrations of exemplary autonomous vehicles according to some embodiments of the present disclosure. [Figure 14B] Figure 14A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 14C] Figure 14A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 14D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 14A, according to some embodiments of the present disclosure. [Figure 15] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 16] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure. [Modes for carrying out the invention]

[0011] One potential solution to the aforementioned problem is the projection of information across sensor modalities, which can assist Labella by providing useful contextual information. However, this possibility presents several challenges. Traditionally, ground-truth LiDAR annotations have been limited to top-down 2D bounding boxes, but top-down bounding boxes do not project well into camera space and may not provide enough additional context to assist Labella. Furthermore, it is difficult when it is impossible to achieve perfect temporal alignment between different sensor modalities. In an ideal scenario, even assuming a configuration with multiple sensors (e.g., camera and LiDAR sensor), the trigger is applied at a specific point in time and all sensors start simultaneously. In reality, this ideal scenario is nearly impossible due to the different types of sensors, differences in delay lines, differences in sampling frequencies (e.g., camera operating at 30fps vs. LiDAR operating at 10fps), and the challenge of synchronizing cameras, similar devices. Taking LiDAR and cameras as specific examples, LiDAR sensors take time to spin (e.g., 100 milliseconds / rotation), and therefore, perfectly synchronizing a particular camera with respect to the LiDAR spin position is difficult, and often practically impossible. Since individual calibration of each sensor often cannot restore all the degrees of freedom required for perfect alignment, there may also be practical limitations to spatial alignment between sensors. Consequently, projecting information across sensor modalities can mix information from different world states, potentially negating any benefits.

[0012] To address these and other challenges, systems and methods relating to annotation pipelines for generating 2D and / or 3D ground truth data for deep neural networks (DNNs), such as those performing perception in autonomous or semi-autonomous vehicles, robots, or other object types are disclosed. While this disclosure may be described in relation to an exemplary autonomous vehicle 1400 (an example thereof is described herein with respect to Figures 14A–14D and referred to herein as "Vehicle 1400" or "Self-Vehicle 1400"), this is not intended to be limiting. For example, the systems and methods described herein may be used to generate ground truth training data for DNNs in non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), robots, warehouse vehicles, off-road vehicles, flying ships, boats, and / or other types of vehicles. In addition, while this disclosure may be described in relation to autonomous driving, this is not intended to be limiting. For example, the systems and methods described herein may be used to generate training data for DNNs in robotics (e.g., route planning for robots), aerospace systems (e.g., route planning for drones or other aircraft), ship systems (e.g., route planning for boats or other vessels), and / or other technical fields for localization, route planning, and / or other processes.

[0013] Generally, the annotation pipeline described herein is an improved workflow and software interface that rationalizes the production of higher quality ground truth data than prior art. First, sensor data is captured by different types of sensors (sensor modalities) during a capture session. To identify useful context information that can assist in annotation tasks, data from different sensor modalities is synchronized to form a sequence of annotation scenes (e.g., a set of sensor data acquired at approximately that time). Desired segments of the sequence can be selected and designed for labeling, and the desired annotations can be decomposed into a set of linear tasks. Different tasks can be split based on sensor type, type of object being labeled, level of annotation detail, and / or others. The tasks can be input into a labeling tool or otherwise encoded using a wizard that assigns tasks to labelers and arranges the order of the input through the tasks. During the annotation tasks, each type of sensor data in the annotation scene can be presented to the labeler (e.g., side by side), and / or the information can be projected across sensor modalities to provide useful context information, e.g., correspondences between different types of sensor data. In some cases, some types of annotations can be automatically generated. When some or all of the annotation tasks are complete, the labeling tool can export the resulting ground truth annotations in any suitable format.

[0014] In some embodiments, an annotation project can be broken down into modular tasks that can be assigned to different labelers. In non-limiting embodiments including cameras and LiDAR, some or all camera images in a sequence may first be labeled, then some or all LiDAR frames in the sequence may be labeled (e.g., a first top-down 2D bounding box, then a 3D bounding box), objects appearing in a sequential annotation scene may be linked, then objects appearing in both sensor modalities may be linked together. Additionally or alternatively, an annotation project can be divided into multiple stages or tasks based on the type of label (e.g., in a given task, label only a car, a traffic sign, or some other elements in a road scene), the level of detail (e.g., in a given task, apply only polylines of object footprints or apply a full 3D bounding box), and / or in other ways. By breaking down an annotation project and prompting labelers to perform tasks specific to individual projects, the annotation pipeline can be modularized, so that labelers can focus on individual tasks at a time.

[0015] During any given annotation task, the labeling tool's interface can present different types of sensor data in the annotation scene (e.g., side-by-side) and / or project information across sensor modalities to provide useful context information. The projected information can include sensor detections (e.g., points, planes, scan lines), annotations, input probes that specify a particular location within the frame of the sensor data, and / or other information. By presenting context information during the annotation task, the labeler can apply more accurate labels. For example, a group of pedestrians may be represented in a particular frame of LiDAR data that has only a few detections, but by presenting the corresponding camera frame, the labeler can easily see pedestrians that are hidden in the LiDAR data. Similarly, if the labeler is unsure which type of obstacle is represented by a particular LiDAR detection, in some embodiments, the labeler can click on the LiDAR detection and visualization can be applied at the corresponding point in the camera frame (e.g., to identify a pickup truck). The labeler can then apply the appropriate label on the LiDAR frame.

[0016] The labeling tools and annotation pipelines described herein can offer various benefits beyond prior art. In general, providing additional context to a human labeler viewing sensor data improves the accuracy and efficiency of manual labeling and enables several types of ground truth labels that were previously impossible. For example, by presenting LiDAR data with additional context, such as (stitched) camera images, the labeler can accurately generate 3D LiDAR labels, such as 3D bounding boxes. With 3D bounding boxes annotated in LiDAR space, these labels can be projected onto the corresponding camera images to provide useful context for camera labeling, enabling the labeler to generate more accurate camera labels. As a result, the techniques described herein provide a specification for a more accurate representation of ground truth that can be used to train more accurate DNNs.

[0017] Furthermore, presenting additional context to the labeler during the labeling process generally makes it easier for the labeler to produce labels. For example, presenting side-by-side views of different types of sensor data and / or demonstrating correspondences across sensor modalities makes it easier for the labeler to understand the data by reducing cognitive load and increasing the speed of recognition. As a result, the labeling tools described herein make the interaction between the labeler and the computer more efficient compared to prior art. This improvement helps the entire workflow in terms of efficiency and throughput, and consequently reduces the need for computational resources compared to prior art. In other words, the better the tool, the more efficient the labeling becomes and the higher the throughput. By making the tool easier to use, various aspects of this technique speed up the annotation workflow by reducing the amount of rework compared to prior art.

[0018] Furthermore, breaking down annotation tasks into more easily digestible tasks and decomposing annotation projects streamlines the annotation process and reduces the occurrence of information overload. Traditionally, many companies have instructed labelers to perform all applicable annotation tasks on each frame of sensor data at once, resulting in a complex labeling process that requires considerable training. By breaking down the required annotations into smaller, more digestible tasks, tasks can be assigned to a larger number of labelers, making the labeling pipeline more scalable. In addition, by dividing annotation tasks by sensor type, the labeling process becomes less susceptible to temporal or spatial misalignments between sensor modalities. For example, even if the LiDAR frame and the corresponding camera frame cannot be perfectly aligned, the LiDAR annotation task is performed directly on the LiDAR frame, minimizing the impact on label accuracy. In this example, the corresponding camera frame is simply used as flexible guidance, and therefore the misalignment is not hardcoded as ground truth. Finally, in embodiments that include a linking task in which the labeller associates previously created annotations across different sensor modalities and / or annotation scenes, by placing this task after the others, the labeller is in a suitable position to review annotations from both sensor modalities and / or annotation scenes, so the linking task can perform a quality check function. As a result, using this linking task eliminates the need to perform a separate quality check and further improves the overall workflow.

[0019] As such, the labeling tools and annotation pipelines described herein can be used to generate more accurate representations of ground truth, which can be used to train more accurate DNNs.

[0020] Exemplary Ground Truth Annotation Pipeline Deep neural networks (DNNs) can be used for a variety of tasks, such as object detection and classification. An annotation pipeline can be used to generate ground truth data to obtain training data for these DNNs. Generally, the ground truth data generated by the annotation pipeline can be customized based on the type of DNN to be trained. For DNNs performing perception, the annotation pipeline can be customized to generate 2D and / or 3D ground truth data, such as labeled camera images (e.g., in perspective views), LiDAR or RADAR data (e.g., in top-down views), and / or other sensor data. While certain embodiments are described in relation to DNNs performing perception, the techniques described herein can be adapted to generate ground truth data for other types of DNNs.

[0021] Referring to Figure 1, Figure 1 is a data flow diagram showing an exemplary annotation pipeline 100 according to some embodiments of the present disclosure. It should be understood that this and other configurations described herein are provided merely as examples. Other configurations and elements (e.g., machines, interfaces, functions, sequences, function groupings, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0022] At a high level, the annotation pipeline 100 may include workflows and software interfaces that streamline the production of high-quality ground truth data. In the example shown in Figure 1, the annotation pipeline 100 includes data capture 110, sensor data alignment 120, scene generation 130, scene curation 140, annotation 150, post-processing 160, quality assurance check 170, ground truth data export 180, and ground truth data consumption 190. For example, real-world data capture 110 may be performed to collect sensor data from different types of sensors (sensor modalities), and sensor data alignment 120 of different types of sensor data may be performed to synchronize sensor data so that sensor data of similar world states (e.g., sensor data captured at approximately the same time) can be grouped together and presented during the annotation task. Scene generation 130 may be used to construct a sequence of annotation scenes (e.g., a set of sensor data acquired at approximately the same time, e.g., scene sequence 145). For example, sensor data alignment 120 may include adding an offset to the sensor data capture time or some other index, and scene generation 130 may include sampling sensor data and / or generating projected images of each annotation scene in the sequence. Scene curation 140 may be performed to select one or more segments of annotation scenes (for example, segments of data capture sessions without rain) and to specify the annotation segments.

[0023] At a high level, software tools such as web tools (also called labeling tools) can be used to facilitate annotation. Generally, a particular annotation project can be broken down and / or arranged into a set of linear tasks that form a labeling recipe, which can divide the project and define different tasks based on sensor type, type of object being labeled, level of annotation detail, and / or other factors. Tasks can be entered into a labeling tool or otherwise encoded, which can arrange the order of inputs using a wizard that assigns tasks to the labeler and guides the labeler through the annotation tasks. In an exemplary annotation project for LiDAR and camera ground truth data, first some or all camera images in a sequence are labeled, then some or all LiDAR frames in a sequence are labeled (e.g., camera and LiDAR output 152), then objects appearing in multiple LiDAR frames are associated with each other (e.g., LiDAR tracking output 154), and then objects appearing in multiple sensor modalities are linked together (e.g., camera + LiDAR link setting output 156). In some embodiments, LiDAR labeling may include first labeling LiDAR frames having a 2D bounding box, and then labeling LiDAR frames having a 3D bounding box or a rectangular prism (e.g., 3D LiDAR labeling output 155). The labeling tools and this exemplary annotation project will be described in more detail later.

[0024] Continuing with the high-level overview, after Labella completes annotation tasks in the project, post-processing 160 may be applied to generate annotations that humans would not normally be able to produce, such as depth values. In some embodiments, quality assurance checks 170 may be performed on the labeled data to identify potential errors and tag certain annotation scenes for rework. In some embodiments, quality assurance checks 170 may be at least partially incorporated into annotation tasks, such as annotation 150, for example, linking tasks, during annotation tasks, eliminating the need for a separate quality assurance check after post-processing 160. After some or all annotation tasks are completed, the resulting ground truth data may be exported from the labeling tool in any suitable format, whether automatically or manually triggered (ground truth data export 180), and the ground truth data may be consumed, for example, by using the ground truth data to train a corresponding DNN (ground truth data consumption 190).

[0025] Generally, ground truth data can be generated at least partially from real-world data. Therefore, in order to perform data capture 110 in some embodiments, one or more vehicles (e.g., vehicles 1400 in Figures 14A-D) can collect sensor data from one or more sensors of the vehicle in a real-world (e.g., physical) environment. Vehicle sensors may include, but are not limited to, global navigation satellite system sensors 1458 (e.g., global positioning system sensors), RADAR sensors 1460, ultrasonic sensors 1462, LiDAR sensors 1464, inertial measurement unit (IMU) sensors 1466 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), self-motion sensors, microphones 1496, stereo cameras 1468, wide-view cameras 1470 (e.g., fisheye cameras), infrared cameras 1472, surround cameras 1474 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1498, speed sensors 1444 (e.g., for measuring vehicle speed), vibration sensors 1442, steering sensors 1440, brake sensors (e.g., as part of a brake sensor system 1446), and / or other sensor types. Vehicles may include autonomous vehicles, semi-autonomous vehicles, non-autonomous vehicles, and / or objects or vehicles other than vehicles 1400, such as robots, drones, ships, airplanes, unmanned aerial vehicles (UAVs), etc.

[0026] A vehicle may include various types of vehicle hardware. For example, vehicle hardware may be responsible for managing sensor data generated by sensors (for example, using a sensor manager in an autonomous driving software stack run by the vehicle hardware). In some embodiments, a vehicle may include an autonomous driving software stack having a world state manager that manages the world using one or more maps (e.g., 3D maps), localization components, perception components, and / or similar. The autonomous driving software stack may include planning components (e.g., as part of a planning layer), control components (e.g., as part of a control layer), actuator components (e.g., as part of an actuator layer), obstacle avoidance components (e.g., as part of an obstacle avoidance layer), and / or other components. Generally, in some embodiments, vehicle hardware may include hardware used to control the vehicle through the real-world environment based on sensor data, one or more machine learning models (e.g., neural networks), and / or similar. As such, various types of vehicle hardware may be configured for use by the vehicle when running an autonomous driving software stack that controls, at least partially, the vehicle's navigation through the real-world physical environment and / or for installation within the vehicle.

[0027] Generally, data capture 110 may include capturing sensor data by observing the real-world environment with different types of sensors (sensor modalities), such as LiDAR and one or more cameras mounted on a vehicle. Generally, sensor data may be acquired from different sensors at different frequencies for various reasons, such as differences in delay lines, differences in sampling frequencies (e.g., a camera operating at 30 fps versus a LiDAR operating at 10 fps), different trigger times, and other reasons. To facilitate grouping and present sensor data of similar world conditions (e.g., sensor data captured approximately simultaneously), sensor data alignment 120 may be performed to synchronize sensor data from different sensor modalities. In some embodiments, a specific sensor may be used as a reference sensor. Non-reference sensors may be referred to as child sensors. For a given frame of sensor data from a reference sensor (reference frame), an offset such as a time delta may be identified between the reference frame and the temporally closest frame of sensor data from each child sensor. The offset of each child sensor may be recorded and / or applied to the capture time of sensor data from the child sensor or some other index.

[0028] As a specific example, LiDAR and cameras are given. In some embodiments, the LiDAR may be selected as a reference sensor, the temporally closest frame from each camera may be selected based on the camera's field of view relative to the LiDAR spin start angle, and a camera-specific time or index offset may be applied in relation to the LiDAR spin start. Figure 2 shows an exemplary alignment between a LiDAR spin and two cameras according to some embodiments of the present disclosure. Consider a completed LiDAR spin with a reference time stamp t pointing to when the spin started (t+0 ms). Using an exemplary LiDAR sensor, assume that it takes 100 ms to complete the spin (spin duration) and the spin frequency is 10 Hz. In Figure 2, the exemplary LiDAR frame 210 is superimposed on a circle to represent the progress of the LiDAR spin over time. In this example, arrow 220 represents the orientation of the center of the field of view of the LiDAR spin at time t+0 ms when the spin started. The spin moves clockwise to arrow 225 at t+50ms, to arrow 230 at t+75ms, and returns to arrow 220 at t+100ms, indicating where the spin ended.

[0029] In Figure 2, the fields of view of the two cameras, the forward-facing camera 240 and the backward-facing camera 250, are also superimposed on the LiDAR frame 210. To identify the frame from each camera that is temporally closest to the LiDAR frame 210, the progress and orientation of the LiDAR spin for each camera may be determined, and an offset related to the LiDAR may be determined for each camera based on when the LiDAR spin reaches the camera's field of view. For example, at t+50ms (arrow 225), the LiDAR spin has progressed into the field of view of the forward-facing camera 240, and therefore an offset of 50ms may be used for the forward-facing camera 240. For some child sensors, such as the backward-facing camera 250, the sensor data observed by the child sensor may have sectors corresponding to disprime or incomplete parts of the LiDAR spin, and therefore it may be impossible to select a particular frame of sensor data from the child sensor that is perfectly aligned with the LiDAR spin. In general, any frame of sensor data from a child sensor may be selected. In the example shown in Figure 2, the frame captured at or near the end of the LiDAR spin at t+100ms (arrow 220) may be selected.

[0030] Exemplary frame selections and corresponding offsets are shown on axis 260, having reference frame 270 (e.g., representing LiDAR frame 210), frame 280 (e.g., representing forward-facing camera 240), and frame 290 (e.g., representing rear-facing camera 250). Note that the temporally closest frame of sensor data captured by a child sensor may not perfectly align with the portion of the LiDAR spin representing the center of the child sensor's field of view. For example, a LiDAR frame (e.g., reference frame 270) may represent a world state that would correspond to an image from a forward-facing camera captured at 50 ms, but the temporally closest image captured by that camera may have been captured at 40 or 45 ms (e.g., frame 280). In that case, the image captured at 45 ms may be selected and paired with the LiDAR frame, and / or the corresponding offset (e.g., 45 ms) may be identified and associated with the corresponding child sensor (e.g., forward-facing camera 240).

[0031] Generally, an identified offset for each child sensor may be recorded and / or applied to the capture time of sensor data from the child sensor or to other indices. For example, assume that the sensor setup produces raw sensor data, where the sensor data from each sensor is individually indexed. In some embodiments, an identified offset for a particular child sensor may be applied to adjust the index of the child sensor's sensor data (or to identify the index of aligned sensor data). Thus, determining and / or applying a sensor-specific offset may serve to align different types of sensor data (for example, by aligning their indices). Figure 3 is a table showing exemplary alignment of sensor data indices according to some embodiments of the present disclosure. In this example, the sensor data indices are shown in column 320 (LiDAR frame index) and columns 330-360 (index of images captured by different cameras). In this example, the chronological scene index is contained in column 310, and for each scene index value, the corresponding row in the table contains an index that identifies the sorted sensor data (e.g., for a particular annotation scene). In this example, the identified offset for each child sensor (e.g., each camera) is applied relative to each reference frame to identify the corresponding index of the temporally closest frame of sensor data captured by the child sensor.

[0032] Returning to Figure 1, scene generation 130 may be performed to construct a sequence of annotation scenes (e.g., a set of sensor data acquired at approximately that time, e.g., scene sequence 145). Generally, reference and child sensor data may be sampled using identified offsets. If the raw sensor data is not in an image format (e.g., LiDAR or RADAR point cloud), in some embodiments, the raw sensor data (e.g., point cloud) may be projected to form a projected image (e.g., a top-down image). Continuing with the example where LiDAR is used as the reference sensor and cameras are used as child sensors, for each frame of LiDAR data (e.g., LiDAR point cloud), an annotation scene may be constructed by projecting the LiDAR point cloud to generate a projected image (e.g., a top-down image) and a sampled image (e.g., based on the corresponding offset and / or index) from each camera to identify the temporally closest image captured by each camera. Projected images (LiDAR frames) and camera images (camera frames and / or composite images or panoramas stitched together from multiple images) can be packaged, grouped, or otherwise associated with each other as annotation scenes. The process may be repeated, for example, to generate or otherwise identify annotation scenes for each reference frame. In some embodiments, scene curation 140 may be performed to select one or more segments of annotation scenes (e.g., segments of data capture sessions without rain) and to specify the segments for labeling.

[0033] Figure 4 shows an example of a data capture session 410, annotation scene 415, and annotation scene segments 421, 422, and 423 according to some embodiments of the present disclosure. For example, assume that the data capture session 410 is performed through the downtown area of ​​a large city, and that the sensor periodically captures sensor data during the data capture session 410. In a non-limiting example, LiDAR spins may occur every 100 ms, and one set of cameras may each take a picture once every 25 ms (or some other increment). Scene 415 may consist of captured data (dividers between consecutive scenes are not shown in Figure 4). In some cases, not all scenes are useful. For example, in a city that frequently experiences rainfall, scene 415 may be filtered to remove scenes 415 captured when it rained during the data capture session 410 (for example, because raindrops were present on the windshield or lens). In general, any type of filter may be applied (for example, automatically or manually, based on numerical or visual features, metadata tags, time stamps, or otherwise). For example, scene 415 may be curated to identify a specific segment of the subject (for example, segments 421, 422, 423), and scenes within the identified segment (for example, sensor data from different sensor modalities) may be designated for annotation (for example, tagged or otherwise identified).

[0034] Returning to Figure 1, annotation 150 can be performed by a human labeler using a software tool (also called a labeling tool), such as a web tool. Generally, the software tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project manager identifying and / or providing the annotation scene to be labeled (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotation may be broken down and / or arranged into a set of linear tasks that form a labeling recipe, and the encoded representation of the tasks may be input into the labeling tool. Different tasks may be divided based on sensor type, type or class of object to be labeled (e.g., labeling only cars, traffic signs, or any other elements in a given road scene in a given task), level of annotation detail (e.g., label bounding box versus silhouette, applying only polylines of object footprints in a given task, applying a full 3D bounding box, applying a top-down 2D bounding box in LiDAR and then upgrading to 3D in subsequent passes), and / or in other ways—and the labeling tool may be configured to divide and / or encode tasks. In some embodiments, separate tasks may be input for labeling obstacles, vehicles (e.g., cars, buses, trucks, etc.), vulnerable road users (e.g., motorcycles, bicycles, pedestrians, etc.), parts of the environment (e.g., drivable space, sidewalks, buildings, trees, poles, etc.), their subclasses (e.g., pedestrians walking), any combination thereof, and / or other.

[0035] Annotation tasks can be input into a labeling tool or encoded in other ways, and the labeling tool can combine the performance of different annotation tasks. For example, the labeling tool can assign tasks to a labeler by assigning tasks in any suitable manner, for example, based on labeler availability, a specified task order, or other factors. In some embodiments, the labeling tool can arrange the order of inputs (e.g., annotations) for a particular task using a wizard that guides the labeler through the tasks. Between several tasks, each type of sensor data in the annotation scene can be presented to the labeler (e.g., side by side), and / or the information can be projected across sensor modalities to provide useful contextual information, for example, correspondences between different types of sensor data.

[0036] Generally, labeling tools can accept inputs that define ground truth annotations (e.g., boundaries, enclosed areas, class labels), and can associate sensor data with annotations. Sensor data (e.g., frames of LiDAR data, RGB images) can be annotated with labels or other markers that identify the location, geometry, orientation, and / or class of instances of relevant objects in the sensor data (e.g., manually, automatically, etc.). Annotations can be input into labeling tools using 2D and / or 3D drawing capabilities, other types of appropriate software capabilities, and / or hand-drawn and imported. Generally, annotations can be manufactured synthetically (e.g., generated from computer models or renderings), manufactured in reality (e.g., designed and manufactured from real-world data), automated by machine (e.g., using feature analysis and learning to extract features from data and then generate labels), annotated by humans (e.g., a labeler or annotation expert inputting annotations), and / or a combination thereof (e.g., a human identifying the vertices of a polyline, and a machine generating polygons using a polygon rasterizer). Generally, annotations may include 2D and / or 3D bounding boxes, closed polylines, or other bounding shapes that are drawn, annotated, superimposed, and / or otherwise associated with sensor data.

[0037] As a non-limiting example including cameras and LiDAR, an annotation project may be specified in an ordered annotation task such that, first, some or all camera images in a sequence are labeled, then some or all LiDAR frames in a sequence are labeled (e.g., a first top-down 2D bounding box, then a 3D bounding box), objects appearing in multiple annotation scenes are then linked (e.g., for already labeled camera images and LiDAR frames), and then objects appearing in multiple sensor modalities are linked together. In some embodiments, the labeling tool can simultaneously present both types of sensor data (e.g., LiDAR frames and camera images) in an annotation scene during a particular annotation task and / or project information across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data. In some embodiments, an auxiliary feature can iterate through an annotated object from a previous frame and prompt the labeler to find the corresponding object in the current frame.

[0038] Continuing with examples involving cameras and LiDAR, Figures 5–10 show exemplary user interfaces for labeling tools for assisted 3D ground truth labeling of LiDAR and camera frames. Consider an exemplary annotation project that includes a first annotation task in which the labeler initially annotates some or all camera images in a sequence; a second annotation task in which the labeler annotates top-down 2D bounding boxes in a first pass of some or all LiDAR frames in a sequence; a third annotation task in which the labeler annotates 3D bounding boxes in a second pass of labeled LiDAR frames in a sequence; a fourth annotation task in which the labeler links the same object across different annotation scenes in a sequence; and a fifth annotation task in which the labeler links the same object across LiDAR and camera frames within each annotation scene in a sequence. Figure 5 is an illustration of an exemplary user interface for image labeling (e.g., a first annotation task), Figures 6-8 are illustrations of exemplary user interfaces for camera-assisted LiDAR labeling (e.g., a second and third annotation task), Figure 9 is an illustration of an exemplary user interface for LiDAR tracking (e.g., a fourth annotation task), and Figure 10 is an illustration of an exemplary user interface for camera-LiDAR link setup (e.g., a fifth annotation task) according to some embodiments of the present disclosure.

[0039] Now, looking at Figure 5, which shows an exemplary user interface 500 for image labeling according to some embodiments of the present disclosure. The user interface 500 includes a panel that presents a particular camera image 510 to be labeled, and a labeling panel having various interactive elements that activate various drawing and / or annotation functions. In this example, the labeling panel includes tag buttons 560 that, when active, allow the user to identify and label the boundaries and / or corresponding enclosed areas of the image 510. Generally, the user interface 500 can incorporate any known 2D or 3D drawing or annotation software functionality. In a non-limiting example, the user can specify a polygon such as a 2D bounding box by clicking on the image 510 (for example, at a position that identifies the first corner of the bounding box), dragging to extend the 2D bounding box from the first corner, and releasing when the bounding box is of the desired size (for example, at a position that identifies a second corner opposite the first one). In some embodiments, the user interface 500 may allow the user to zoom and / or pan across the image 510. As such, the user can draw or otherwise identify any number of regions (e.g., region 520). In some embodiments (e.g., in any of user interfaces 500-1000), the labeling tool may initialize a set of annotations with annotations from previous frames of sensor data, and the labeler may adjust the annotations to fit the current frame. As such, the labeling tool may reduce or eliminate the need to recreate annotations for the same object over and over again.

[0040] In some embodiments (for example, any of user interfaces 500–1000), the labeling tool may include a labeling panel containing a list of annotated objects within a particular frame of sensor data (e.g., list 570) or other identifications. Each entry in the list may include one or more interaction elements, such as tag buttons 580 and comment buttons 585. The tag button 580 (or any other interaction element) may perform functions such as selecting or identifying a corresponding annotation in the image (e.g., enabling input to edit an annotation), prompting the user to select or edit a class label, or other functions. The comment button 585 may accept text input defining any notes about the annotation. These are intended merely as examples, and any appropriate drawing or annotation functionality may be incorporated into the labeling tool. In some embodiments, the labeling tool may accept input to navigate back and forth in time (i.e., previous and next frames in a sequence) to improve the user's understanding of the data. When a user has finished labeling an image, they can indicate that they are finished (for example, by clicking the submit button 590).

[0041] Figure 6 is an illustration of an exemplary user interface 600 for camera-assisted LiDAR labeling according to several embodiments of the present disclosure. The user interface 600 includes a panel that presents a specific LiDAR frame 620 to be labeled. In some embodiments, the user interface 600 can simultaneously present the specific LiDAR frame 620 to be labeled, as well as a corresponding image 610 from the same annotation scene. For example, the presented image may be a spatially registered 360-degree view image composed of images from surrounding cameras within the same annotation scene, stitched together to form a composite image. Generally, the user interface 600 can present a visualization of the correspondence (e.g., corresponding regions) between the LiDAR frame 620 and the image 610. In some embodiments, the user interface 600 can allow the user to zoom and / or pan within one of the frames, and the user interface 600 can otherwise make corresponding adjustments.

[0042] In some embodiments, labeling tools can project information across sensor modalities to indicate correspondences across sensor data. For example, the known orientation and position of a camera that captured a particular image may be used to project the image onto a 3D representation of the environment (e.g., 3D LiDAR coordinates) and to identify the 3D position (e.g., in LiDAR space) corresponding to a particular image pixel. To project in other directions, a specific 3D position in LiDAR space may be projected into image space using the known orientation and position of the camera that captured the image. In general, various types of information may be projected from a LiDAR frame onto a corresponding image, such as detections (e.g., points, planes), annotations, or other areas. In some embodiments, the user interface 600 can project the position of an input probe, moving with user input, across sensor modalities. For example, when a user places a mouse over LiDAR frame 620, an input probe 640 (also shown in the enlarged area 630) may be used to identify a corresponding 3D location in LiDAR space (for example, by setting a z-value of zero, a z-value of a fitted ground plane, etc.), and the 3D location may be projected into image space to indicate a corresponding point 650 in image 610. In some embodiments, the input probe may be projected additionally or alternatively in the opposite direction (for example, a specified point in image 610 may be projected into LiDAR frame 620). By indicating correspondences across sensor modalities, the user interface 600 can provide useful contextual information to help the labeler generate more accurate annotations.

[0043] In some embodiments, as the first step for each LiDAR spin, the labeler may be prompted to adapt the ground plane to the LiDAR spin. In some embodiments, the adapted ground plane from a previous LiDAR spin may be propagated to the current spin, and the labeler may be prompted to fine-tune the previous plane to the current plane, thereby speeding up the labeling process. In some embodiments, this data-adapted ground plane can provide the z-values ​​for top-down LiDAR annotations. Thus, the received annotations can be aligned to a specified ground plane, improving the accuracy of top-view annotations. Additionally or alternatively, in some embodiments, the user interface 600 may initialize a set of annotations with annotations from a previous LiDAR spin, and the labeler may adjust the annotations to fit the current spin. As such, the labeling tool can reduce or eliminate the need to recreate annotations for the same object over and over again.

[0044] In some embodiments, the user interface 600 can accept input annotations specifying the orientation of a 2D polygon region of the LiDAR frame 620 and / or an object represented by the 2D polygon region. Since the LiDAR frame 620 may also be a projection of 3D data (e.g., a LiDAR point cloud), the input annotations can be adapted to the corresponding 3D representation in LiDAR space. For example, if the LiDAR frame 620 is partly a top-down projection image, the annotated region may be created in the XY plane within the LiDAR coordinate system (e.g., with a z value of zero, a z value of the adapted ground plane), and the 2D annotated region may be extended to 3D (e.g., in subsequent passes through a sequence of annotation scenes, such as in subsequent annotation tasks), as will be described in more detail below.

[0045] Figure 7 is an illustration of an exemplary user interface 700 for camera-assisted LiDAR labeling with orientation vectors, according to several embodiments of the present disclosure. Like user interface 600, user interface 700 can accept input annotations specifying polygonal regions of a LiDAR frame 720. In this example, the labeler has a defined bounding box 730. In some embodiments, the labeling tool can accept inputs defining orientation vectors 740 (e.g., by clicking and dragging in the orientation direction), class labels 750, or other types of annotations. Like user interface 600, user interface 700 can present a visualization of correspondences (e.g., corresponding regions) between the LiDAR frame 720 and an image 710 to assist in the annotation task. In this example, the class and orientation of an object represented in the LiDAR frame 720 are perceptible in the image 710. As a result, by presenting image 710 in relation to (for example, simultaneously with) LiDAR frame 720, Labella can perceive the class and orientation, thereby encoding the corresponding annotation.

[0046] Figure 8 is an illustration of an exemplary user interface 800 for camera-assisted LiDAR labeling with a 3D bounding box, according to some embodiments of the present disclosure. In some embodiments, camera-assisted LiDAR labeling can be broken down into separate tasks for 2D and 3D annotations. The 3D annotations can be generated manually and / or automatically. For example, LiDAR labeling can be broken down into a first pass that accepts input defining a 2D annotation (e.g., a 2D bounding box), and a second pass that accepts input adapting the 2D annotation to a 3D annotation (e.g., a 3D bounding box), or a second stage (e.g., as part of post-processing 160 in Figure 1) that automatically generates the 3D annotation from the 2D annotation. In an exemplary embodiment that includes accepting input adapting a 2D annotation to a 3D annotation, after receiving the specification of the 2D annotation (e.g., a bounding box) during the first pass through a top-down sequence of LiDAR frames, the labeling tool can guide the labeler through an object-by-object procedure. In this example, for each object (e.g., each 2D annotation defined during the first pass), the labeling tool can present a zoomed-in, aligned view of the object in the current LiDAR frame. For example, the raw data of the current LiDAR frame (e.g., LiDAR point cloud) can be projected into one or more views, each of which may be presented in the user interface 800, such as a forward view of the object (e.g., forward view 830), a side view of the object (e.g., side view 820), and a top-down view of the object (e.g., top-down view 840). In some embodiments, a 3D representation of the LiDAR point cloud may be presented additionally or by alternative means (e.g., 3D view 810).

[0047] The user interface 800 can prompt Labella to define the dimensions and / or orientation (e.g., yaw, pitch, roll) of each object in any view. In an example where the first pass is in 2D, the initial 2D annotation may be presented (in views 810, 820, 830, and 840, respectively). Note that a 2D annotation created in a top-down view may first be visualized as a line in the forward view 830 and the side view 820, and / or in all two dimensions in the top-down view 840 and the 3D view 810. In the example shown in Figure 8, Labella can adapt a 2D annotation to a 3D annotation or define a 3D annotation in other ways, for example, by placing vertices, dragging handles, or by otherwise defining or manipulating the representation of the annotation in any view (e.g., bounding boxes 815, 825, 835, and 845 in views 810, 820, 830, and 840, respectively). In some embodiments, modifying an annotation in one view updates the annotation in any other view where the modification is visible. As such, Labella can adapt 3D annotations to LiDAR data (e.g., LiDAR point cloud). After Labella defines a 3D annotation for a particular object, the user can move on to the next object in the LiDAR frame by submitting the annotation (e.g., by clicking a next button or activating some other interaction element).

[0048] Figure 9 is an illustration of an exemplary user interface 900 for LiDAR tracking according to several embodiments of the present disclosure. In this example, the user interface 900 presents consecutive LiDAR frames 910 and 920, as well as images 930 and 940, respectively, from the same annotation scene. The labeling tool can iterate over objects in previous frames (e.g., annotations entered during a previous annotation task), and the user interface 900 can prompt the labeler to find the corresponding annotation (e.g., entered during a previous annotation task) in the current frame. In some embodiments, object detection and tracking may be applied to track the movement of annotated objects from frame to frame over time and / or to initialize the links for confirmation or fine-tuning by a human labeler. In the example shown in Figure 9, the labeler can identify annotations 950 and 960 as the same object. In some cases, an object may be occluded in some frames but not in others. In some embodiments, such as those described above, the labeling tool can accept input (annotations) that link objects across disjoint frames (e.g., jumps).

[0049] In some embodiments, the user interface 900 can highlight estimated regions where objects are predicted to be located. For example, one or more positional values ​​of current or selected objects in a first frame (e.g., LiDAR data) may be adjusted to complement the known self-motion of the sensor that captured the data (e.g., the known self-motion of the data-capturing vehicle). In a non-limiting example, one or more positional values ​​of object annotations (e.g., bounding boxes) and / or representative positional values ​​of one or more objects in a first frame of LiDAR data (e.g., center points of bounding boxes, corners of bounding boxes) may be self-motion-complemented, and the user interface 900 may highlight, outline, pan to, zoom to, and / or otherwise emphasize the corresponding predicted regions in a second frame of LiDAR data. In exemplary implementations, when Labella iterates over labeled objects from a previous annotation scene, the user interface 900 can guide Labella by presenting a visualization in which the labeled objects from the previous annotation scene are predicted to be in the subsequent annotation scene, for example, by zooming in to an estimated region in the sensor data. In such a way, the user interface 900 can guide Labella in identifying corresponding objects from scene to scene. In some embodiments, self-motion interpolation can provide even more useful predictions at lower sampling rates, so even at relatively low sampling rates, the user interface 900 can provide even more useful guidance.

[0050] In Figure 9, the user interface 900 includes a labeling panel 970 which contains various interaction elements that activate the corresponding annotation function. For example, a label pair button 975, when active, may allow the user to identify and link annotations (e.g., enter a new link). One of the reasons for non-matching 980 may be selected to indicate that annotations from one frame are invisible or not labeled in adjacent frames. In some embodiments, the labeling panel 970 may present thumbnail images of the presented sequential frames (e.g., thumbnails 985a, 985b) and may accept input to flag camera and / or LiDAR frame annotation issuance. As such, an annotation task in which annotations are linked across frames of sensor data may at least partially serve the function of a quality assurance check (e.g., quality assurance check 170 in Figure 1). When the labeler has finished linking annotations in pairs of adjacent frames, the labeler can confirm the link 990 and proceed to the next pair of adjacent frames. This example demonstrates an annotation task in which a labeller links objects appearing in multiple LiDAR frames, but some embodiments may include, additionally or alternatively, an annotation task in which a labeller links objects appearing in multiple camera frames.

[0051] Figure 10 is an illustration of an exemplary user interface 1000 for camera-LiDAR linking according to some embodiments of the present disclosure. In this example, the user interface 1000 presents a LiDAR frame 1020, as well as images 1010 from the same annotation scene. A labeling tool can iterate over objects in the annotation scene (e.g., annotations entered during a previous annotation task), and the user interface 1000 can prompt the labeler to find corresponding annotations in other sensor modalities. Generally, annotations, object tracking, and / or object detection from sensor data from a particular sensor can be linked to corresponding annotations, object tracking, and / or object detection of the same object from sensor data from different sensors. For example, the labeler may identify annotations 1040 and 1050 as the same object.

[0052] In some embodiments, the user interface 1000 can highlight estimated regions where objects are predicted to be located. For example, one or more positional values ​​of current or selected objects in a first frame (e.g., a camera frame or LiDAR frame) may be projected onto a second frame (e.g., a LiDAR frame or camera frame) and adjusted to complement the known self-motion of the data-capturing sensor (e.g., the known self-motion of the data-capturing vehicle). In a non-limiting example, one or more positional values ​​of object annotations (e.g., bounding boxes) and / or representative positional values ​​of one or more objects in the first frame of sensor data (e.g., the center point of the bounding box, the corners of the bounding box) may be projected onto a second frame, adjusted for self-motion, and the user interface 1000 may highlight, outline, pan to, zoom in on, and / or otherwise emphasize the corresponding predicted regions in the second frame. As such, the user interface 1000 can leverage known correspondences between sensor modalities to guide the labeller in identifying corresponding objects across sensor modalities.

[0053] Like user interface 900, user interface 1000 includes a labeling panel 1060 which includes various interaction elements that activate the corresponding annotation function. In some embodiments, user interface 900 may accept inputs that flag camera and / or LiDAR frame annotation issuance. As such, annotation tasks in which annotations are linked across sensor modalities may at least partially perform the function of a quality assurance check (e.g., quality assurance check 170 in Figure 1). When Labella has completed linking annotations across sensor modalities in an annotation scene, Labella can verify the links 990 and proceed to the next pair of adjacent frames. While the preceding discussion has focused on LiDAR-to-camera linking, any type of sensor data may be linked to any other type of sensor data (including linking sensor data from the same sensor of two different types, e.g., camera-to-camera linking). Additionally or alternatively, annotation tasks may be integrated, rearranged, split, or otherwise arranged. For example, in some embodiments, LiDAR labeling may be performed in the same annotation task as linking LiDAR to a camera (e.g., labeling an object in the camera image, and then simultaneously linking and labeling the corresponding object in the LiDAR frame). These are intended merely as examples, and other variations may be implemented within the scope of this disclosure.

[0054] Returning to Figure 1, in some embodiments, post-processing 160 may be performed to automatically detect features that are not normally perceptible to humans. In non-limiting examples, image processing may be applied to camera images to determine depth values ​​(e.g., using a machine learning model to predict pixel-by-pixel depth values), and depth values ​​or subsets thereof (e.g., representative depths for each annotation, such as nearest or average depth) may be associated with the image. In some cases, post-processing 160 may be re-executed (e.g., based on improved calibration) to recalculate any automatically detected features and / or improve corresponding visualizations (e.g., by projecting 3D LiDAR labels back onto the corresponding camera images). In some embodiments, the annotation project may include one or more manual quality assurance checks 170 to verify the accuracy of annotations manually generated and / or automatically generated (e.g., during post-processing 160) by a human labeler. Figure 11 is an illustration of exemplary ground truth annotations according to some embodiments of the present disclosure.

[0055] In some embodiments, post-processing 160 may be performed to transform the ground truth annotations into an encoded representation that matches the view, size, and dimensionality of the output of the machine learning model to be trained. For example, if the machine learning model outputs classification data (e.g., one or more channels, each outputting a different class confidence map), the ground truth annotations in a given frame of sensor data may be transformed into corresponding class confidence maps for each class. As a non-restrictive example, for a given class, the values ​​of pixels that fall within the annotated region of that class may be set to a value indicating a positive classification (e.g., 1), and the values ​​of other pixels in the image may be set to a value indicating a negative classification (e.g., 0). In such a way, different class confidence maps may be stacked to form a ground truth tensor that matches the output of the machine learning model.

[0056] In another example, if a machine learning model regresses instance data (for example, outputting one or more channels, where each channel regresses different types of object instance data such as location, geometry, and / or orientation data), the location, geometry, orientation, and / or class of each annotation may be used to generate object instance data that matches the view, size, and dimensionality of the output of the machine learning model to be trained. For example, for each pixel containing an annotation, the annotation may be used to compute the corresponding location, geometry, and / or orientation information (for example, where the object is located relative to each pixel, object height, object width, object orientation (e.g., rotation angle relative to the orientation of the projected image), and / or similar things—e.g., object center). The computed object instance data may be stored in the corresponding channels of a ground truth tensor. These are just a few examples, and other types of post-processing may be performed additionally or alternatively.

[0057] After some or all annotation tasks in an annotation project are completed, the resulting ground truth data can be exported in any suitable format (e.g., ground truth data export 180 in Figure 1). The ground truth data can be paired with corresponding input training data that matches the type of input accepted by the machine learning model to be trained. Thus, one or more machine learning models can be trained using the input training data and the exported ground truth data (e.g., ground truth data consumption 190 in Figure 1). For example, one or more loss functions (e.g., a single loss function, classification loss and / or regression loss for each output type, etc.) can be used to compare the accuracy of the machine learning model's output to ground truth, and the parameters of the machine learning model can be updated until the accuracy reaches an optimal or acceptable level (e.g., using backward pass, backpropagation, forward pass, etc.).

[0058] Referring here to Figures 12 and 13, each block of Methods 1200 and 1300 described herein includes a computation process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. The methods may also be performed as computer-available instructions stored in a computer storage medium. The methods may be provided, to name a few, by a standalone application, a service, or a hosted service (either standalone or in combination with another hosted service), or by plugging into another product. In addition, Methods 1200 and 1300 may be understood, for example, with respect to the annotation pipeline 100 in Figure 1. However, these methods may be performed additionally or alternatively by any one system or any combination of systems, including but not limited to those described herein.

[0059] Figure 12 is a flowchart illustrating Method 1200 for generating ground truth annotations for sensor data from different types of sensors, according to some embodiments of the present disclosure. Method 1200 includes, in block B1202, accessing sensor data captured by different types of sensors during a capture session. For example, one or more vehicles (e.g., Vehicle 1400 in Figures 14A–D) can collect sensor data from one or more sensors of the vehicle in a real-world (e.g., physical) environment (e.g., as part of data capture 110 in Figure 1). The sensor data can be stored and accessed in any manner.

[0060] Method 1200 includes, in block B1204, identifying at least one offset to synchronize sensor data and constructing a sequence of annotation scenes, each annotation scene containing frames of sensor data from two or more sensors of different types. For example, since sensor data may be acquired from different sensors at different frequencies, the sensor data may be aligned to facilitate grouping of sensor data for similar world states (e.g., sensor data alignment 120 in Figure 1). In some embodiments, a particular sensor may be used as a reference sensor. Non-reference sensors may be referred to as child sensors. For a given frame of sensor data from a reference sensor (reference frame), an offset such as a time delta may be identified between the reference frame and the temporally closest frame of sensor data from each child sensor. The offset for each child sensor may be stored and / or applied to the capture time of the sensor data from the child sensor or some other index. In general, the reference and child sensor data may be sampled using the identified offset to identify and construct frames of sensor data from different sensors for each annotation scene.

[0061] Method 1200, in block B1206, includes encoding multiple linear annotation tasks into a labeling tool for annotating an annotation scene with ground truth annotations. Generally, the labeling tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project manager identifying and / or providing the annotation scenes to be labeled (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotations may be broken down into a set of linear tasks, and the encoded representations of the tasks may be input into the labeling tool (e.g., as part of annotation 150 in annotation pipeline 100 in Figure 1).

[0062] Method 1200 includes assigning annotation tasks to labelers using a labeling tool in block B1208. For example, as part of annotation 150 in annotation pipeline 100 in Figure 1, the labeling tool can assign each annotation task to a specific labeler by assigning tasks based on any suitable method, for example, labeler availability, a defined task order, or otherwise.

[0063] Method 1200, in block B1210, involves using a labeling tool to guide the labeler through a sequence of annotation scenes. For example, as part of annotation 150 in annotation pipeline 100 in Figure 1, the labeling tool can arrange the order of inputs (e.g., annotations) for a particular task using a wizard that guides the labeler through the task.

[0064] Method 1200 includes, in block B1212, using a labeling tool to prompt and accept inputs that define a set of ground truth annotations defined by an annotation task, while presenting sensor data from two or more sensors of different types in the annotation scene. For example, as part of annotation 150 in annotation pipeline 100 in Figure 1, the labeling tool can arrange the order of inputs for a particular task (for example, by walking through the object-by-object annotation procedure for each annotation scene in the sequence). Figure 6 shows an exemplary embodiment in which the labeling tool can present sensor data from different types of sensors simultaneously and can project information across sensor modalities to show correspondence across sensor data.

[0065] Method 1200 includes exporting a representation of ground truth annotations in block B1214. For example, after some or all annotation tasks in an annotation project are completed, the resulting ground truth data can be exported in any suitable format (e.g., ground truth data export 180 in Figure 1).

[0066] Figure 13 is a flowchart illustrating Method 1300 for generating ground truth annotations for LiDAR and camera frames according to some embodiments of the present disclosure. Method 1300 includes, in block B1302, accessing sensor data captured during a capture session, the sensor data including LiDAR frames from a LiDAR sensor and camera frames from at least one camera. For example, one or more vehicles (e.g., vehicle 1400 in Figures 14A–D) can collect sensor data from one or more sensors of the vehicle in a real-world (e.g., physical) environment (e.g., as part of data capture 110 in Figure 1). The vehicle sensors may include one or more LiDAR sensors and one or more cameras. The captured sensor data can be accessed at any time.

[0067] Method 1300, in block B1304, includes identifying an offset for each camera of at least one camera to synchronize camera frames with LiDAR frames to constitute a sequence of annotation scenes, each annotation scene comprising one LiDAR frame and at least one camera frame. For example, as a LiDAR spin travels and views different parts of the environment, the temporally closest camera frame for any given LiDAR spin may be selected based on the camera's field of view angle relative to the LiDAR spin start angle and how long it takes for the LiDAR spin to align with the camera's field of view (e.g., a portion such as its center). In general, a camera-specific time or index offset relative to the LiDAR spin start may be determined and / or applied to align each camera's camera frame with the LiDAR frame. Thus, each LiDAR frame and the temporally closest camera frame (e.g., for each camera) may be sampled and packaged into corresponding annotation scenes to constitute a sequence.

[0068] Method 1300 includes encoding multiple linear annotation tasks into a labeling tool in block B1306 to annotate an annotation scene with ground truth annotations. Generally, the labeling tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project manager identifying and / or providing the annotation scenes to be labeled (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotations may be broken down into a set of linear tasks, and the encoded representations of the tasks may be input into the labeling tool (e.g., as part of annotation 150 in annotation pipeline 100 in Figure 1).

[0069] Method 1300, in block B1308, includes using a labeling tool to guide the labeler through a sequence of annotation scenes. For example, as part of annotation 150 in annotation pipeline 100 in Figure 1, the labeling tool can use a wizard to guide the labeler through tasks to arrange the order of inputs (e.g., annotations) for a particular task.

[0070] Method 1300 includes, for each annotation scene, using a labeling tool to prompt and accept input defining a set of ground truth annotations defined by an annotation task, while presenting at least one of a LiDAR frame and a camera frame in the annotation scene in block B1310. For example, as part of annotation 150 in annotation pipeline 100 in Figure 1, the labeling tool can arrange the order of inputs for a particular task (for example, by walking through the object-by-object annotation procedure for each annotation scene in the sequence). Figure 6 shows an exemplary embodiment in which the labeling tool can present LiDAR and camera frames simultaneously and project information from LiDAR to camera and / or vice versa.

[0071] Method 1300 includes exporting a representation of ground truth annotations in block B1312. For example, after some or all annotation tasks in an annotation project are completed, the resulting ground truth data can be exported in any suitable format (e.g., ground truth data export 180 in Figure 1). As such, one or more machine learning models can be trained using the exported ground truth data (e.g., ground truth data consumption 190 in Figure 1).

[0072] Exemplary autonomous vehicle Figure 14A shows an exemplary autonomous vehicle 1400 according to some embodiments of the present disclosure. The autonomous vehicle 1400 (or referred to herein as "vehicle 1400") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttles, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Mobile vehicle 1400 may have the capability to perform functions at one or more of the autonomous driving levels from Level 3 to Level 5. For example, depending on the embodiment, mobile vehicle 1400 may have the capability of conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0073] The vehicle 1400 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. The vehicle 1400 may include a propulsion system 1450, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 1450 may be connected to the vehicle 1400's drivetrain, which may include a transmission, to enable propulsion for the vehicle 1400. The propulsion system 1450 may be controlled in response to receiving a signal from a throttle / accelerator 1452.

[0074] A steering system 1454, which may include a steering wheel, may be used to steer the vehicle 1400 (for example, along a desired course or route) when the propulsion system 1450 is operating (for example, when the vehicle is moving). The steering system 1454 may receive signals from the steering actuator 1456. The steering wheel may also be an option for fully automated (level 5) functionality.

[0075] The brake sensor system 1446 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1448 and / or the brake sensor.

[0076] The controller 1436, which may include one or more system-on-a-chip (SoC) 1404 (Figure 14C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 1400. For example, the controller can send signals to operate the vehicle brakes via one or more brake actuators 1448, to operate the steering system 1454 via one or more steering actuators 1456, and to operate the propulsion system 1450 via one or more throttle / accelerators 1452. The controller 1436 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist the driver in driving the vehicle 1400. The controller 1436 may include a first controller 1436 for autonomous driving functions, a second controller 1436 for functional safety functions, a third controller 1436 for artificial intelligence functions (e.g., computer vision), a fourth controller 1436 for infotainment functions, a fifth controller 1436 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1436 may handle two or more of the aforementioned functions, and two or more controllers 1436 may handle a single function, and / or any combination thereof.

[0077] The controller 1436 can provide signals for controlling one or more components and / or systems of the mobile vehicle 1400 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example and without limitation, global navigation satellite system sensors 1458 (e.g., global positioning system sensors), RADAR sensors 1460, ultrasonic sensors 1462, LIDAR sensors 1464, inertial measurement unit (IMU) sensors 1466 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1496, stereo cameras 1468, wide-view cameras 1470 (e.g., fisheye cameras), infrared cameras 1472, surround cameras 1474 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1498, speed sensors 1444 (e.g., for measuring the speed of a moving vehicle 1400), vibration sensors 1442, steering sensors 1440, brake sensors (e.g., as part of a brake sensor system 1446), and / or other sensor types.

[0078] One or more of the controllers 1436 may receive inputs (represented, for example, by input data) from the instrument cluster 1432 of the mobile vehicle 1400 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 1434, an audible annunciator, a loudspeaker, and / or other components of the mobile vehicle 1400. The outputs may include information such as mobile vehicle velocity, speed, time, map data (e.g., HD map 1422 in Figure 14C), location data (e.g., the location of the mobile vehicle 1400, such as on the map), direction, the locations of other mobile vehicles (e.g., occupied grids), and information about objects and the status of objects as grasped by the controller 1436. For example, the HMI display 1434 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or driving operations that the moving vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0079] The mobile vehicle 1400 further includes a network interface 1424 that can communicate over one or more networks using one or more wireless antennas 1426 and / or a modem. For example, the network interface 1424 may have the capability to communicate over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antenna 1426 can also enable communication between objects in the environment (e.g., mobile vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWAN) such as LoRaWAN, SigFox.

[0080] Figure 14B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 1400 of Figure 14A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different positions on the mobile vehicle 1400.

[0081] The camera type may include, but is not limited to, a digital camera that can be used with components and / or systems of the mobile vehicle 1400. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have the capability of any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the capability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0082] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0083] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0084] A camera having a field of view that includes a portion of the environment in front of the moving vehicle 1400 (e.g., a forward-facing camera) may be used for surround view to assist in identifying the forward path and obstacles and, with the help of one or more controllers 1436 and / or control SoCs, to assist in providing information essential for generating an occupied grid and / or determining a preferred moving vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as Lane Departure Warning ("LDW"), Autonomous Cruise Control ("ACC"), and / or traffic sign recognition.

[0085] Various cameras may be used in a forward-facing configuration, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 1470, which can be used to capture objects that come into view from the periphery (e.g., pedestrians, intersecting traffic, or bicycles). Although only one wide-view camera is shown in Figure 14B, any number of wide-view cameras 1470 may be present in the mobile vehicle 1400. In addition, long-range cameras 1498 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 1498 may also be used for object detection and classification, as well as basic object tracking.

[0086] One or more stereo cameras 1468 may also be included in the forward-facing configuration. The stereo camera 1468 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the moving vehicle's environment, including distance estimates of all points in the image. An alternative stereo camera 1468 may include a compact stereo vision sensor that includes two camera lenses (one on the left and one on the right) and an image processing chip that can measure the distance from the moving vehicle to an object and activate autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 1468 may be used in addition to or instead of those described herein.

[0087] A camera having a field of view including a portion of the environment on the sides of the mobile vehicle 1400 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 1474 (e.g., four surround cameras 1474 as shown in Figure 14B) may be positioned on the mobile vehicle 1400. The surround cameras 1474 may include wide-view cameras 1470, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the mobile vehicle. In an alternative configuration, the mobile vehicle may use three surround cameras 1474 (e.g., left, right, and rear), and one or more other cameras (e.g., forward-facing cameras) may be utilized as a fourth surround view camera.

[0088] A camera having a field of view that includes a portion of the environment behind the mobile vehicle 1400 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 1498, stereo camera 1468), infrared cameras 1472, etc., as described herein.

[0089] Figure 14C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 1400 of Figure 14A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted together. Furthermore, many of the elements described herein are functional entities that can be implemented as individual or distributed components or in combination with other components, and in any appropriate combination and location. The various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0090] Each component, feature, and system of the mobile vehicle 1400 in Figure 14C is illustrated as being connected via a bus 1402. Bus 1402 may include a Controller Area Network (CAN) data interface (or referred to as the "CAN bus"). CAN may also be a network within the mobile vehicle 1400 used to help control various features and functions of the mobile vehicle 1400, such as the operation of brakes, acceleration, steering, windshield wipers, etc. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other mobile vehicle status indicators. The CAN bus may be ASIL B compliant.

[0091] Bus 1402 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or as an alternative to, a CAN bus. In addition, a single line is used to represent bus 1402, but this is not intended to limit it. There may be any number of buses 1402, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1402 may be used to perform different functions and / or for redundancy. For example, a first bus 1402 may be used for collision avoidance and a second bus 1402 may be used for operation control. In any example, each bus 1402 may communicate with any of the components of the mobile vehicle 1400, and two or more buses 1402 may communicate with the same component. In some examples, each SoC 1404, each controller 1436, and / or each computer within the mobile vehicle may have access to the same input data (e.g., input from sensors in the mobile vehicle 1400) and may be connected to a common bus such as a CAN bus.

[0092] The mobile vehicle 1400 may include one or more controllers 1436, such as those described herein with respect to Figure 14A. The controllers 1436 may be used for a variety of functions. The controllers 1436 may be connected to any of the various other components and systems of the mobile vehicle 1400 and may be used for the control of the mobile vehicle 1400, the artificial intelligence of the mobile vehicle 1400, infotainment for the mobile vehicle 1400, and / or the like.

[0093] The mobile vehicle 1400 may include a system-on-a-chip (SoC) 1404. The SoC 1404 may include a CPU 1406, a GPU 1408, a processor 1410, a cache 1412, an accelerator 1414, a data store 1416, and / or other components and features not shown. The SoC 1404 can be used to control the mobile vehicle 1400 in various platforms and systems. For example, the SoC 1404 may be coupled in a system (e.g., a system of the mobile vehicle 1400) that has an HD map 1422 that can obtain map refreshes and / or updates from one or more servers (e.g., server 1478 in Figure 14D) via a network interface 1424.

[0094] The CPU 1406 may include a CPU cluster or CPU complex (also referred to as "CCPLEX"). The CPU 1406 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 1406 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 1406 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 1406 (e.g., CCPLEX) may be configured to support concurrent cluster operation, allowing any combination of the CPU 1406 clusters to be active at any given time.

[0095] The CPU1406 can implement power management capabilities including one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU1406 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0096] The GPU1408 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU1408 may be programmable and efficient for parallel workloads. In some embodiments, the GPU1408 may be able to use an enhanced tensor instruction set. The GPU1408 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share a cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU1408 may include at least eight streaming microprocessors. The GPU1408 may be able to use a Computation Application Programming Interface (API). In addition, the GPU1408 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0097] The GPU1408 can be power-optimized for optimal performance in automotive and embedded use cases. For example, the GPU1408 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be a limitation, and the GPU1408 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA tensor cores for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0098] In some examples, the GPU1408 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0099] The GPU1408 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges across processors. In some examples, address translation service (ATS) support can be used to allow the GPU1408 to directly access the CPU1406 page table. In such examples, when the GPU1408 memory management unit (MMU) experiences a miss, an address translation request can be sent to the CPU1406. In response, the CPU1406 can examine its page table for virtual-to-real-address mapping and send the translation back to the GPU1408. As such, unified memory technology can enable a single, unified virtual address space for both the CPU1406 and GPU1408 memory, thereby simplifying GPU1408 programming and porting of applications to the GPU1408.

[0100] In addition, the GPU1408 may include an access counter that can record how often the GPU1408 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0101] The SoC1404 may include any number of caches 1412, including those described herein. For example, cache 1412 may include an L3 cache available to both the CPU 1406 and the GPU 1408 (e.g., connected to both the CPU 1406 and the GPU 1408). Cache 1412 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4MB or more, depending on the embodiment, although a smaller cache size may be used.

[0102] The SoC1404 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 1400 (for example, a processing DNN). In addition, the SoC1404 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC104 may include one or more FPUs integrated as execution units within the CPU1406 and / or GPU1408.

[0103] The SoC1404 may include one or more accelerators 1414 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC1404 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4 MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU1408 and to offload some of the tasks of the GPU1408 (e.g., to free up more cycles of the GPU1408 to perform other tasks). As an example, accelerator 1414 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0104] Accelerator 1414 (e.g., hardware acceleration cluster) may include a deep learning accelerator (DLA). A DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. A DLA design can provide more performance per millisecond than a general-purpose GPU and significantly exceed CPU performance. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0105] DLA can quickly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and mobile vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0106] DLA can perform any function of GPU1408, and by using inference accelerators, for example, a designer can target either DLA or GPU1408 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU1408 and / or other accelerators 1414.

[0107] The accelerator 1414 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may include, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0108] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may contain any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0109] DMA can enable PVA components to access system memory independent of the CPU1406. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0110] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0111] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA can run the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA can run different computer vision algorithms simultaneously on the same image, or even run different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0112] The accelerator 1414 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 1414. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, and not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0113] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0114] In some embodiments, the SoC1404 may include a real-time ray tracing hardware accelerator, as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0115] Accelerators 1414 (e.g., hardware accelerator clusters) have diverse applications for autonomous driving. PVAs may also be programmable vision accelerators that can be used in critical processing stages in ADAS and autonomous vehicles. The capabilities of PVAs are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, PVAs perform well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because PVAs are efficient in object detection and integer computation, in relation to platforms for autonomous vehicles, PVAs are designed to run classic computer vision algorithms.

[0116] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. While semi-global matching-based algorithms may be used in some examples, this is not intended to be a limitation. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0117] In some applications, PVA can be used to perform high-density optical flow by processing raw RADAR data (e.g., using 4D Fast Fourier Transform) to provide processed RADAR data. In other applications, PVA is used for flight depth processing, for example, by processing raw flight data to provide processed flight data.

[0118] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the moving vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), object orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation acquired from the neural network and / or other sensors (e.g., LIDAR sensor 1464 or RADAR sensor 1460), and others.

[0119] The SoC1404 may include a data store 1416 (for example, memory). The data store 1416 may also be the on-chip memory of the SoC1404 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 1416 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 1412 may comprise an L2 or L3 cache 1412. References to the data store 1416 may include references to memory associated with the PVA, DLA, and / or other accelerators 1414, as described herein.

[0120] The SoC1404 may include one or more processors 1410 (e.g., embedded processors). The processors 1410 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC1404 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC1404 thermal and temperature sensors, and / or management of SoC1404 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC1404 may use the ring oscillators to detect the temperatures of the CPU 1406, GPU 1408, and / or accelerator 1414. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC1404 into a lower power state and / or putting the mobile vehicle 1400 into chauffeur safe shutdown mode (for example, safely shutting down the mobile vehicle 1400).

[0121] The processor 1410 may further include a set of integrated processors capable of performing the functions of an audio processing engine. The audio processing engine may also be an audio subsystem enabling full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0122] The processor 1410 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0123] The processor 1410 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic for detecting any differences between their operations.

[0124] The processor 1410 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0125] The processor 1410 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0126] The processor 1410 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 1470, the surround camera 1474, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0127] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0128] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU1408 is not required to continuously render new surfaces. Even when the GPU1408 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU1408 to improve performance and responsiveness.

[0129] The SoC1404 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC1404 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0130] The SoC1404 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC1404 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., LiDAR sensor 1464, RADAR sensor 1460, etc., which may be connected via Ethernet®), data from bus 1402 (e.g., speed of vehicle 1400, steering wheel position, etc.), and data from GNSS sensor 1458 (connected, for example, via Ethernet® or CAN bus). The SoC1404 may further include a dedicated high-performance mass storage controller which may include its own DMA engine and which may be used to free up CPU 1406 from routine data management tasks.

[0131] The SoC1404 may also be an inter-terminal platform with a flexible architecture that extends to automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and provides a platform for a flexible, reliable driving software stack along with deep learning tools. The SoC1404 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 1414 is coupled with the CPU 1406, the GPU 1408, and the data store 1416 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0132] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0133] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU1420) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of the signs and passing that semantic understanding to a route planning module running on the CPU complex.

[0134] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU1408, for example.

[0135] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of the vehicle 1400. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and, in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, the SoC 1404 provides security against theft and / or vehicle hijacking.

[0136] In another example, a CNN for emergency vehicle detection and identification can detect and identify emergency vehicle sirens using data from microphone 1496. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 1404 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In a preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the moving vehicle is operating, as identified by GNSS sensor 1458. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, the control program may be used, with the assistance of the ultrasonic sensor 1462, to perform an emergency vehicle safety routine, such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle, until the emergency vehicle has passed.

[0137] The vehicle may include a CPU 1418 (e.g., a separate CPU, or dCPU) which can be connected to the SoC 1404 via a high-speed interconnect (e.g., PCIe). The CPU 1418 may include, for example, an x86 processor. The CPU 1418 may be used to perform any of a variety of functions, including, for example, mediating the consequences of a potential mismatch between ADAS sensors and the SoC 1404, and / or monitoring the status and condition of the controller 1436 and / or the infotainment SoC 1430.

[0138] The mobile vehicle 1400 may include a GPU 1420 (e.g., a separate GPU, or dGPU) which can be connected to the SoC 1404 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 1420 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from the mobile vehicle 1400's sensors (e.g., sensor data).

[0139] Mobile vehicle 1400 may further include a network interface 1424 which may include one or more wireless antennas 1426 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth antennas). The network interface 1424 may be used to enable wireless connectivity to a cloud over the Internet (e.g., with server 1478 and / or other network devices), to other mobile vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other mobile vehicles, a direct link may be established between two mobile vehicles, and / or an indirect link may be established (e.g., over a network and over the Internet). The direct link may be provided using a mobile vehicle-to-mobile communication link. The mobile vehicle-to-mobile communication link may provide mobile vehicle 1400 information about mobile vehicles in close proximity to mobile vehicle 1400 (e.g., mobile vehicles in front of, beside, and / or behind mobile vehicle 1400). This function may also be part of the joint adaptive cruise control function of the mobile vehicle 1400.

[0140] The network interface 1424 may include an SoC that provides modulation and demodulation functions and enables the controller 1436 to communicate over a wireless network. The network interface 1424 may include a radio frequency front end for up-conversion from baseband to radio frequency and down-conversion from radio frequency to baseband. Frequency conversion can be performed through well-known processes and / or using a superheterodyne process. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0141] The mobile unit 1400 may further include a data store 1428 which may include storage outside the chip (for example, outside the SoC 1404). The data store 1428 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0142] The vehicle 1400 may further include GNSS sensors 1458. The GNSS sensors 1458 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) assist in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 1458 may be used, including, but not limited to, a GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0143] The mobile vehicle 1400 may further include a RADAR sensor 1460. The RADAR sensor 1460 may be used by the mobile vehicle 1400 for long-range mobile vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 1460 may use CAN and / or bus 1402 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 1460) with access to Ethernet® for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 1460 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0144] The RADAR sensor 1460 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 1460 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include a monostatic multimodal RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record around the moving vehicle 1400 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of moving vehicles entering or leaving the lane of the moving vehicle 1400.

[0145] As an example, a medium-range RADAR system may include a range of up to 1460 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 1450 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the moving vehicle.

[0146] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0147] The mobile vehicle 1400 may further include ultrasonic sensors 1462. Positioned on the front, rear, and / or sides of the mobile vehicle 1400, the ultrasonic sensors 1462 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 1462 may be used, and different ultrasonic sensors 1462 may be used for detection of different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 1462 may operate at a functional safety level of ASIL B.

[0148] The mobile vehicle 1400 may include a LiDAR sensor 1464. The LiDAR sensor 1464 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 1464 may also have a functional safety level of ASIL B. In some examples, the mobile vehicle 1400 may include multiple LiDAR sensors 1464 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0149] In some examples, the LIDAR sensor 1464 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 1464 may have an advertised range of approximately 1400m, for example, with an accuracy of 2cm to 3cm and support for 1400Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 1464 may be used. In such examples, the LIDAR sensor 1464 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a mobile vehicle 1400. In such examples, the LIDAR sensor 1464 may have a range of 200m even for low-reflection objects and can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees. A front-mounted LIDAR sensor 1464 may be configured for a horizontal field of view between 45 and 135 degrees.

[0150] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around a moving vehicle up to approximately 200m. The flash LiDAR unit includes receptors that record the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the moving vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the moving vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1464 may be less susceptible to motion blur, vibration, and / or shock.

[0151] The mobile vehicle may further include an IMU sensor 1466. In some examples, the IMU sensor 1466 may be positioned in the center of the rear axle of the mobile vehicle 1400. The IMU sensor 1466 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 1466 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 1466 may include an accelerometer, a gyroscope, and a magnetometer.

[0152] In some embodiments, the IMU sensor 1466 may be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor 1466 may enable the moving vehicle 1400 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating velocity changes from the GPS to the IMU sensor 1466. In some embodiments, the IMU sensor 1466 and the GNSS sensor 1458 may be combined in a single integrated unit.

[0153] The mobile vehicle may include a microphone 1496 placed inside and / or around the mobile vehicle 1400. The microphone 1496 may, among other things, be used for emergency vehicle detection and identification.

[0154] The mobile vehicle may further include any number of camera types, including a stereo camera 1468, a wide-view camera 1470, an infrared camera 1472, a surround camera 1474, a long-range and / or medium-range camera 1498, and / or other camera types. The cameras may be used to capture image data around the entire exterior surface of the mobile vehicle 1400. The type of camera used will depend on the embodiment and requirements of the mobile vehicle 1400, and any combination of camera types may be used to achieve the required coverage around the mobile vehicle 1400. In addition, the number of cameras may vary depending on the embodiment. For example, the mobile vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 14A and 14B.

[0155] The mobile vehicle 1400 may further include a vibration sensor 1442. The vibration sensor 1442 can measure vibrations of components of the mobile vehicle, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 1442 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0156] The mobile vehicle 1400 may include an ADAS system 1438. In some examples, the ADAS system 1438 may include a System of Control (SoC). The ADAS system 1438 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0157] The ACC system may utilize a radar sensor 1460, a lithium-ion sensor 1464, and / or a camera. The ACC system may include longitudinal ACC and / or transverse ACC. Longitudinal ACC monitors and controls the distance of vehicle 1400 to the vehicle immediately in front of it and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Transverse ACC performs distance maintenance and advises vehicle 1400 to change lanes when necessary. Transverse ACC is related to other ADAS applications such as LCA and CWS.

[0158] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through the network interface 1424 and / or wireless antenna 1426, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., the vehicle immediately in front of vehicle 1400, in the same lane as vehicle 1400), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about the vehicle ahead of vehicle 1400, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0159] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 1460, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual warnings, vibration, and / or quick brake pulses.

[0160] An AEB system can detect an imminent forward collision with another moving vehicle or other object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or impending collision braking.

[0161] The LDW system warns the driver when the vehicle 1400 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0162] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 1400 when it begins to drift out of its lane.

[0163] The BSW system detects and warns the driver of a moving vehicle in the vehicle's blind spot. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0164] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 1400 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0165] Conventional ADAS systems warn the driver and allow the driver to determine whether a safe condition truly exists and act accordingly. However, conventional ADAS systems have sometimes tended to produce misjudgments that, while not usually catastrophic, can be troubling and distracting to the driver. In the autonomous vehicle 1400, however, if the results are contradictory, the vehicle 1400 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 1436 or the second controller 1436). For example, in some embodiments, the ADAS system 1438 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run a variety of redundant software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 1438 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0166] In some implementations, a primary computer may be configured to provide a supervising MCU with a reliability score indicating the reliability of the primary computer in a selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0167] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer is providing a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW is identifying a metal object that is not actually dangerous, such as a sewer grate or manhole cover that triggers an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments involving a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC1404.

[0168] In other examples, ADAS system 1438 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0169] In some examples, the output of the ADAS system 1438 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 1438 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0170] The mobile vehicle 1400 may further include an infotainment SoC 1430 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more separate components. The infotainment SoC 1430 may include a combination of hardware and software that can be used to provide the mobile vehicle 1400 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other mobile vehicle-related information). For example, the infotainment SoC 1430 may also include wireless, disc player, navigation system, video player, USB and Bluetooth connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 1434, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 1430 may be further used to provide information (for example, visual and / or audible) to the user of the vehicle, such as information from the ADAS system 1438, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0171] The infotainment SoC 1430 may include GPU functionality. The infotainment SoC 1430 can communicate with other devices, systems, and / or components of the vehicle 1400 via bus 1402 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 1430 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 1436 (e.g., the primary and / or backup computer of the vehicle 1400) failure. In such examples, the infotainment SoC 1430 can put the vehicle 1400 into a chauffeur-safe stop mode as described herein.

[0172] The mobile vehicle 1400 may further include an instrument cluster 1432 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 1432 may include a controller and / or a supercomputer (e.g., a separate controller or supercomputer). The instrument cluster 1432 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some examples, information may be displayed and / or shared between the infotainment SoC 1430 and the instrument cluster 1432. In other words, the instrument cluster 1432 may be included as part of the infotainment SoC 1430, and vice versa.

[0173] Figure 14D is a system diagram of communication between the cloud-based server of Figure 14A and an exemplary autonomous vehicle 1400 according to some embodiments of the present disclosure. System 1476 may include a server 1478, a network 1490, and a mobile vehicle including the mobile vehicle 1400. Server 1478 may include a plurality of GPUs 1484(A) to 1484(H) (collectively referred to herein as GPU 1484), PCIe switches 1482(A) to 1482(H) (collectively referred to herein as PCIe switch 1482), and / or CPUs 1480(A) to 1480(B) (collectively referred to herein as CPU 1480). The GPUs 1484, CPUs 1480, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 1488 and / or PCIe connections 1486 developed by NVIDIA. In some examples, the GPU1484 is connected via NVLink and / or NVSwitch SoCs, and the GPU1484 and PCIe switch 1482 are connected via PCIe interconnects. Eight GPU1484s, two CPU1480s, and two PCIe switches are illustrated, but this is not intended to be an limitation. Depending on the embodiment, each server 1478 may contain any number of GPU1484s, CPU1480s, and / or PCIe switches. For example, server 1478 may contain eight, sixteen, thirty-two, and / or more GPU1484s, respectively.

[0174] Server 1478 can receive image data from mobile vehicles via network 1490, representing images showing unexpected or altered road conditions, such as recently commenced road construction. Server 1478 can transmit map information 1494, including information about traffic and road conditions, to mobile vehicles via network 1490, including information about neural networks 1492, updated neural networks 1492, and / or map information 1494. Updates to map information 1494 may include updates to HD map 1422, including information about construction sites, potholes, detours, floods, and / or other obstacles. In some examples, neural networks 1492, updated neural networks 1492, and / or map information 1494 may have arisen from new training and / or experience represented in data received from any number of mobile vehicles in the environment, and / or based on training performed in a data center (e.g., using server 1478 and / or other servers).

[0175] Server 1478 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a mobile device and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been traced, it may be used by the mobile vehicle (for example, transmitted to the mobile vehicle via network 1490), and / or the machine learning model may be used by server 1478 to remotely monitor the mobile vehicle.

[0176] In some examples, Server 1478 can receive data from a mobile vehicle and apply it to a state-of-the-art real-time neural network for real-time intelligent inference. Server 1478 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 1484, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 1478 may include deep learning infrastructure that uses only CPU-powered data centers.

[0177] The deep learning infrastructure of server 1478 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware in mobile vehicle 1400. For example, the deep learning infrastructure can receive periodic updates from mobile vehicle 1400, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by mobile vehicle 1400, and if the results do not match and the infrastructure concludes that the AI ​​in mobile vehicle 1400 is not functioning properly, server 1478 can send a signal to mobile vehicle 1400 instructing the failsafe computer in mobile vehicle 1400 to infer control, notify passengers, and complete a safe parking operation.

[0178] For inference, server 1478 may include GPU 1484 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, servers powered by a CPU, FPGA, and other processors may be used for inference.

[0179] Exemplary computing devices Figure 15 is a block diagram of an example of a computing device 1500 suitable for use in implementing some embodiments of the present disclosure. The computing device 1500 may include an interconnection system 1502 that indirectly or directly connects the following devices: memory 1504, one or more central processing units (CPUs) 1506, one or more graphics processing units (GPUs) 1508, a communication interface 1510, input / output (I / O) ports 1512, input / output components 1514, a power supply unit 1516, one or more presentation components 1518 (e.g., a display), and one or more logical units 1520. In at least one embodiment, the computing device 1500 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 1508 may include one or more vGPUs, one or more of the CPUs 1506 may include one or more vCPUs, and / or one or more of the logical units 1520 may include one or more virtual logical units. As such, the computing device 1500 may include individual components (e.g., an entire GPU dedicated to the computing device 1500), virtual components (e.g., a portion of a GPU dedicated to the computing device 1500), or a combination thereof.

[0180] The various blocks in Figure 15 are shown connected by lines via the interconnection system 1502, but this is not intended to be restrictive and is simply for clarity. For example, in some embodiments, a presentation component 1518, such as a display device, could be considered an I / O component 1514 (for example, if the display is a touchscreen). As another example, the CPU 1506 and / or GPU 1508 could include memory (for example, memory 1504 could represent a storage device in addition to the memory of the GPU 1508, CPU 1506, and / or other components). In other words, the computing devices in Figure 15 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 15 and are therefore not distinguished.

[0181] The interconnection system 1502 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 1502 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 1506 may be directly connected to the memory 1504. Furthermore, the CPU 1506 may be directly connected to the GPU 1508. Where direct or point-to-point connections exist between components, the interconnection system 1502 may include PCIe links for implementing the connections. In these examples, the PCI bus does not need to be included in the computing device 1500.

[0182] Memory 1504 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 1500. The computer-readable media may include both volatile and non-volatile media, and removable and non-removable media. For example, but not limited to, the computer-readable media may include computer storage media and communication media.

[0183] Computer storage media may include both volatile and non-volatile media and / or removable and non-removable media implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1504 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 1500. In this specification, computer storage media does not include signals themselves.

[0184] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” can refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. For example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0185] The CPU 1506 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1500 to execute one or more of the methods and / or processes described herein. The CPU 1506 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 1506 may include any type of processor, and depending on the type of computing device 1500 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 1500, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1500 may include one or more CPUs 1506 in one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0186] In addition to or instead of the CPU 1506, the GPU 1508 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 1500 to execute one or more of the methods and / or processes described herein. One or more of the GPUs 1508 may be an integrated GPU (for example, with one or more of the CPUs 1506, and / or one or more of the GPUs 1508 may be discrete GPUs). In embodiments, one or more of the GPUs 1508 may be coprocessors of one or more of the CPUs 1506. The GPUs 1508 may be used by the computing device 1500 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPUs 1508 may be used for General-Purpose computing on a GPU (GPGPU). It may be used for a GPU. The GPU 1508 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU 1508 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU 1506 received via the host interface). The GPU 1508 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 1504. The GPU1508 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU1508 can generate pixel data or GPGPU data for different parts of an output or different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0187] In addition to or instead of the CPU 1506 and / or GPU 1508, the logic unit 1520 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 1500 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 1506, GPU 1508, and / or the logic unit 1520 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 1520 may be part of and / or integrated with one or more of the CPU 1506 and / or GPU 1508, and / or one or more of the logic units 1520 may be discrete components of the CPU 1506 and / or GPU 1508 or otherwise external to them. In embodiments, one or more of the logic units 1520 may be coprocessors of one or more of the CPU 1506 and / or one or more of the GPU 1508.

[0188] Examples of the logic unit 1520 include one or more processing cores and / or components thereof, such as Tensor Cores (TCs), Tensor Processing Units (TPUs), Pixel Visual Cores (PVCs), Vision Processing Units (VPUs), Graphics Processing Clusters (GPCs), Texture Processing Clusters (TPCs), Streaming Multiprocessors (SMs), Tree Traversal Units (TTUs), Artificial Intelligence Accelerators (AIAs), Deep Learning Accelerators (DLAs), Logical Units (ALUs), Application-Specific Integrated Circuits (ASICs), Floating-Point Units (FPUs), Input / Output (I / O) elements, Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) elements, and / or similar.

[0189] The communication interface 1510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1500 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1510 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0190] I / O port 1512 can enable the computing device 1500 to be logically connected to other devices, including I / O components 1514, presentation components 1518, and / or other components, some of which can be built into (e.g., integrated into) the computing device 1500. Exemplary I / O components 1514 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 1514 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of speech recognition, stylus recognition, facial recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 1500 (as described in more detail later). The computing device 1500 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 1500 may include accelerometers or gyroscopes to enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 1500 to render immersive augmented reality or virtual reality.

[0191] The power supply device 1516 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 1516 can provide power to the computing device 1500 to enable the components of the computing device 1500 to operate.

[0192] The presentation component 1518 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 1518 can receive data from other components (e.g., GPU 1508, CPU 1506, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0193] Exemplary data center Figure 16 shows an exemplary data center 1600 that may be used in at least one embodiment of the present disclosure. The data center 1600 may include a data center infrastructure layer 1610, a framework layer 1620, a software layer 1630, and / or an application layer 1640.

[0194] As shown in Figure 16, the data center infrastructure layer 1610 may include a resource orchestrator 1612, grouped computing resources 1614, and node computing resources ("node CRs") 1616(1) to 1616(N), where "N" represents any integer or natural number. In at least one embodiment, the node CRs 1616(1) to 1616(N) may include, but are not limited to, any number of central processing units ("CPUs") or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output ("NW I / O") devices, network switches, virtual machines ("VMs"), power modules, and / or cooling modules. In some embodiments, one or more nodes CR1616(1) to 1616(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CR1616(1) to 16161(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more nodes CR1616(1) to 1616(N) may correspond to a virtual machine (VM).

[0195] In at least one embodiment, the grouped computing resources 1614 may include a separate group of nodes CR1616 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CR1616 within the grouped computing resources 1614 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CR1616, including CPUs, GPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0196] The resource orchestrator 1622 can configure or otherwise control one or more nodes CR1616(1) to 1616(N) and / or grouped computing resources 1614. In at least one embodiment, the resource orchestrator 1622 may include a software design infrastructure ("SDI") management entity for the data center 1600. The resource orchestrator 1622 may include hardware, software, or any combination thereof.

[0197] In at least one embodiment, as shown in Figure 16, the framework layer 1620 may include a job scheduler 1632, a configuration manager 1634, a resource manager 1636, and / or a distributed file system 1638. The framework layer 1620 may include a framework to support the software 1632 of the software layer 1630 and / or one or more applications 1642 of the application layer 1640. The software 1632 or application 1642 may include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure, respectively. The framework layer 1620 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which may use the distributed file system 1638 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1632 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 1600. The configuration manager 1634 may have the ability to configure different layers, for example, a software layer 1630 and a framework layer 1620 including Spark and a distributed file system 1638 to support large-scale data processing. The resource manager 1636 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 1638 and the job scheduler 1632. In at least one embodiment, the clustered or grouped computing resources may include computing resources 714 grouped in the data center infrastructure layer 1610. The resource manager 1036 can coordinate with the resource orchestrator 1612 to manage these mapped or allocated computing resources.

[0198] In at least one embodiment, the software 1632 included in the software layer 1630 may include software used by at least a portion of nodes CR1616(1) to 1616(N), grouped computing resources 1614, and / or the distributed file system 1638 of the framework layer 1620. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0199] In at least one embodiment, application 1642 included in application layer 1640 may include one or more types of applications used by at least a portion of nodes CR1616(1) to 1616(N), grouped computing resources 1614, and / or the distributed file system 1638 of framework layer 1620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0200] In at least one embodiment, any of the configuration manager 1634, resource manager 1636, and resource orchestrator 1612 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 1600 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0201] The data center 1600 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 1600. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 1600 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0202] In at least one embodiment, the data center 1600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) for performing training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0203] Exemplary network environment A network environment suitable for use in implementing the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 1500 in Figure 15—for example, each device may include similar components, features, and / or functionalities of the computing device 1500. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 1600, examples of which are further detailed herein with respect to Figure 16.

[0204] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may include multiple networks, or a network of networks. For example, a network may include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0205] Compatible network environments may include one or more peer-to-peer network environments—in which case the server may not be included in the network environment—and one or more client-server network environments—in which case one or more servers may be included in the network environment. In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0206] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In the embodiment, one or more client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0207] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more parts thereof) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (e.g., one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may delegate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0208] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 1500 described herein with respect to Figure 15. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0209] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0210] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0211] The subject matter of this disclosure is described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the claimed subject matter may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, expressed.

Claims

1. Sensor data acquired using two or more sensors, the step of accessing the sensor data corresponding to two or more sensor modalities, A step of generating a representation of a sequence of annotation scenes including corresponding frames of sensor data from each of the two or more sensors, based on at least one or more representations of temporal offsets that align sensor data corresponding to the two or more sensor modalities, The steps include: generating a representation of a sequence of annotation tasks that, by using a labeling tool, order annotations for at least a portion of the sensor data corresponding to two or more sensor modalities using ground truth annotations based on at least one sensor modality; A step of receiving input using the labeling tool, wherein the input annotates one or more annotation scenes from the sequence of annotation scenes with the set of ground truth annotations defined by at least one annotation task in the sequence of annotation tasks, The steps include exporting the representation of the set of ground truth annotations and Methods that include...

2. The method according to claim 1, wherein the step of generating a representation of the sequence of annotation scenes includes identifying the frame of acquired image data that is temporally closest to the acquired LiDAR spin, and calculating the corresponding of one or more temporal offsets between the start angle of the acquired LiDAR spin and the field of view represented by the acquired image data.

3. The two or more sensors include a LiDAR sensor and one or more cameras, and the multiple annotation tasks are A first annotation task, which includes annotating camera frames within the sequence of the aforementioned annotation scenes, A second annotation task, which includes annotations in two-dimensional space of LiDAR frames within the sequence of the aforementioned annotation scenes, A third annotation task, which includes linking annotations that appear in multiple annotation scenes within the set of annotation scenes, The method according to claim 1, including the method described in claim 1.

4. The sequence of annotation tasks described above is further: A fourth annotation task, which includes annotations in three-dimensional space of the LiDAR frames within the sequence of annotation scenes, A fifth annotation task, which includes linking annotations that appear in both LiDAR frames and camera frames in any specific annotation scene within the sequence of annotation scenes, The method according to claim 3, including the method described in claim 3.

5. The method according to claim 1, wherein the plurality of annotation tasks divide annotations or annotation details of different types of objects into separate annotation tasks.

6. The method according to claim 1, further comprising the step of initializing a set of annotations for a particular annotation scene in the set of annotation scenes among the plurality of annotation tasks, with the labeling tool using a preceding set of annotations for preceding annotation scenes in the set of annotation scenes among the particular annotation tasks.

7. The method according to claim 1, wherein at least one of the annotation tasks restricts a particular sensor modality of the two or more sensor modalities.

8. A particular annotation scene within the set of annotation scenes includes a first frame of the sensor data from a first type of sensor and a second frame of the sensor data from a second type of sensor, and the method is performed by the labeling tool, The steps include determining a portion of the first frame identified by the input probe, The steps include projecting the coordinates of the portion of the first frame onto the coordinates of the corresponding portion of the second frame, The steps of causing visualization of the corresponding portion of the second frame and The method according to claim 1, further comprising:

9. The labeling tool further includes a step of walking through the per-object procedure between the at least one annotation task, wherein the per-object procedure is for at least one annotation of an annotation scene in the sequence of annotation scenes that was received during the previous annotation task in the sequence of annotation tasks. The steps include: causing a zoomed-in view of the annotation within the annotation scene; A step of prompting and accepting input to adjust or confirm the aforementioned annotations. The method according to claim 1, including the method described in claim 1.

10. The aforementioned labeling tool, One or more labeling interfaces of the labeling tool include the step of iterating over the annotation scene, The steps include prompting and accepting input to annotate one or more annotation scenes with a corresponding set of ground truth annotations defined by the at least one annotation task, while causing the presentation of the corresponding frames of the sensor data from each of the two or more sensors, and The method according to claim 1, further comprising:

11. Accessing sensor data acquired during a capture session, wherein the sensor data includes LiDAR frames from a LiDAR sensor and camera frames from at least one camera. A sequence of annotation scenes is constructed based at least on a representation of a temporal offset between a specific LiDAR frame and a specific camera frame, wherein a specific annotation scene within the set of annotation scenes includes the specific LiDAR frame among the LiDAR frames and at least one camera frame among the camera frames. Using a labeling tool, generate a representation of a sequence of annotation tasks that order annotations for at least one portion of the sequence of annotation scenes with ground truth annotations based on at least one sensor modality, Accepting input using the labeling tool, wherein the input annotates one or more of the annotation scenes in the set of annotation scenes in the sequence of ground truth annotations defined by the at least one annotation task in the sequence of annotation tasks, Exporting the representation of the aforementioned set of ground truth annotations One or more processors, each comprising one or more circuits for performing the following tasks.

12. The processor according to claim 11, further comprising one or more circuits that generate at least one annotation scene based on determining that at least one camera frame is temporally closest to at least one LiDAR frame, and calculating at least one temporal offset between at least one acquired LiDAR spin start angle represented by at least one LiDAR frame and the corresponding viewing angle represented by at least one camera frame.

13. The sequence of annotation tasks is, A first annotation task, which includes annotating the set of camera frames within the sequence of the annotation scene, A second annotation task, which includes annotations in two-dimensional space of the set of LiDAR frames within the sequence of annotation scenes, A third annotation task, which includes linking annotations that appear in multiple annotation scenes within the sequence of annotation scenes, The processor according to claim 11, including the following:

14. The processor according to claim 11, wherein the one or more circuits further initialize a set of annotations for at least one annotation scene in the sequence of annotation scenes between at least one annotation task in the sequence of annotation tasks with a set of preceding annotations for a preceding annotation scene in the sequence of annotation scenes between at least one annotation task.

15. The aforementioned one or more circuits may further The correspondence between a portion of the at least one LiDAR frame and a portion of the at least one camera frame is identified, at least based on projecting information from the coordinates of the at least one LiDAR frame to the coordinates of the at least one camera frame. This is intended to elicit a visualization of the aforementioned correspondence. The processor according to claim 11.

16. The aforementioned one or more circuits may further Determine the portion of at least one LiDAR frame identified by the input probe, The coordinates of the portion of the at least one LiDAR frame are projected onto the coordinates of the corresponding portion of the at least one camera frame. This is for causing visualization of the corresponding portion of at least one camera frame. The processor according to claim 11.

17. The one or more circuits further iterate over the per-object steps between the at least one annotation task, wherein the per-object steps have at least one annotation of an annotation scene received during the previous annotation task in the sequence of annotation tasks, To cause the presentation of a zoomed-in view of at least one annotation within the aforementioned annotation scene, Prompting and accepting input to adjust or confirm at least one of the aforementioned annotations The processor according to claim 11, including the following:

18. In a system with one or more processors, The aforementioned one or more processors The method involves generating a representation of a sequence of annotation tasks that uses a labeling tool to order at least a portion of annotations in a sequence of annotation scenes based at least on sensor modalities, wherein at least one annotation scene in the sequence of annotation scenes includes aligned frames of sensor data corresponding to multiple sensor modalities. Using the labeling tool, the sequence of annotation scenes is iterated over in one or more labeling interfaces of the labeling tool, The labeling tool is used to receive an input that annotates at least one annotation scene of the sequence of annotation tasks, while causing the presentation of the aligned set of frames of sensor data corresponding to the plurality of sensor modalities, wherein the input identifies and receives a set of ground truth annotations defined by the at least one annotation task. A system that includes exporting representations of the aforementioned set of ground truth annotations.

19. At least one annotation scene in the sequence of annotation scenes includes a first frame of the sensor data from a first type of sensor and a second frame of the sensor data from a second type of sensor, The aforementioned one or more processors further use the labeling tool, Identifying the correspondence between the portion of the first frame and the portion of the second frame, at least based on projecting information from the coordinates of the portion of the first frame to the coordinates of the portion of the second frame, This will lead to the presentation of a visualization of the aforementioned correspondence. The system according to claim 18, which performs the following:

20. The aforementioned system Control systems for autonomous or semi-autonomous machines, Perceptual systems for autonomous or semi-autonomous machines, A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 18, which is included in at least one of the following.

Images