Generation of ground reality data from deep neural network perception in autonomous driving applications

Abstract

This paper discloses a method for generating ground reality data for deep neural network perception in autonomous driving applications. The annotation pipeline can be used to generate 2D and / or 3D ground reality data for deep neural networks such as autonomous or semi-autonomous vehicle perception networks. Initially, sensor data can be captured and synchronized using different types of sensors to align sensor data frames representing similar world states. Aligned frames can be sampled and packaged into a sequence of annotated scenes to be annotated. Annotation projects can be broken down into modular tasks and encoded into a labeling tool that assigns tasks to labelers and arranges the order of inputs using a wizard that guides the step-by-step execution of tasks. During the task, each type of sensor data in the annotated scene can be presented simultaneously, and information can be projected across sensor modalities to provide useful contextual information. After all annotation tasks have been completed, the generated ground reality data can be exported in any suitable format.

Description

Background Technology

[0001] Designing a system to drive a vehicle autonomously and safely without supervision is extremely difficult. An autonomous vehicle should at least be able to function like a careful driver—utilizing a perception and action system with an impressive ability to identify and respond to moving and static obstacles in complex environments—to avoid collisions with other objects or structures along the vehicle's path. Therefore, the ability to detect instances of animated objects (such as cars, pedestrians, etc.) and other parts of the environment is often critical for autonomous driving perception systems. Traditional perception methods typically rely on cameras or LiDAR sensors to detect objects in the environment, and various methods have been developed using deep neural networks (DNNs) to perform LiDAR and camera perception. Such categories of DNNs include those that perform panoramic segmentation of camera images in perspective views, and those that perform top-down or "bird's-eye view" (BEV) object detection from LiDAR point clouds.

[0002] To train a DNN to perform perception with appropriate accuracy, it needs to be trained using accurate ground reality data. Due to computational limitations, real-time DNN perception is typically performed in two dimensions (2D), so the ground reality data for these networks is usually presented in 2D. However, with technological advancements, three-dimensional (3D) perception is becoming practical, and the demand for high-quality 3D ground reality data remains unmet.

[0003] Conventional techniques for generating ground reality for DNN perception in autonomous driving applications have several drawbacks. Taking the example above, the first DNN performs panoramic segmentation of camera images in a perspective view, while the second DNN performs object detection from a top-down projection of a LiDAR point cloud. In this scenario, the first DNN would require camera images with ground reality annotations, while the second DNN would require top-down LiDAR projections with ground reality annotations. Typically, these types of ground reality annotations are generated in separate labeling processes. However, in some cases, generating accurate labels is difficult or even impossible. For example, LiDAR and RADAR labels generate sparse data, sometimes lacking the granularity and context required for accurate labeling. Distinguishing between pedestrians or bicycles in a top-down view can be challenging or even impossible, as these objects often appear similar to their top-down views compared to other objects such as telephone poles, tree trunks, or shrubs. Therefore, conventional labeling techniques can lead to errors in the ground reality data. Ideally, these errors would be caught during quality checks, but they still negatively impact throughput and efficiency, and waste computational resources. Summary of the Invention

[0004] Embodiments of this disclosure relate to annotation pipelines for generating 2D and / or 3D ground reality data for deep neural networks (DNNs), such as those performing perception in autonomous or semi-autonomous vehicles, robots, or other object types.

[0005] Typically, the annotation pipeline described herein is an improved workflow and software interface that simplifies the generation of high-quality ground-based data. Initially, sensor data can be captured using different types of sensors (sensor modalities) during a capture session. Data from different sensors can be synchronized to align sensor data frames representing similar world states. In the example involving LiDAR and cameras, as the LiDAR spins and observes different parts of the environment, the camera frame that is temporally closest for any given LiDAR spin can be selected based on the camera's field of view relative to the LiDAR spin's initiation angle and how long it takes for the LiDAR spin to align with the camera's field of view. In some embodiments, a time or index offset for each camera relative to the start of the LiDAR spin can be determined and / or applied to align each camera's camera frame with the LiDAR frame. Typically, frames of sensor data from different types can be aligned, sampled, and packaged into a series of annotation scenes to be annotated.

[0006] In some embodiments, annotation items can be broken down into modular tasks that can be assigned to different markers. In a non-limiting example involving cameras and LiDAR, some or all camera images in a sequence can initially be annotated, followed by some or all LiDAR frames in the sequence (e.g., first top-down 2D bounding boxes, then 3D bounding boxes). Objects appearing in consecutive annotation scenes can be linked, and objects appearing in both sensor modalities (LiDAR and camera frames) can be linked together. Annotation tasks can be encoded into a marker tool that assigns tasks to markers and uses a wizard to arrange the order of input, guiding the markers through the task. During the annotation task, each type of sensor data in the annotation scene can be presented to the marker (e.g., side-by-side), and / or information can be projected across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data. In some embodiments, the marker tool can enable the marker to annotate each object in each annotation scene of the sequence.

[0007] After some or all of the annotation tasks in the annotation project have been completed, the resulting ground-based data can be exported in any suitable format. Therefore, the exported ground-based data can be used to train one or more machine learning models. Attached Figure Description

[0008] The system and method of the present invention for generating ground real-time data for deep neural network perception are described in detail below with reference to the accompanying drawings, wherein:

[0009] Figure 1 This is a data flow diagram illustrating an example annotated pipeline according to some embodiments of this disclosure;

[0010] Figure 2 This is a diagram illustrating an example alignment between LiDAR spins and two cameras according to some embodiments of this disclosure;

[0011] Figure 3 This is an example aligned table showing a sensor data index according to some embodiments of this disclosure;

[0012] Figure 4 This is a diagram illustrating examples of data capture sessions, annotation scenarios, and annotation scenario fragments according to some embodiments of the present disclosure;

[0013] Figure 5 These are illustrations of an example user interface for image tagging according to some embodiments of this disclosure;

[0014] Figure 6 This is an illustration of an example user interface for camera-assisted LiDAR tagging according to some embodiments of this disclosure;

[0015] Figure 7 This is an illustration of an example user interface for camera-assisted LiDAR markers with orientation vectors, according to some embodiments of this disclosure;

[0016] Figure 8 This is an illustration of an example user interface for camera-assisted LiDAR markers with a three-dimensional (3D) bounding box, according to some embodiments of this disclosure;

[0017] Figure 9 This is an illustration of an example user interface for LiDAR tracking according to some embodiments of this disclosure;

[0018] Figure 10 This is an illustration of an example user interface for a camera-LiDAR link according to some embodiments of this disclosure;

[0019] Figure 11 These are illustrations of example ground conditions according to some embodiments of the present disclosure;

[0020] Figure 12 This is a flowchart illustrating a method for generating ground condition annotations from sensor data from different types of sensors, according to some embodiments of the present disclosure;

[0021] Figure 13This is a flowchart illustrating a method for generating ground reality annotations for LiDAR and camera frames according to some embodiments of the present disclosure;

[0022] Figure 14A These are illustrations of example autonomous vehicles according to some embodiments of the present disclosure;

[0023] Figure 14B According to some embodiments of this disclosure Figure 14A Examples of camera positions and fields of view for autonomous vehicles;

[0024] Figure 14C According to some embodiments of this disclosure Figure 14A A block diagram of an example system architecture for an example autonomous vehicle;

[0025] Figure 14D It is for one or more cloud-based servers and according to some embodiments of this disclosure. Figure 14A A system diagram illustrating communication between autonomous vehicles;

[0026] Figure 15 This is a block diagram of an example computing device suitable for implementing some embodiments of the present disclosure; and

[0027] Figure 16 This is a block diagram of an example data center applicable to implementing some embodiments of this disclosure. Detailed Implementation

[0028] One potential solution to the aforementioned problems is to project information across sensor modalities, which can assist taggers by providing useful contextual information. However, this possibility presents several challenges. Traditionally, ground-based LiDAR annotation is limited to top-down 2D bounding boxes, but these boxes do not project well into camera space and may not provide sufficient additional context to aid taggers. Furthermore, achieving perfect temporal alignment between different sensor modalities is difficult, if not impossible. Even assuming multiple sensors are configured (e.g., cameras and LiDAR sensors), ideally, when a trigger is applied at a specific time point, all sensors would fire simultaneously. In reality, this ideal is nearly impossible due to challenges such as synchronizing cameras, synchronizing different types of sensors, differences in delay lines, and differences in sampling frequencies (e.g., a camera running at 30fps versus a LiDAR running at 10fps). Taking LiDAR and cameras as a specific example, LiDAR sensor spins over time (e.g., 100 milliseconds per spin), so perfectly synchronizing a particular camera's position relative to the LiDAR spin is challenging and often virtually impossible. Spatial alignment between sensors may also have practical limitations, as individual calibration of each sensor often cannot recover all the degrees of freedom required for perfect alignment. Therefore, cross-sensor modal projection information may be mixed with information from different world states, potentially negating any benefits.

[0029] To address these and other challenges, systems and methods related to annotation pipelines are disclosed that generate 2D and / or 3D ground reality data for deep neural networks (DNNs), such as those performing perception in autonomous or semi-autonomous vehicles, robots, or other object types. Although this disclosure may relate to an example autonomous vehicle 1400 (also referred to herein as "vehicle 1400" or "self-vehicle 1400"), examples thereof are provided herein. Figures 14A-14D The description herein is intended to be descriptive and is not limiting. For example, the systems and methods described herein can be used to generate ground-based training data for DNNs in non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), robots, warehouse vehicles, off-road vehicles, aircraft, boats, and / or other vehicle types. Furthermore, while this disclosure may be described with respect to autonomous driving, this is not intended to be limiting. For example, the systems and methods described herein can be used to generate training data for DNNs in robotics (e.g., path planning for robots), aviation systems (e.g., path planning for drones or other aircraft), boating systems (e.g., path planning for boats or other watercraft), and / or other technical fields, such as localization, path planning, and / or other processes.

[0030] Typically, the annotation pipeline described in this paper is an improved workflow and software interface that streamlines the generation of higher-quality ground-based data than existing technologies. Initially, sensor data is captured using different types of sensors (sensor modalities) during a capture session. To identify useful contextual information that contributes to the annotation task, data from different sensor modalities are synchronized to form a series of annotation scenarios (e.g., sensor datasets acquired approximately at the same time). Desired segments of the sequence can be selected and specified for labeling, and the desired annotations can be decomposed into a set of linear tasks. Different tasks can be split based on sensor type, type of object being labeled, level of annotation detail, and / or other factors. Task inputs can be encoded or otherwise encoded into a labeling tool that assigns tasks to labelers and arranges the input order using a wizard guiding the labelers through the tasks. During the annotation task, each type of sensor data in the annotation scenario can be presented to the labeler (e.g., side-by-side), and / or information can be projected across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data. In some cases, certain types of annotations may be generated automatically. When some or all of the annotation tasks are completed, the marking tool can export the resulting ground fact annotations in any suitable format.

[0031] In some embodiments, annotation projects can be decomposed into modular tasks that can be assigned to different labelers. In a non-limiting example involving cameras and LiDAR, some or all camera images in a sequence are initially labeled, followed by some or all LiDAR frames in the sequence (e.g., first top-down 2D bounding boxes, then 3D bounding boxes). Objects appearing in consecutively annotated scenes can be linked, and then objects appearing in both sensor modalities can be linked together. Additionally or alternatively, annotation projects can be broken down into multiple stages or tasks based on the type of label (e.g., labeling only cars, traffic signs, or some other elements in a road scene in a given task), level of detail (e.g., applying polylines or full 3D bounding boxes to object footprints in a given task), and / or otherwise. By decomposing annotation projects and prompting labelers to perform discrete project-specific tasks, the annotation pipeline can be modularized, allowing labelers to focus on discrete tasks at a time.

[0032] During any given annotation task, the tagging tool's interface can present different types of sensor data (e.g., side-by-side) and / or cross-sensor modal projection information in the annotation scene to provide useful contextual information. Projection information may include sensor detections (e.g., points, planes, scan lines), annotations, input probes specifying a particular location within a sensor data frame, and / or other information. By presenting contextual information during the annotation task, taggers are able to apply more accurate labels. For example, a group of pedestrians might be represented in a particular LiDAR data frame with only a few point detections, but by presenting the corresponding camera frame, a tagger can easily see the pedestrians hidden within the LiDAR data. Similarly, if a tagger is unsure what type of obstacle a particular LiDAR detection represents, in some embodiments, the tagger can click on the LiDAR detection and apply visualization to the corresponding point in the camera frame (e.g., identifying a pickup truck). The tagger may then be able to apply appropriate labels to the LiDAR frame.

[0033] The labeling tools and annotation pipeline described in this paper offer several advantages over existing techniques. Typically, providing additional context to human labelers viewing sensor data can improve the accuracy and efficiency of manual labeling and enable certain types of ground reality labeling that were previously impossible. For example, by presenting additional context (e.g., (stitched) camera images) to LiDAR data, labelers can now accurately generate 3D LiDAR labels, such as 3D bounding boxes. These labels, annotated in LiDAR space, can be projected onto the corresponding camera images, providing useful context for camera labeling, which allows labelers to generate more accurate camera labels. Therefore, the techniques described in this paper provide a specification for a more accurate representation of ground reality, which can be used to train more accurate DNNs.

[0034] Furthermore, providing additional context to the tagger during tagging often makes it easier for them to generate labels. For example, presenting side-by-side views of different types of sensor data and / or illustrating correspondences across sensor modalities makes the data easier for the tagger to understand, reducing cognitive load and accelerating cognition. As a result, the tagging tool described herein makes the interaction between the tagger and the computer more efficient compared to existing technologies. This improvement contributes to the entire workflow in terms of efficiency and throughput, thus reducing the demand for computing resources compared to existing technologies. In other words, the better the tool, the more efficient and efficient the tagging. By making the tool more user-friendly, various aspects of this technology accelerate the annotation workflow by reducing rework from existing technologies.

[0035] Furthermore, splitting annotation tasks and breaking down annotation items into easily digestible tasks simplifies the annotation process and reduces the incidence of information overload. Many companies typically instruct annotators to perform all annotation tasks applicable to each frame of sensor data at once, resulting in a complex annotation process requiring extensive training. By breaking down the required annotations into smaller, more digestible tasks, these tasks can be assigned to many annotators, making the annotation pipeline more scalable. Additionally, by splitting annotation tasks by sensor type, the annotation process is less sensitive to temporal or spatial misalignments between sensor modalities. For example, even if a LiDAR frame and its corresponding camera frame are not perfectly aligned, the impact on label accuracy is minimal because the LiDAR annotation task is performed directly within the LiDAR frame. In this example, the corresponding camera frame is only used for soft guidance, so any misalignment is not hardcoded into ground reality. Finally, in embodiments that include a linking task, where annotators correlate annotations previously made in different sensor modalities and / or annotation scenarios, the linking task can serve as a quality check by placing it after other tasks, as it benefits the annotator by allowing them to review annotations from different sensor modalities and / or annotation scenarios. Therefore, using this linked task eliminates the need for separate quality checks, further improving the overall workflow.

[0036] Therefore, the labeling tools and annotation pipeline described in this paper can be used to generate more accurate ground reality representations, which can be used to train more accurate DNNs.

[0037] Example ground-based annotation pipeline

[0038] Deep neural networks (DNNs) have been used for a variety of tasks, such as object detection and classification. To obtain training data for such DNNs, an annotation pipeline can be used to generate ground reality data. Typically, the ground reality data generated by the annotation pipeline can be customized depending on the type of DNN being trained. For execution-aware DNNs, the annotation pipeline can be customized to generate 2D and / or 3D ground reality data, such as labeled camera images (e.g., in perspective views), LiDAR or RADAR data (e.g., in top-down views), and / or other sensor data. While certain embodiments have been described for execution-aware DNNs, the techniques described herein can be applied to generate ground reality data for other types of DNNs.

[0039] refer to Figure 1 , Figure 1This is a data flow diagram illustrating example annotation pipeline 100 according to some embodiments of the present disclosure. It should be understood that such and other arrangements described herein are merely illustrative examples. Other arrangements and elements (e.g., machines, interfaces, functions, commands, function groups, etc.) may be used in addition to or instead of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and location. The various functions described herein as being performed by entities can be performed by hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory.

[0040] At a high level, the annotation pipeline 100 can include workflows and software interfaces for streamlining the generation of high-quality ground condition data. Figure 1 In the example shown, the annotation pipeline 100 includes data capture 110, sensor data alignment 120, scene generation 130, scene management 140, annotation 150, post-processing 160, quality assurance checks 170, ground reality data export 180, and ground reality data consumption 190. For example, data capture 110 of real-world data can be performed to collect sensor data from different types of sensors (sensor modalities), and sensor data alignment 120 of different types of sensor data can be performed to synchronize the sensor data, allowing sensor data of similar world states (e.g., sensor data captured substantially simultaneously) to be combined and presented during the annotation task. Scene generation 130 can be used to compose a sequence of annotated scenes (e.g., sensor datasets acquired at approximately the same time, such as scene sequence 145). For example, sensor data alignment 120 may involve adding an offset to the capture time of the sensor data or some other index, and scene generation 130 may involve sampling the sensor data and / or generating a projected image for each annotated scene in the sequence. Scene management 140 can be performed to select one or more segments of an annotation scene (e.g., segments of a data capture session without rain) and specify those segments for annotation.

[0041] At a high level, software tools such as network tools (also known as tagging tools) can be used to facilitate annotation 150. Typically, a specific annotation project can be broken down and / or arranged into a set of linear tasks to form a tagging recipe. This can split the project and specify different tasks based on sensor type, type of object being labeled, level of annotation detail, and / or other factors. Task inputs can be encoded or otherwise coded into the tagging tool, which can assign tasks to taggers and guide them through a wizard that arranges the inputs in the order of the annotation tasks. In an example annotation project for LiDAR and camera ground-based data, some or all camera images in the sequence are first labeled, then some or all LiDAR frames in the sequence are labeled (e.g., camera and LiDAR outputs 152), then objects appearing in multiple LiDAR frames are correlated (e.g., LiDAR tracking output 154), and then objects appearing in multiple sensor modalities are linked together (e.g., camera + LiDAR linked output 156). In some embodiments, LiDAR tagging may involve first tagging the LiDAR frame with a 2D bounding box, and then tagging the LiDAR frame with a 3D bounding box or cuboid (e.g., 3D LiDAR tagging output 155). The tagging tool and this example annotation item will be described in more detail below.

[0042] Continuing with the high-level overview, after the labelers have completed the annotation tasks in the project, post-processing 160 can be used to generate annotations that humans typically cannot produce, such as generating depth values. In some embodiments, quality assurance checks 170 can be performed on the labeled data to identify potential errors and label certain annotation scenarios for rework. In some embodiments, quality assurance checks 170 can be at least partially incorporated into the annotation task during annotation 150, such as a linking task, which eliminates the need for separate quality assurance checks after post-processing 160. After some or all of the annotation tasks are completed, the resulting ground truth data can be exported from the labeling tool in any suitable format, whether automatically or manually triggered (Ground Truth Data Export 180), and can be consumed (Ground Truth Data Consumption 190), for example, by using the ground truth data to train a corresponding DNN.

[0043] Typically, ground-based real-world data can be generated at least partially from real-world data. Therefore, in some embodiments, to perform data capture 110, one or more vehicles (e.g., Figure 14AThe vehicle 1400 (D) can collect sensor data from one or more sensors of one or more vehicles in a real-world (e.g., physical) environment. The sensors of one or more vehicles may include, but are not limited to, one or more Global Navigation Satellite System sensors 1458 (e.g., Global Positioning System sensors), one or more RADAR sensors 1460, one or more ultrasonic sensors 1462, one or more LiDAR sensors 1464, one or more Inertial Measurement Unit (IMU) sensors 1466 (e.g., one or more accelerometers, one or more gyroscopes, one or more magnetic compasses, one or more magnetometers, etc.), one or more self-motion sensors, one or more microphones 1496, and one... One or more stereo cameras 1468, one or more wide-angle cameras 1470 (e.g., fisheye cameras), one or more infrared cameras 1472, one or more surround cameras 1474 (e.g., 360-degree cameras), one or more long-range and / or mid-range cameras 1498, one or more speed sensors 1444 (e.g., for measuring the speed of one or more vehicles), one or more vibration sensors 1442, one or more steering sensors 1440, one or more braking sensors (e.g., as part of a braking sensor system 1446), and / or other sensor types. One or more vehicles may include autonomous vehicles, semi-autonomous vehicles, non-autonomous vehicles, and / or may include objects or vehicles other than vehicle 1400, such as robots, drones, boats, aircraft, unmanned aerial vehicles (UAVs), etc.

[0044] One or more vehicles may include various types of vehicle hardware. For example, the vehicle hardware may be responsible for managing sensor data generated by sensors (e.g., a sensor manager using an autonomous driving software stack executed by the vehicle hardware). In some embodiments, one or more vehicles may include an autonomous driving software stack with a world state manager that uses one or more maps (e.g., 3D maps), one or more localization components, one or more perception components, and / or the like to manage the world. The autonomous driving software stack may include one or more planning components (e.g., as part of a planning layer), one or more control components (e.g., as part of a control layer), one or more actuation components (e.g., as part of an actuation layer), one or more obstacle avoidance components (e.g., as part of an obstacle avoidance layer), and / or other components. Typically, in some embodiments, the vehicle hardware may include components for controlling one or more vehicles based on sensor data, one or more machine learning models (e.g., neural networks), and / or the like. Thus, various types of vehicle hardware can be configured to be installed in and / or used by the vehicle when executing an autonomous driving software stack that at least partially controls the vehicle's navigation in one or more real-world physical environments.

[0045] Typically, data capture 110 may involve capturing sensor data by observing the real-world environment using different types of sensors (sensor modalities), such as LiDAR and one or more cameras mounted on a vehicle. Sensor data may typically be acquired from different sensors at different frequencies for various reasons, such as differences in delay lines, sampling frequencies (e.g., a camera operating at 30 fps versus a LiDAR operating at 10 fps), different trigger times, etc. To facilitate grouping and presenting sensor data from similar world states (e.g., sensor data captured substantially simultaneously), sensor data alignment 120 may be performed to synchronize sensor data from different sensor modalities. In some embodiments, a particular sensor may be used as a reference sensor. Non-reference sensors may be referred to as sub-sensors. For a given sensor data frame (reference frame) from the reference sensor, an offset, such as a time increment, can be identified between the reference frame and the sensor data frame that is closest in time from each sub-sensor. The offset for each sub-sensor may be recorded and / or applied to the capture time of sensor data from the sub-sensor or some other index.

[0046] Using LiDAR and cameras as specific examples, in some embodiments, LiDAR can be selected as the reference sensor, and the frame closest in time to each camera can be selected based on the viewpoint of the camera relative to the spin start angle of the LiDAR, and a time or index offset of each camera can be applied relative to the spin start angle of the LiDAR. Figure 2 This is a diagram illustrating an example alignment between a LiDAR spin and two cameras according to some embodiments of this disclosure. Consider a completed LiDAR spin with a reference timestamp t, referring to the spin start time (t+0 ms). Using an example LiDAR sensor, assume that completing one spin (spin duration) takes 100 milliseconds and the spin frequency is 10 Hz. Figure 2 In the example, LiDAR frame 210 is superimposed with a circle to represent the progression of the LiDAR spin over time. In this example, arrow 220 indicates the orientation of the LiDAR spin's field of view center at time t+0 ms (when the spin has already begun). The spin progresses clockwise to arrow 225 at t+50 ms, to arrow 230 at t+75 ms, and returns to arrow 220 at t+100 ms, indicating that the spin has ended.

[0047] exist Figure 2 In this configuration, the fields of view of both the front-facing camera 240 and the rear-facing camera 250 are superimposed on top of the LiDAR frame 210. To identify the frame from each camera that is temporally closest to the LiDAR frame 210, the progression and orientation of the LiDAR spin relative to each camera can be determined, and the offset of each camera relative to the LiDAR can be determined based on the time it takes for the LiDAR spin to reach the camera's field of view. For example, at t+50ms (arrow 225), the LiDAR spin has entered the field of view of the front-facing camera 240, so the front-facing camera 240 can use an offset of 50ms. For some child sensors, such as the rear-facing camera 250, the sensor data observed by the sub-sensor may have sectors corresponding to non-overlapping or incomplete portions of the LiDAR spin, thus making it impossible to select a specific frame of sensor data perfectly aligned with the LiDAR spin from the sub-sensor. Typically, any frame of sensor data from the sub-sensor can be selected. Figure 2 In the example shown, you can select a frame captured at or near the end of the LiDAR spin at t+100ms (arrow 220).

[0048] Example frame selections and corresponding offsets are shown on axis 260, with reference frame 270 (e.g., representing LiDAR frame 210), frame 280 (e.g., representing front camera 240), and frame 290 (e.g., representing rear camera 250). Note that the temporally closest sensor data frame captured by a sub-sensor may not be perfectly aligned with the portion of the LiDAR spin representing the center of the sub-sensor's field of view. For example, a LiDAR frame (e.g., reference frame 270) may represent a world state corresponding to an image captured by the front camera at 50 ms, but the temporally closest image captured by that camera may have already been captured at 40 ms or 45 ms (e.g., frame 280). In this case, the image captured at 45 ms can be selected and paired with the LiDAR frame, and / or the corresponding offset (e.g., 45 ms) can be identified and associated with the corresponding sub-sensor (e.g., front camera 240).

[0049] Typically, the offset identified for each sub-sensor can be recorded and / or applied to the capture time or other indexes of the sensor data from the sub-sensor. For example, suppose a sensor setup produces raw sensor data, where the sensor data from each sensor is indexed individually. In some embodiments, the identified offset for a particular sub-sensor can be applied to adjust the indexing of the sub-sensor's sensor data (or identify aligned indexes of the sensor data). Therefore, determining and / or applying the offset for each sensor can be used to align different types of sensor data (e.g., by aligning their indexes). Figure 3 This is a table illustrating an example alignment of sensor data indexes according to some embodiments of the present disclosure. In this example, the indexes of sensor data are illustrated in columns 320 (LiDAR frame index) and 330-360 (indexes of images captured by different cameras). In this example, scene indices arranged chronologically are included in column 310, and for each scene index value, the corresponding row in the table contains the index identifying the aligned sensor data (e.g., for a specific annotated scene). In this example, an identification offset for each sub-sensor (e.g., each camera) has been applied relative to each reference frame to identify the corresponding index of the frame whose sensor data captured by the sub-sensor is closest in time.

[0050] return Figure 1Scene generation 130 can be performed to compose an annotated scene sequence (e.g., a sensor dataset acquired at approximately a certain time, such as scene sequence 145). Typically, reference and sub-sensor data can be sampled using one or more identified offsets. If the raw sensor data is not in an image format (e.g., LiDAR or RADAAR point cloud), in some embodiments, the raw sensor data (e.g., point cloud) can be projected to form a projected image (e.g., a top-down image). Continuing with the example of using LiDAR as the reference sensor and cameras as sub-sensors, for each frame of LiDAR data (e.g., LiDAR point cloud), an annotated scene can be composed by projecting the LiDAR point cloud to generate a projected image (e.g., a top-down image) and sampled images from each camera (e.g., based on corresponding offsets and / or indices) to identify the temporally closest image captured by each camera. The projected images (LiDAR frames) and camera images (camera frames and / or composite images or panoramas stitched together from multiple images) can be packaged, grouped, or otherwise correlated with each other as an annotated scene. For example, this process can be repeated to generate or otherwise identify annotated scenes for each reference frame. In some embodiments, scene management 140 can be performed to select one or more segments of the annotated scene (e.g., segments from a data capture session without rain) and specify one or more segments for tagging.

[0051] Figure 4 This is a diagram illustrating examples of data capture session 410, annotation scene 415, and segments 421, 422, 423 of the annotation scene according to some embodiments of the present disclosure. It is assumed, for example, that data capture session 410 is performed in the city center of a large city, and that sensors periodically capture sensor data during data capture session 410. As a non-limiting example, LiDAR spin may occur once every 100 ms, and a set of cameras may take a picture every 25 ms (or some other increment). Scene 415 may consist of the captured data (in... Figure 4 (The division between consecutive scenes is not shown in the text). In some cases, not all scenes are useful. For example, in cities that frequently experience rainfall, scene 415 can be filtered out to remove scenes 415 captured during rain during data capture session 410 (e.g., because of raindrops on the windshield or lens). Generally, any type of filter can be applied (e.g., automatic or manual, based on digital or visual features, metadata tags, timestamps, or others). Thus, scene 415 can be curated to identify specific segments of interest (e.g., segments 421, 422, 423), and scenes within the identified segments (e.g., sensor data from different sensor modalities) can be specified (e.g., labeled or otherwise identified) for annotation.

[0052] Return now Figure 1 Annotation 150 can be performed by human annotators using software tools (also known as annotation tools), such as web tools. Typically, the software tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project administrator, identify and / or provide the annotation scene to be annotated (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotations can be broken down and / or arranged into a set of linear tasks, forming a annotation recipe, and the coded representation of the tasks can be input into the annotation tool. Different tasks may be split—the annotation tool may be configured to split and / or encode tasks—based on sensor type, the type or category of objects to be labeled (e.g., in a given task, only labeling cars, traffic signs, or certain other elements in a road scene), the level of annotation detail (e.g., in a given task, labeling bounding boxes with outlines, applying only polylines of object footprints, applying full 3D bounding boxes, applying top-down 2D bounding boxes in LiDAR, and then upscaling to 3D in subsequent channels), and / or other methods. In some embodiments, individual tasks can be entered to mark obstacles, vehicles (e.g., cars, buses, trucks, etc.), vulnerable road users (e.g., motorcycles, bicycles, pedestrians, etc.), environmental elements (e.g., drivable spaces, sidewalks, buildings, trees, utility poles, etc.), their subclasses (e.g., pedestrians), some combination thereof, etc.

[0053] Annotation tasks can be input or otherwise encoded into a tagging tool, and the tagging tool can coordinate the execution of different annotation tasks. For example, the tagging tool can assign tasks to taggers in any suitable manner, such as by assigning tasks based on tagger availability, a specified task order, or other means. In some embodiments, the tagging tool can use a wizard to guide taggers through the task to arrange the order of input (e.g., annotation) for a specific task. During some tasks, each type of sensor data in the annotation scenario can be presented to the tagger (e.g., side-by-side), and / or information can be projected across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data.

[0054] Typically, labeling tools accept input specifying ground-based annotations (e.g., boundaries, closed regions, category labels), and can associate these annotations with sensor data. Sensor data (e.g., LiDAR data frames, RGB images) can be annotated (e.g., manually, automatically, etc.) with labels or other markers that identify the location, geometry, orientation, and / or category of instances of relevant objects in the sensor data. Annotations can be input into the labeling tool using 2D and / or 3D drawing capabilities, another suitable type of software capability, and / or can be hand-drawn and imported. Annotations can typically be synthesized (e.g., generated from computer models or renders), generated practically (e.g., designed and generated based on real-world data), automated by machines (e.g., using feature analysis and learning to extract features from data and then generate labels), human-annotated (e.g., a labeler or annotation expert inputting the annotation), and / or a combination thereof (e.g., manually identifying the vertices of a polyline, with a machine generating polygons using a polygon rasterizer). Annotations can typically include 2D and / or 3D bounding boxes, closed polylines, or other boundary shapes that are drawn, annotated, overlaid, and / or otherwise associated with the sensor data.

[0055] As a non-limiting example involving cameras and LiDAR, annotation projects can specify ordered annotation tasks such that some or all camera images in the sequence are initially labeled, followed by some or all LiDAR frames in the sequence (e.g., first top-down 2D bounding boxes, then 3D bounding boxes), then objects appearing in multiple annotation scenes (e.g., for already labeled camera images and LiDAR frames), and then objects appearing in multiple sensor modalities are linked together. In some embodiments, the labeling tool can simultaneously present two types of sensor data (e.g., LiDAR frames and camera images) in the annotation scene during a particular annotation task, and / or can project information across sensor modalities to provide useful contextual information, such as correspondences between different types of sensor data. In some embodiments, auxiliary features can traverse annotation objects from the previous frame and prompt the labeler to find the corresponding object in the current frame.

[0056] Continuing with examples involving cameras and LiDAR, Figure 5-10This document describes a sample user interface for a marker tool used for auxiliary 3D ground reality markers for LiDAR and camera frames. Consider the following sample annotation project: A first annotation task where one or more markers initially annotate some or all of the camera images in the sequence; a second annotation task where one or more markers annotate top-down 2D bounding boxes in the first pass of some or all of the LiDAR frames in the sequence; a third annotation task where one or more markers annotate 3D bounding boxes in the second pass of the marked LiDAR frames in the sequence; a fourth annotation task where one or more markers link the same objects in different annotation scenes in the sequence; and a fifth annotation task where one or more markers link the same objects in the LiDAR and camera frames in each annotation scene in the sequence. Figure 5 This is a diagram of an example user interface for image labeling (e.g., the first annotation task). Figure 6-8 This is a diagram of an example user interface for camera-assisted LiDAR labeling (e.g., the second and third annotation tasks). Figure 9 This is a diagram of an example user interface for LiDAR tracing (e.g., the fourth annotation task), and Figure 10 This is an illustration of an example user interface for camera-LiDAR linking (e.g., fifth annotation task) according to some embodiments of this disclosure.

[0057] Turn now Figure 5 , Figure 5An example user interface 500 for image labeling according to some embodiments of the present disclosure is shown. User interface 500 includes a panel presenting a specific camera image 510 to be labeled, and a labeling panel with various interactive elements that activate various drawing and / or annotation functions. In this example, the labeling panel includes a labeling button 560 that, when active, allows the user to identify and label the boundaries and / or corresponding enclosed regions of image 510. Typically, user interface 500 can incorporate any known 2D or 3D drawing or annotation software functionality. As a non-limiting example, a user can specify a polygon (e.g., a 2D bounding box) by clicking (e.g., at the location identifying an initial corner of the bounding box), dragging to expand the 2D bounding box from the initial corner, and releasing when the bounding box reaches the desired size (e.g., at the location identifying a second corner opposite the initial corner). In some embodiments, user interface 500 may allow the user to zoom in on image 510 and / or pan over image 510. Thus, the user can draw or otherwise identify any number of regions (e.g., region 520). In some embodiments (e.g., in any user interface 500-1000), the tagging tool can initialize a set of annotations using annotations from the previous frame of sensor data, and the tagger can adjust the annotations to fit the current frame. Therefore, the tagging tool can reduce or eliminate the need to recreate annotations for the same object over and over again.

[0058] In some embodiments (e.g., any of user interfaces 500-1000), the labeling tool may include a labeling panel that includes a list (e.g., list 570) or other identifiers of annotation objects in a specific frame of sensor data. Each entry in the list may include one or more interactive elements, such as a label button 580 and a comment button 585. The label button 580 (or some other interactive element) may be used to select or identify a corresponding annotation in the image (e.g., input that enables editing annotations), prompt the user to select or edit a category label, or otherwise. The comment button 585 may accept text input specifying any notes about the annotation. These are merely examples, and any suitable drawing or annotation functionality may be incorporated into the labeling tool. In some embodiments, the labeling tool may accept input that navigates back and forth in time (i.e., the previous frame and the next frame in a sequence) to improve the user's understanding of the data. When the user completes labeling the image, the user can provide an indication of completion (e.g., by clicking the submit button 590).

[0059] Figure 6This is an illustration of an example user interface 600 for camera-assisted LiDAR labeling according to some embodiments of the present disclosure. The user interface 600 includes a panel that presents a specific LiDAR frame 620 to be labeled. In some embodiments, the user interface 600 may simultaneously present the specific LiDAR frame 620 to be labeled and a corresponding image 610 from the same annotation scene. For example, the presented image may be a spatially registered 360-degree view image composed of images stitched together from the same annotation scene from surrounding cameras to form a composite image. Typically, the user interface 600 may present a visualization of the correspondence (e.g., corresponding regions) between the LiDAR frame 620 and the image 610. In some embodiments, the user interface 600 may allow the user to zoom and / or pan in one of the frames, and the user interface 600 may make corresponding adjustments to the other.

[0060] In some embodiments, the labeling tool can project information across sensor modalities to illustrate correspondences across sensor data. For example, the known orientation and position of the camera capturing a particular image can be used to deproject the image onto a 3D representation of the environment (e.g., 3D LiDAR coordinates) and identify the 3D location (e.g., in LiDAR space) corresponding to a particular image pixel. To project in another direction, a specific 3D location in LiDAR space can be projected into image space using the known orientation and position of the camera capturing the image. Typically, various types of information can be projected from the LiDAR frame into the corresponding image, such as detections (e.g., points, planes), annotations, or other regions. In some embodiments, the user interface 600 can project the position of an input probe across sensor modalities, which moves with user input. For example, when the user hovers the mouse over LiDAR frame 620, input probe 640 (also shown in magnified region 630) can be used to identify the corresponding 3D location in LiDAR space (e.g., by setting the z-value to zero, fitting the z-value of the ground plane, etc.) and can be projected into image space to illustrate the corresponding point 650 in image 610. In some embodiments, the input probe may be projected in the opposite direction, either additionally or alternatively (e.g., a designated point of image 610 may be projected onto LiDAR frame 620). By illustrating the correspondence across sensor modalities, user interface 600 can provide useful contextual information to help annotators generate more accurate annotations.

[0061] In some embodiments, as an initial step for each LiDAR spin, the marker may be prompted to fit a ground plane to the LiDAR spin. In some embodiments, the fitted ground plane from a previous LiDAR spin may be propagated to the current spin, and the marker may be prompted to fine-tune the previous plane to the current plane, which can speed up the marker process. In some embodiments, the data-fitted ground plane can provide z-values ​​for top-down LiDAR annotations. Therefore, the received annotations can be aligned with a specified ground plane, which can improve the accuracy of top-view annotations. Additionally or alternatively, in some embodiments, the user interface 600 may use annotations from previous LiDAR spins to initialize a set of annotations, and the marker may adjust the annotations to fit the current spin. Therefore, the marker tool can reduce or eliminate the need to recreate annotations for the same object over and over again.

[0062] In some embodiments, the user interface 600 may accept input annotations specifying a 2D polygonal region of LiDAR frame 620 and / or the orientation of an object represented by the 2D polygonal region. Since LiDAR frame 620 may be a projection of 3D data (e.g., a LiDAR point cloud), the input annotations can be adapted to the corresponding 3D representation in LiDAR space. For example, in some cases where LiDAR frame 620 is a top-down projected image, an annotated region (e.g., with a z-value of zero, fitting the z-value of the ground plane) can be created in the XY plane of the LiDAR coordinate system, and the 2D annotated region can be extended to 3D (e.g., in subsequent annotation scenarios, such as in a subsequent annotation task), as explained in more detail below.

[0063] Figure 7 This is an illustration of an example user interface 700 for camera-assisted LiDAR labeling using orientation vectors, according to some embodiments of this disclosure. Similar to user interface 600, user interface 700 can accept input annotations specifying a polygonal region of LiDAR frame 720. In this example, the labeler has specified a bounding box 730. In some embodiments, the labeling tool can accept input specifying an orientation vector 740 (e.g., by clicking and dragging in the orientation direction), category labels 750, or other types of annotations. Similar to user interface 600, user interface 700 can present a visualization of the correspondence (e.g., corresponding regions) between LiDAR frame 720 and image 710 to assist the annotation task. In this example, the category and orientation of the object represented in LiDAR frame 720 are perceptible in image 710. As a result, by presenting image 710 in association with (e.g., simultaneously) LiDAR frame 720, the labeler is able to perceive the category and orientation, and thus can encode one or more corresponding annotations.

[0064] Figure 8This is an illustration of an example user interface 800 for camera-assisted LiDAR tagging utilizing 3D bounding boxes, according to some embodiments of this disclosure. In some embodiments, camera-assisted LiDAR tagging can be decomposed into separate tasks for 2D and 3D annotation. 3D annotations can be generated manually and / or automatically. For example, LiDAR tagging can be decomposed into a first pass that accepts input specifying 2D annotations (e.g., 2D bounding boxes), and a second pass that accepts input adjusting the 2D annotations to 3D annotations (e.g., 3D bounding boxes), or a second stage that automatically generates 3D annotations from 2D annotations (e.g., as a second pass, as a third pass, as a fourth pass, as a fifth pass, as a sixth ... Figure 1 (As part of post-processing 160). In an example embodiment involving accepting input that adapts 2D annotations to 3D annotations, after receiving specifications of 2D annotations (e.g., bounding boxes) during an initial pass through a series of top-down LiDAR frames, the tagging tool can guide the tagger through the process for each object. In this example, for each object (e.g., each 2D annotation specified during the first pass), the tagging tool can render an object-aligned, magnified view of the object in the current LiDAR frame. For example, the raw data of the current LiDAR frame (e.g., LiDAR point cloud) can be projected into one or more views, such as a front view of the object (e.g., front view 830), a side view of the object (e.g., side view 820), and a top-down view of the object (e.g., top-down view 840), each of which can be rendered in the user interface 800. In some embodiments, a 3D representation of the LiDAR point cloud (e.g., 3D view 810) can be additionally or alternatively rendered.

[0065] User interface 800 can prompt the annotator to specify the size and / or orientation (e.g., yaw, pitch, roll) of each object in any view. In the example of the first pass in 2D, initial 2D annotations can be presented (in each of views 810, 820, 830, and 840). Note that 2D annotations in the top-down views can initially be visualized as a line in the front view 830 and the side view 820, and / or can be visualized in both dimensions in the top-down view 840 and the 3D view 810. Figure 8In the example shown, the marker can adapt a 2D annotation to a 3D annotation or otherwise specify a 3D annotation, for example, by placing vertices, dragging handles, or otherwise defining or manipulating the representation of the annotation in any view (e.g., bounding boxes 815, 825, 835, 845 in the respective views 810, 820, 830, 840). In some embodiments, modifying an annotation in one view updates the annotation in any other view that is visible to the modified annotation. Thus, the marker can fit 3D annotations to LiDAR data (e.g., a LiDAR point cloud). After the marker specifies a 3D annotation for a particular object, the user can submit the annotation (e.g., by clicking the Next button or activating some other interactive element) to proceed to the next object in the LiDAR frame.

[0066] Figure 9 This is an illustration of an example user interface 900 for LiDAR tracking according to some embodiments of this disclosure. In this example, user interface 900 presents consecutive LiDAR frames 910 and 920, and corresponding images 930 and 940 from the same annotation scene. A tagging tool can iterate through objects in the previous frame (e.g., annotations entered during a previous annotation task), and user interface 900 can prompt the tagger to find the corresponding annotation (e.g., entered during a previous annotation task) in the current frame. In some embodiments, object detection and tracking can be applied to track the movement of annotated objects from frame to frame over time, and / or to initialize links for human tagger confirmation or fine-tuning. Figure 9 In the example shown, the marker can identify annotations 950 and 960 as the same object. In some cases, an object may be occluded in some frames but not in others. Therefore, in some embodiments, the marker tool can accept input that links objects (annotations) across disjoint frames (e.g., skips).

[0067] In some embodiments, the user interface 900 may emphasize the estimated region in which an object is predicted to be located. For example, one or more position values ​​of the current or selected object in the first frame (e.g., LiDAR data) may be adjusted to compensate for known ego motion of the sensor capturing the data (e.g., known ego motion of the data acquisition vehicle). As a non-limiting example, one or more position values ​​of the annotation of an object (e.g., a bounding box) and / or one or more representative position values ​​of the object in the first frame of the LiDAR data (e.g., the center of the bounding box, a corner of the bounding box) may be subject to ego motion compensation, and the user interface 900 may highlight, outline, pan to, zoom to, and / or otherwise emphasize the corresponding predicted region in the second frame of the LiDAR data. In an example implementation, as the marker iterates through the marked objects from previously annotated scenes, the user interface 900 may guide the marker by presenting a visualization of the predicted locations of the marked objects from previously annotated scenes in subsequent annotated scenes, for example, by zooming in on the estimated regions in the sensor data. In this way, the user interface 900 may guide the marker from scene to scene to identify the corresponding object. In some embodiments, the user interface 900 can still provide useful guidance even with a relatively low sampling rate because self-motion compensation can still provide useful predictions at lower sampling rates.

[0068] exist Figure 9 In this embodiment, the user interface 900 includes a label panel 970, which includes various interactive elements that activate corresponding annotation functions. For example, a label pair button 975, when activated, allows the user to identify and link annotations (e.g., enter a new link). One of the mismatch reasons 980 can be selected to indicate that an annotation from one frame is not visible or unlabeled in adjacent frames. In some embodiments, the label panel 970 can present thumbnail images of consecutive frames being presented (e.g., thumbnails 985a, 985b) and can accept input regarding annotation issues for camera and / or LiDAR frames. Therefore, the annotation task of annotating links across sensor data frames can be at least partially used as a quality assurance check (e.g., Figure 1 (Quality assurance check 170). When the marker completes linking annotations in a pair of adjacent frames, the marker can confirm the link to proceed to the next pair of adjacent frames. Although this example illustrates the annotation task in which one or more markers link objects appearing in multiple LiDAR frames, some embodiments may additionally or alternatively include the annotation task in which one or more markers link objects appearing in multiple camera frames.

[0069] Figure 10This is an illustration of an example user interface 1000 for camera-LiDAR linking according to some embodiments of this disclosure. In this example, user interface 1000 presents a LiDAR frame 1020 and a corresponding image 1010 from the same annotation scene. A tagging tool can traverse objects in the annotation scene (e.g., annotations entered during a previous annotation task), and user interface 1000 can prompt the tagger to find corresponding annotations in other sensor modalities. Typically, annotations, object trajectories, and / or object detections from sensor data of a particular sensor can be linked to corresponding annotations, object trajectories, and / or object detections of the same object from sensor data of different sensors. For example, a tagger can identify annotations 1040 and 1050 as the same object.

[0070] In some embodiments, the user interface 1000 may emphasize the estimated region in which an object is predicted to be located. For example, one or more position values ​​of the current or selected object in a first frame (e.g., a camera frame or a LiDAR frame) may be projected into a second frame (e.g., a LiDAR frame or a camera frame) and adjusted to compensate for known self-motion of one or more sensors capturing the data (e.g., known self-motion of the data acquisition vehicle). As a non-limiting example, one or more position values ​​of the annotation of an object (e.g., a bounding box) and / or one or more representative position values ​​of the object in the first frame of sensor data (e.g., the center point of the bounding box, a corner of the bounding box) may be projected into the second frame and self-motion compensated, and the user interface 1000 may highlight, outline, translate, zoom in, and / or otherwise emphasize the corresponding predicted region in the second frame. In this way, the user interface 1000 can utilize known correspondences between sensor modalities to guide taggers in identifying corresponding objects across sensor modalities.

[0071] Similar to user interface 900, user interface 1000 includes a labeling panel 1060, which includes various interactive elements that activate corresponding annotation functions. In some embodiments, user interface 900 may accept input for labeling camera and / or LiDAR frame annotation questions. Therefore, annotation tasks that annotate across sensor modal links can be used at least in part as quality assurance checks (e.g., Figure 1(Quality assurance check 170). When the marker completes the cross-sensor modality linking annotation in the annotation scene, the marker can confirm the 990 link to proceed to the next pair of adjacent frames. Although the preceding discussion focused on LiDAR-to-camera linking, any type of sensor data can be linked to any other type of sensor data (including links between two different types of sensor data from the same sensor, such as camera-to-camera links). Additionally or alternatively, annotation tasks can be combined, reordered, split, or otherwise arranged. For example, in some embodiments, LiDAR tagging can be performed in the same annotation task as LiDAR-to-camera linking (e.g., after tagging objects in a camera image, simultaneously linking and tagging corresponding objects in a LiDAR frame). These are merely examples, and other variations can be implemented within the scope of this disclosure.

[0072] Return now Figure 1 In some embodiments, post-processing 160 may be performed to automatically detect features that are not normally perceptible to humans. As a non-limiting example, image processing may be applied to camera images to determine dense depth values ​​(e.g., using one or more machine learning models to predict depth values ​​based on each pixel), and depth values ​​or subsets thereof (e.g., representative depths for each annotation, such as closest or average depths) that can be associated with the image. In some cases, post-processing 160 may be rerun (e.g., based on improved calibration) to recalculate any automatically detected features and / or improve the visualization of correspondences (e.g., by backprojecting 3D LiDAR labels onto the corresponding camera images). In some embodiments, annotation items may include one or more manual quality assurance checks 170 to verify the accuracy of annotations manually generated by human annotators and / or automatically generated (e.g., during post-processing 160). Figure 11 These are illustrations of example ground-based annotations based on some embodiments of the present disclosure.

[0073] In some embodiments, post-processing 160 can be performed to convert ground reality annotations into encoded representations that match the view, size, and dimensions of one or more outputs of one or more machine learning models to be trained. For example, if one or more machine learning models output categorical data (e.g., one or more channels, where each channel outputs a different class confidence map), ground reality annotations in a given sensor data frame can be converted into a corresponding class confidence map for each class. As a non-limiting example, for a given class, the values ​​of pixels falling within the annotated regions of that class can be set to values ​​indicating positive classification (e.g., 1), and the values ​​of other pixels in the image can be set to values ​​indicating negative classification (e.g., 0). Thus, different class confidence maps can be stacked to form a ground reality tensor that matches the output of one or more machine learning models.

[0074] In another example, if one or more machine learning models output instance regression data (e.g., one or more channels, where each channel regresses object instance data of different types, such as location, geometry, and / or orientation data, the location, geometry, orientation, and / or category of each annotation can be used to generate object instance data that matches the view, size, and dimensions of one or more outputs of one or more machine learning models to be trained. For example, for each pixel contained in the annotation, the annotation can be used to compute the corresponding location, geometry, and / or orientation information (e.g., the location of the object—e.g., the object center—relative to each pixel, the object height, the object width, the object orientation (e.g., the spin angle of the orientation relative to the projected image), and / or similar information). The computed object instance data can be stored in the corresponding channel of the ground reality tensor. These are just a few examples; other types of post-processing can be performed additionally or alternatively.160

[0075] After some or all of the annotation tasks in the annotation project have been completed, the resulting ground data can be exported in any suitable format (e.g., Figure 1 The ground-based data can be exported (180). Ground-based data can be paired with corresponding input training data that matches one or more types of inputs accepted by one or more machine learning models to be trained. Therefore, one or more machine learning models can be trained using the input training data and the exported ground-based data (e.g., Figure 1The ground-based data consumption is 190%. For example, one or more loss functions (e.g., a single loss function, loss functions for each output type, such as classification loss and / or regression loss, etc.) can be used to compare the accuracy of one or more outputs of one or more machine learning models with ground-based data, and the parameters of one or more machine learning models can be updated (e.g., using backpropagation, forward propagation, etc.) until the accuracy reaches an optimal or acceptable level.

[0076] Now for reference Figure 12 and Figure 13 Each block of methods 1200 and 1300 described herein includes a computational process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by a processor executing instructions stored in memory. The method can also be embodied as computer-usable instructions stored on a computer storage medium. These methods can be provided by standalone applications, services, or managed services (standalone or in combination with another managed service) or plug-ins to another product, to name a few. Furthermore, as an example, methods 1200 and 1300 can be relative to… Figure 1 The annotations in pipeline 100 are for illustrative purposes. However, these methods may be additionally or alternatively implemented by any system or any combination of systems, including but not limited to those described herein.

[0077] Figure 12 This is a flowchart illustrating a method 1200 for generating ground reality annotations from sensor data of different types of sensors according to some embodiments of the present disclosure. In block B1202, method 1200 includes accessing sensor data captured using different types of sensors during a capture session. For example, one or more vehicles (e.g., Figure 14A -D vehicles 1400) can be used in real-world (e.g., physical) environments (e.g., as...). Figure 1 (Data capture 110 is a part of the collection of sensor data from one or more sensors of one or more vehicles. Sensor data can be stored and accessed in any way.)

[0078] In box B1204, method 1200 includes identifying at least one offset of synchronized sensor data to form an annotated scene sequence, each annotated scene comprising sensor data frames from two or more different types of sensors. For example, since sensor data can be obtained from different sensors at different frequencies, the sensor data can be aligned (e.g., Figure 1Sensor data alignment (120) facilitates grouping sensor data from similar world states. In some embodiments, a particular sensor may be used as a reference sensor. Non-reference sensors may be referred to as sub-sensors. For a given sensor data frame (reference frame) from the reference sensor, an offset, such as a time increment, can be identified between the reference frame and the sensor data frame from each sub-sensor that is closest in time. The offset for each sub-sensor may be recorded and / or applied to the acquisition time of the sensor data from the sub-sensor or some other index. Typically, reference and sub-sensor data may be sampled using one or more identified offsets to identify and compose sensor data frames from different sensors in each annotated scene.

[0079] In box B1206, method 1200 includes encoding multiple linear annotation tasks into a labeling tool to annotate annotated scenes using ground-based annotation. Typically, the labeling tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project administrator, identify and / or provide the annotation scene to be labeled (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotation can be decomposed into a set of linear tasks, and the encoded representations of the tasks can be input into the labeling tool (e.g., as ground-based annotation). Figure 1 (Part of note 150 in note pipeline 100).

[0080] In box B1208, method 1200 includes assigning annotation tasks to taggers using a tagging tool. For example, as Figure 1 As part of the annotation 150 in the annotation pipeline 100, the tagging tool can assign each annotation task to a specific tagger in any suitable manner, such as by assigning tasks based on tagger availability, specified task order, or other means.

[0081] In box B1210, method 1200 includes using a tagging tool to guide the tagger through annotating a sequence of scenes. For example, as Figure 1 As part of the annotation 150 in the annotation pipeline 100, the tagging tool can use a wizard that guides the tagger through one or more tasks to arrange the input order of a specific task (e.g., annotations).

[0082] In box B1212, method 1200 includes, for each annotation scenario, using tagging tooltips and accepting input specifying a set of ground reality annotations defined by the annotation task, while presenting sensor data from two or more different types of sensors in the annotation scenario. For example, as Figure 1 As part of annotation 150 in the annotation pipeline 100, the tagging tool can arrange the order of inputs for a specific task (e.g., by iterating through the annotation process of each object for each annotation scene in the sequence). Figure 6 An example embodiment is illustrated, in which the labeling tool can simultaneously present sensor data from different types of sensors and can project information across sensor modalities to show the correspondence across sensor data.

[0083] In box B1214, method 1200 includes exporting a representation of the ground condition annotations. For example, after some or all of the annotation tasks in an annotation project have been completed, the resulting ground condition data can be exported in any suitable format (e.g., Figure 1 Export ground condition data (180).

[0084] Figure 13 This is a flowchart illustrating a method 1300 for generating ground reality annotations of LiDAR and camera frames according to some embodiments of the present disclosure. At block B1302, method 1300 includes accessing sensor data captured during a capture session, the sensor data including LiDAR frames from a LiDAR sensor and camera frames from at least one camera. For example, one or more vehicles (e.g., Figure 14A -D vehicles 1400) can be used in real-world (e.g., physical) environments (e.g., as...). Figure 1 The data capture (a portion of 110) collects sensor data from one or more sensors on one or more vehicles. Vehicle sensors may include one or more LiDAR sensors and one or more cameras. The captured sensor data can be accessed at any time.

[0085] In box B1304, method 1300 includes identifying an offset for each camera in at least one camera, synchronizing camera frames with LiDAR frames to form an annotated scene sequence, each annotated scene including a LiDAR frame and at least one camera frame. For example, as the LiDAR spin proceeds and different parts of the environment are observed, the temporally closest camera frame for any given LiDAR spin can be selected based on the camera's viewing angle relative to the LiDAR spin start angle and how long it takes for the LiDAR spin to align with the camera's field of view (e.g., a portion such as the center of the camera's field of view). Typically, a time or index offset relative to the start of the LiDAR spin for each camera can be determined and / or applied to align the camera frame for each camera with the LiDAR frame. Therefore, each LiDAR frame and the temporally closest camera frame (e.g., for each camera) can be sampled and packaged into the corresponding annotated scene to form a sequence.

[0086] In box B1306, method 1300 includes encoding multiple linear annotation tasks into a labeling tool to annotate an annotation scene using ground-based annotation. Typically, the labeling tool may include one or more interfaces (e.g., a graphical user interface) that accept input from a project administrator, identify and / or provide the annotation scene to be labeled (e.g., sensor data from different sensor modalities) and one or more annotation tasks. The desired annotation can be decomposed into a set of linear tasks, and the encoded representations of the tasks can be input into the labeling tool (e.g., as ground-based annotation). Figure 1 (Part of note 150 in note pipeline 100).

[0087] In box B1308, method 1300 includes using a tagging tool to guide the tagger through annotating a sequence of scenes. For example, as Figure 1 As part of the annotation 150 in the annotation pipeline 100, the tagging tool can use a wizard that guides the tagger through one or more tasks to arrange the input (e.g., annotation) order for a specific task.

[0088] In box B1310, method 1300 includes, for each annotation scene, using marker tooltips and accepting input specifying a set of ground reality annotations defined by the annotation task, while presenting LiDAR frames and at least one camera frame from the annotation scene. For example, as Figure 1 As part of annotation 150 in the annotation pipeline 100, the tagging tool can arrange the order of inputs for a specific task (e.g., by iterating through the annotation process of each object for each annotation scene in the sequence). Figure 6 An example embodiment is shown, in which the marking tool can simultaneously render LiDAR and camera frames, and can project information from LiDAR to the camera and / or vice versa.

[0089] In box B1312, method 1300 includes exporting a representation of the ground condition annotations. For example, after some or all of the annotation tasks in an annotation project have been completed, the resulting ground condition data can be exported in any suitable format (e.g., Figure 1 The ground-based real-time data is exported (180). In this way, one or more machine learning models can be trained using the exported ground-based real-time data (e.g., Figure 1 Ground real-time data consumption: 190.

[0090] Example autonomous vehicles

[0091] Figure 14AThis is an illustration of an example autonomous vehicle 1400 according to some embodiments of the present disclosure. The autonomous vehicle 1400 (or, alternatively, referred to herein as “vehicle 1400”) may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first-response vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire trucks, police vehicles, ambulances, boats, construction vehicles, underwater vessels, drones, and / or other types of vehicles (e.g., driverless and / or vehicles accommodating one or more passengers). Autonomous vehicles are generally described according to the level of automation defined by a branch of the U.S. Department of Transportation—the National Highway Traffic Safety Administration (NHTSA)—and the Society of Automotive Engineers (SAE) in its “Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles” (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and previous and future versions of that standard). Vehicle 1400 may be able to perform one or more functions that meet the requirements of Level 3 to Level 5 autonomous driving. For example, depending on the embodiment, vehicle 1400 may be able to perform conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5).

[0092] Vehicle 1400 may include components such as chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. Vehicle 1400 may include a propulsion system 1450, such as an internal combustion engine, a hybrid power plant, an all-electric motor, and / or another type of propulsion system. Propulsion system 1450 may be connected to the drivetrain of vehicle 1400, which may include a transmission, to enable propulsion of vehicle 1400. Propulsion system 1450 may be controlled in response to receiving a signal from throttle / accelerator 1452.

[0093] A steering system 1454, which may include a steering wheel, can be used to steer the vehicle 1400 (e.g., along a desired path or route) when the propulsion system 1450 is operating (e.g., when the vehicle is in motion). The steering system 1454 may receive signals from the steering actuator 1456. For fully automatic (level 5) functionality, the steering wheel may be optional.

[0094] The brake sensor system 1446 can be used to operate the vehicle brakes in response to receiving signals from the brake actuator 1448 and / or the brake sensor.

[0095] It may include one or more System-on-Chip (SoC) 1404 ( Figure 14C One or more controllers 1436, including and / or one or more GPUs, may provide signals (e.g., signals representing commands) to one or more components and / or systems of vehicle 1400. For example, one or more controllers may send signals to operate vehicle brakes via one or more brake actuators 1448, to operate steering system 1454 via one or more steering actuators 1456, and to operate propulsion system 1450 via one or more throttles / accelerators 1452. One or more controllers 1436 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operating commands (e.g., signals representing commands) to enable autonomous driving and / or assist a human driver in driving vehicle 1400. One or more controllers 1436 may include a first controller 1436 for autonomous driving functions, a second controller 1436 for functional safety functions, a third controller 1436 for artificial intelligence functions (e.g., computer vision), a fourth controller 1436 for infotainment functions, a fifth controller 1436 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 1436 can handle two or more of the functions described above, and two or more controllers 1436 can handle a single function, and / or any combination thereof.

[0096] One or more controllers 1436 may provide signals for controlling one or more components and / or systems of vehicle 1400 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, global navigation satellite system sensors 1458 (e.g., GPS sensors), RADAR sensors 1460, ultrasonic sensors 1462, LIDAR sensors 1464, inertial measurement unit (IMU) sensors 1466 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 1496, stereo cameras 1468, wide-angle cameras 1470 (e.g., fisheye cameras), infrared cameras 1472, surround cameras 1474 (e.g., 360-degree cameras), long-range and / or medium-range cameras 1498, speed sensors 1444 (e.g., for measuring the rate of vehicle 1400), vibration sensors 1442, steering sensors 1440, braking sensors (e.g., as part of braking sensor system 1446), and / or other sensor types.

[0097] One or more of the controllers 1436 may receive inputs (e.g., represented by input data) from the instrument cluster 1432 of the vehicle 1400 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 1434, an auditory signaling device, a speaker, and / or via other components of the vehicle 1400. These outputs may include information such as vehicle speed, rate, time, map data (e.g., [missing information]). Figure 14C Information such as the HD map 1422, location data (e.g., the location of vehicle 1400 on the map), direction, and the location of other vehicles (e.g., occupying a grid), as well as information about objects and their states perceived by the controller 1436, etc. For example, the HMI display 1434 may display information about the existence of one or more objects (e.g., street signs, warning signs, traffic light changes, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, leaving 34B in two miles, etc.).

[0098] The vehicle 1400 also includes a network interface 1424, which can communicate via one or more networks using one or more wireless antennas 1426 and / or a modem. For example, the network interface 1424 may be able to communicate via LTE, WCDMA, UMTS, GSM, CDMA2000, etc. One or more wireless antennas 1426 may also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using one or more local area networks such as Bluetooth, Bluetooth LE, Z-Wave, ZigBee, etc., and / or one or more low-power wide area networks (LPWAN) such as LoRaWAN, SigFox, etc.

[0099] Figure 14B For use in accordance with some embodiments of this disclosure Figure 14A This is an example of the camera position and field of view of an autonomous vehicle 1400. The camera and its respective field of view are an example embodiment and are not intended to be limiting. For example, additional and / or replaceable cameras may be included, and / or these cameras may be located at different positions on the vehicle 1400.

[0100] The camera type used for the camera may include, but is not limited to, a digital camera suitable for use with components and / or systems of vehicle 1400. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. The camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc., depending on the embodiment. The camera may be able to use a rolling shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include a red-white-white-white (RCCC) color filter array, a red-white-white-blue (RCCB) color filter array, a red-blue-green-white (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, a sharp-pixel camera, such as a camera with RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to improve light sensitivity.

[0101] In some examples, one or more of the cameras can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function monocular camera can be installed to provide functions including lane departure warning, traffic sign assistance, and intelligent headlight control. One or more of the cameras (e.g., all cameras) can simultaneously record and provide image data (e.g., video).

[0102] One or more of the cameras can be mounted in mounting components such as custom-designed (3-D printed) parts to cut off stray light and reflections from inside the vehicle (e.g., reflections from the dashboard in the windshield mirror) that may interfere with the camera's image data capture capabilities. Regarding the wing mirror mounting components, the wing mirror components can be custom-3-D printed so that the camera mounting plate matches the shape of the wing mirror. In some examples, one or more cameras can be integrated into the wing mirror. For side-view cameras, one or more cameras can also be integrated into the four pillars at each corner of the cab.

[0103] A camera with a field of view that includes the environment in front of the vehicle 1400 (e.g., a front-facing camera) can be used for surround view to help identify forward paths and obstacles, and, with the assistance of one or more controllers 1436 and / or control SoCs, to provide information crucial for generating an occupancy grid and / or determining a preferred vehicle path. The front-facing camera can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. The front-facing camera can also be used in ADAS functions and systems, including Lane Departure Warning (“LDW”), Autonomous Cruise Control (“ACC”), and / or other functions such as traffic sign recognition.

[0104] A variety of cameras can be used in front-facing configurations, including, for example, monocular camera platforms including CMOS (Complementary Metal-Oxide-Semiconductor) color imagers. Another example could be a wide-angle camera 1470, which can be used to perceive objects entering the field of view from the periphery (such as pedestrians, traffic at intersections, or bicycles). Although Figure 14B The middle image shows only one wide-angle camera, but any number of wide-angle cameras 1470 can be present on the vehicle 1400. Furthermore, a remote camera 1498 (e.g., a pair of long-view stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera 1498 can also be used for object detection and classification, as well as basic object tracking.

[0105] One or more stereo cameras 1468 may also be included in a front-mounted configuration. The stereo camera 1468 may include an integrated control unit comprising a scalable processing unit that can provide a multi-core microprocessor and programmable logic (FPGA) with an integrated CAN or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle environment, including distance estimates for all points in the image. Alternative stereo cameras 1468 may include a compact stereo vision sensor that may include two camera lenses (one on each side) and an image processing chip capable of measuring the distance from the vehicle to a target object and using the generated information (e.g., metadata) to activate autonomous emergency braking and lane departure warning functions. Other types of stereo cameras 1468 may be used in addition to those described herein, or alternatively.

[0106] Cameras with a field of view including the side portion of the vehicle 1400 (e.g., side-view cameras) can be used for surround view, providing information for creating and updating occupancy grids and generating side-impact collision warnings. For example, surround camera 1474 (e.g., ...) Figure 14B The four surround cameras 1474 shown can be mounted on the vehicle 1400. The surround cameras 1474 can include a wide-angle camera 1470, a fisheye camera, a 360-degree camera, and / or the like. Four examples are provided; the four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround cameras 1474 (e.g., left, right, and rear) and can utilize one or more other cameras (e.g., a forward-facing camera) as a fourth surround-view camera.

[0107] A camera with a field of view that includes the environment behind the vehicle 1400 (e.g., a rear-view camera) can be used for parking assistance, surround view, rear collision warning, and creating and updating occupancy grids. A wide variety of cameras can be used, including but not limited to those also suitable as front-facing cameras as described herein (e.g., long-range and / or mid-range camera 1498, stereo camera 1468, infrared camera 1472, etc.).

[0108] Figure 14C For use in accordance with some embodiments of this disclosure Figure 14A The example autonomous vehicle 1400 is illustrated in the block diagram of an example system architecture. It should be understood that this arrangement, and other arrangements described herein, are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, functional groupings, etc.) may be used in addition to or in place of those shown, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities, which may be implemented as discrete or distributed components or in combination with other components, and in any suitable combination and location. The various functions described herein as being performed by these entities can be implemented via hardware, firmware, and / or software. For example, the various functions can be implemented by a processor executing instructions stored in memory.

[0109] Figure 14C Each component, feature, and system in vehicle 1400 is illustrated as being connected via bus 1402. Bus 1402 may include a Controller Area Network (CAN) data interface (or, alternatively, referred to herein as the "CAN bus"). CAN may be a network within vehicle 1400 used to assist in the control of various features and functions of vehicle 1400, such as the actuation of brakes, acceleration, braking, steering, windshield wipers, etc. CAN bus can be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., CAN ID). CAN bus can be read to find steering wheel angle, ground speed, engine speed per minute (RPM), button positions, and / or other vehicle status indicators. CAN bus may be ASIL B compliant.

[0110] Although bus 1402 is described herein as a CAN bus, this is not intended to be limiting. For example, FlexRay and / or Ethernet may be used in addition to or alternatively to a CAN bus. Furthermore, although bus 1402 is represented by a single line, this is not intended to be limiting. For example, any number of buses 1402 may exist, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 1402 may be used to perform different functions and / or may be used for redundancy. For example, a first bus 1402 may be used for a collision avoidance function, and a second bus 1402 may be used for drive control. In any example, each bus 1402 may communicate with any component of vehicle 1400, and two or more buses 1402 may communicate with the same component. In some examples, each SoC 1404, each controller 1436, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 1400) and may be connected to a common bus such as the CAN bus.

[0111] Vehicle 1400 may include one or more controllers 1436, such as those described herein. Figure 14A The controllers described herein. Controller 1436 can be used for a wide variety of functions. Controller 1436 can be coupled to any other different components and systems of vehicle 1400 and can be used for the control of vehicle 1400, artificial intelligence of vehicle 1400, infotainment and / or the like for vehicle 1400.

[0112] Vehicle 1400 may include one or more System-on-Chip (SoC) 1404. SoC 1404 may include CPU 1406, GPU 1408, processor 1410, cache 1412, accelerator 1414, data storage 1416, and / or other components and features not shown. SoC 1404 can be used to control vehicle 1400 across a wide variety of platforms and systems. For example, one or more SoCs 1404 may be combined with an HD map 1422 in a system (e.g., the system of vehicle 1400), the HD map being accessible from one or more servers (e.g., via a network interface 1424). Figure 14D One or more servers (1478) receive map refresh and / or updates.

[0113] CPU 1406 may include CPU clusters or CPU complexes (or, alternatively, referred to herein as "CCPLEX"). CPU 1406 may include multiple cores and / or L2 cache. For example, in some embodiments, CPU 1406 may include eight cores in a coherent multiprocessor configuration. In some embodiments, CPU 1406 may include four dual-core clusters, each with a dedicated L2 cache (e.g., 2MB L2 cache). CPU 1406 (e.g., CCPLEX) may be configured to support simultaneous cluster operation, such that any combination of clusters of CPU 1406 can be active at any given time.

[0114] The CPU 1406 can implement power management capabilities including one or more of the following features: automatic clock gating of hardware blocks when idle to conserve dynamic power; clock gating of each core when the core is not actively executing instructions due to the execution of WFI / WFE instructions; independent power gating of each core; independent clock gating of each core cluster when all cores are clock-gated or power-gated; and / or independent power gating of each core cluster when all cores are power-gated. The CPU 1406 can further implement enhanced algorithms for managing power states, where allowed power states and desired wake-up times are specified, and the hardware / microcode determines the optimal power state to enter for the core, cluster, and CCPLEX. The processing core can support simplified power state entry sequences in software, with this work offloaded to the microcode.

[0115] GPU 1408 may include an integrated GPU (or, alternatively, referred to herein as an "iGPU"). GPU 1408 may be programmable and efficient for parallel workloads. In some examples, GPU 1408 may use an enhanced tensor instruction set. GPU 1408 may include one or more streaming microprocessors, wherein each streaming microprocessor may include an L1 cache (e.g., an L1 cache with at least 96KB of storage capacity), and two or more of these streaming microprocessors may share an L2 cache (e.g., an L2 cache with 512KB of storage capacity). In some embodiments, GPU 1408 may include at least eight streaming microprocessors. GPU 1408 may use a computation application programming interface (API). Furthermore, GPU 1408 may use one or more parallel computing platforms and / or programming models (e.g., NVIDIA's CUDA).

[0116] In automotive and embedded applications, the GPU 1408 can be power-optimized for optimal performance. For example, the GPU 1408 can be fabricated on FinFETs. However, this is not intended to be limiting, and the GPU 1408 can be fabricated using other semiconductor manufacturing processes. Each streaming microprocessor can combine several mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block can be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two mixed-precision NVIDIA Tensor cores for deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, dispatch units, and / or a 64KB register file. Furthermore, the streaming microprocessor can include independent parallel integer and floating-point data paths to leverage the mixture of computation and addressing computations for efficient workload execution. Streaming microprocessors may include independent thread scheduling capabilities to allow for finer-grained synchronization and cooperation between parallel threads. Streaming microprocessors may include combined L1 data caches and shared memory units to improve performance while simplifying programming.

[0117] The GPU 1408 may include, in some examples, a High Bandwidth Memory (HBM) and / or a 16GB HBM2 memory subsystem providing a peak memory bandwidth of approximately 900GB / s. In some examples, in addition to HBM memory or alternatively, Synchronous Graphics Random Access Memory (SGRAM), such as Generation 5 Graphics Double Data Rate Synchronous Random Access Memory (GDDR5), may be used.

[0118] The GPU 1408 may include unified memory technology, which includes access counters to allow memory pages to be migrated more precisely to the processors that access them most frequently, thereby improving the efficiency of shared memory ranges between processors. In some examples, Address Translation Service (ATS) support can be used to allow the GPU 1408 to directly access the CPU 1406 page tables. In such examples, when the GPU 1408 Memory Management Unit (MMU) experiences a miss, the address translation request can be transferred to the CPU 1406. In response, the CPU 1406 can look up the virtual-physical mapping for the address in its page tables and transfer the translation back to the GPU 1408. Thus, unified memory technology can allow a single unified virtual address space for the memory of both the CPU 1406 and the GPU 1408, simplifying GPU 1408 programming and porting applications to the GPU 1408.

[0119] In addition, the GPU 1408 may include access counters that track how frequently the GPU 1408 accesses the memory of other processors. Access counters can help ensure that memory pages are moved to the physical memory of the processor that accesses those pages most frequently.

[0120] SoC 1404 may include any number of caches 1412, including those described herein. For example, cache 1412 may include an L3 cache available to both CPU 1406 and GPU 1408 (e.g., it is connected to both CPU 1406 and GPU 1408). Cache 1412 may include a write-back cache, which can track the state of rows, for example, using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). Depending on the embodiment, the L3 cache may include 4 MB or more, but a smaller cache size may also be used.

[0121] SoC 1404 may include an arithmetic logic unit (ALU) that can be utilized in processing of any of the various tasks or operations performed on vehicle 1400, such as processing a DNN. Furthermore, SoC 1404 may include a floating-point unit (FPU) (or other mathematical coprocessor or digital coprocessor type) for performing mathematical operations within the system. For example, SoC 1404 may include one or more FPUs integrated as execution units within CPU 1406 and / or GPU 1408.

[0122] SoC 1404 may include one or more accelerators 1414 (e.g., hardware accelerators, software accelerators, or combinations thereof). For example, SoC 1404 may include a hardware accelerator cluster, which may include optimized hardware accelerators and / or large on-chip memory. This large on-chip memory (e.g., 4MB SRAM) can enable the hardware accelerator cluster to accelerate neural networks and other computations. The hardware accelerator cluster can be used to complement GPU 1408 and offload some tasks from GPU 1408 (e.g., freeing up more cycles of GPU 1408 to perform other tasks). As an example, accelerator 1414 can be used for targeted workloads (e.g., perceptrons, convolutional neural networks (CNNs), etc.) that are stable enough to be easily controlled for acceleration. When used herein, the term "CNN" can include all types of CNNs, including region-based or region convolutional neural networks (RCNNs) and fast RCNNs (e.g., for object detection).

[0123] Accelerator 1414 (e.g., a hardware accelerator cluster) may include a Deep Learning Accelerator (DLA). The DLA may include one or more Tensor Processing Units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may be an accelerator configured to perform image processing functions (e.g., for CNNs, RCNNs, etc.) and optimized for performing image processing functions. The DLA may be further optimized for a specific set of neural network types and floating-point operations and inference. The DLA is designed to provide higher performance per millimeter than a general-purpose GPU and significantly outperform CPUs. The TPU may perform several functions, including single-instance convolution functions, support for INT8, INT16, and FP16 data types for both features and weights, and post-processor functions.

[0124] DLA can execute neural networks, especially CNNs, quickly and efficiently on processed or unprocessed data for any function across a wide variety of applications, such as, but not limited to: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection and recognition using data from microphones; CNNs for face recognition and vehicle owner recognition using data from camera sensors; and / or CNNs for safety and / or safety-related events.

[0125] The DLA can perform any function of the GPU 1408, and by using inference accelerators, for example, designers can target either the DLA or the GPU 1408 for any function. For instance, a designer can focus the CNN processing and floating-point operations on the DLA and leave other functions to the GPU 1408 and / or other accelerators 1414.

[0126] Accelerator 1414 (e.g., a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may alternatively be referred to herein as a computer vision accelerator. The PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. The PVA can provide a balance between performance and flexibility. For example, each PVA may include, for example, but not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0127] RISC cores can interact with image sensors (such as the image sensor of any camera described herein), image signal processors, and / or the like. Each of these RISC cores may include any amount of memory. Depending on the embodiment, the RISC core may use any of several protocols. In some examples, the RISC core may execute a real-time operating system (RTOS). RISC cores may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0128] DMA enables PVA components to access system memory independently of the CPU 1406. DMA can support any number of features to provide optimizations to the PVA, including but not limited to support for multidimensional addressing and / or circular addressing. In some examples, DMA can support addressing in up to six or more dimensions, which can include block width, block height, block depth, horizontal block step, vertical block step, and / or depth step.

[0129] A vector processor can be a programmable processor designed to efficiently and flexibly execute programming for computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, one or more DMA engines (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may operate as the main processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as, for example, a Single Instruction Multiple Data (SIMD) or Very Long Instruction Word (VLIW) digital signal processor. The combination of SIMD and VLIW can enhance throughput and speed.

[0130] Each of the vector processors may include an instruction cache and may be coupled to dedicated memory. As a result, in some examples, each of the vector processors may be configured to execute independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to employ data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even different algorithms on a sequence of images or portions of an image. Among other things, any number of PVAs may be included in a cluster of hardware accelerators, and any number of vector processors may be included in each of these PVAs. Furthermore, the PVA may include additional error correction code (ECC) memory to enhance overall system security.

[0131] Accelerator 1414 (e.g., a hardware accelerator cluster) may include an on-chip computer vision network and SRAM to provide high-bandwidth, low-latency SRAM for accelerator 1414. In some examples, on-chip memory may include at least 4MB of SRAM consisting of, for example, but not limited to, eight field-configurable memory blocks, accessible by both PVA and DLA. Each pair of memory blocks may include an Advanced Peripheral Bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. PVA and DLA may access memory via a backbone that provides high-speed memory access to PVA and DLA. The backbone may include (e.g., using an APB) an on-chip computer vision network that interconnects PVA and DLA to memory.

[0132] On-chip computer vision networks can include interfaces that ensure both the PVA and DLA provide ready and valid signals before transmitting any control signals / addresses / data. Such interfaces can provide separate phases and channels for transmitting control signals / addresses / data, as well as burst communication for continuous data transmission. This type of interface can conform to ISO 26262 or IEC 61508 standards, but other standards and protocols can also be used.

[0133] In some examples, the SoC 1404 may include, for example, a real-time ray tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed August 10, 2018. This real-time ray tracing hardware accelerator can be used to quickly and efficiently determine the location and extent of objects (e.g., within a world model) to generate real-time visualization simulations for RADAR signal interpretation, sound propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison with LiDAR data for localization and / or other functional purposes, and / or other uses. In some embodiments, one or more Tree Traversal Units (TTUs) may be used to perform one or more ray tracing-related operations.

[0134] Accelerators 1414 (e.g., hardware accelerator clusters) have broad applications in autonomous driving. PVAs can be programmable vision accelerators used in critical processing stages of ADAS and autonomous vehicles. The capabilities of PVAs are a good match for algorithmic domains requiring predictable processing, low power, and low latency. In other words, PVAs perform well in semi-dense or dense rule computation, even on small datasets requiring predictable runtimes with low latency and low power. Therefore, in the context of platforms for autonomous vehicles, PVAs are designed to run classical computer vision algorithms because they are efficient in object detection and integer mathematical operations.

[0135] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. In some examples, semi-global matching-based algorithms may be used, but this is not intended to be limiting. Many applications for Level 3-5 autonomous driving require instantaneous motion estimation / stereo matching (e.g., from moving structures, pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions on input from two monocular cameras.

[0136] In some examples, PVA can be used to perform intensive optical flow, providing processed RADAR data from the raw RADAR data (e.g., using 4D Fast Fourier Transform). In other examples, PVA is used for time-of-flight depth processing, which, for example, involves processing raw time-of-flight data to provide processed time-of-flight data.

[0137] DLA can be used to run any type of network to enhance control and driving safety, including, for example, neural networks that output a confidence metric for each object detection. Such a confidence value can be interpreted as a probability or as providing a relative “weight” for each detection compared to other detections. This confidence value allows the system to make further decisions about which detections should be considered true positives rather than false positives. For example, the system can set a threshold for the confidence and only consider detections exceeding the threshold as true positives. In an Automatic Emergency Braking (AEB) system, false positives can cause the vehicle to automatically perform emergency braking, which is clearly undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. DLA can run neural networks to regress the confidence values. The neural network can take at least some subset of parameters as its input, such as bounding box dimensions, ground plane estimates obtained (e.g. from another subsystem), outputs from inertial measurement unit (IMU) sensors 1466 related to the orientation and distance of vehicle 1400, 3D position estimates of objects obtained from the neural network and / or other sensors (e.g., LiDAR sensor 1464 or RADAR sensor 1460), etc.

[0138] SoC 1404 may include one or more data stores 1416 (e.g., memory). Data stores 1416 may be on-chip memory of SoC 1404, which may store neural networks to be executed on the GPU and / or DLA. In some examples, for redundancy and security, data stores 1416 may be large enough to store multiple instances of the neural network. Data stores 1412 may include L2 or L3 cache 1412. References to data stores 1416 may include references to memory associated with PVA, DLA, and / or other accelerators 1414 as described herein.

[0139] SoC 1404 may include one or more processors 1410 (e.g., embedded processors). Processor 1410 may include a startup and power management processor, which may be a dedicated processor and subsystem for handling startup power and management functions, as well as safety implementation. The startup and power management processor may be part of the SoC 1404 startup sequence and may provide runtime power management services. The startup power and management processor may provide clock and voltage programming, auxiliary system low-power state transitions, SoC 1404 thermal and temperature sensor management, and / or SoC 1404 power state management. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to the temperature, and SoC 1404 may use the ring oscillator to detect the temperature of CPU 1406, GPU 1408, and / or accelerator 1414. If it is determined that the temperature exceeds a threshold, the startup and power management processor may enter a temperature fault routine and place SoC 1404 into a lower power state and / or place vehicle 1400 into a driver-safe parking mode (e.g., safely stop vehicle 1400).

[0140] The processor 1410 may also include a set of embedded processors that can be used as an audio processing engine. The audio processing engine can be an audio subsystem that allows for full hardware support for multi-channel audio via multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0141] The processor 1410 may also include an always-on-processor engine that can provide the necessary hardware features to support low-power sensor management and wake-up use cases. This always-on-processor engine may include a processor core, tightly coupled RAM, support for peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0142] The processor 1410 may also include a security cluster engine, which includes a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine may include two or more processor cores, tightly coupled RAM, support for peripheral devices (e.g., timers, interrupt controllers, etc.), and / or routing logic. In secure mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic that detects any differences between their operations.

[0143] The processor 1410 may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0144] The processor 1410 may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0145] Processor 1410 may include a video image compositer, which may be (e.g., implemented on a microprocessor) a processing block, implementing video post-processing functions required by the video playback application to generate the final image for the player window. The video image compositer may perform lens distortion correction on the wide-angle camera 1470, the surround camera 1474, and / or the in-cabin monitoring camera sensor. The in-cabin monitoring camera sensor is preferably monitored by a neural network running on another instance of an advanced SoC, configured to recognize in-cabin events and respond accordingly. The in-cabin system may perform lip reading to activate mobile phone services and make calls, dictate emails, change vehicle destinations, activate or change the vehicle's infotainment system and settings, or provide voice-activated web browsing. Some functions are only available to the driver when the vehicle is operating in autonomous mode and are disabled in other situations.

[0146] Video image compositers can include enhanced temporal denoising for both spatial and temporal noise reduction. For example, in the case of motion in the video, denoising appropriately weights spatial information, reducing the weight of information provided by neighboring frames. In cases where the image or part of the image does not contain motion, the temporal denoising performed by the video image compositer can use information from previous images to reduce noise in the current image.

[0147] The video image compositer can also be configured to perform stereo correction on input stereo camera frames. When the operating system desktop is in use and the GPU 1408 does not need to continuously render new surfaces, the video image compositer can be further used for user interface components. Even when the GPU 1408 is powered on and active, performing 3D rendering, the video image compositer can be used to offload the GPU 1408 to improve performance and responsiveness.

[0148] The SoC 1404 may also include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and pixel-related input functions for receiving video and input from a camera. The SoC 1404 may also include an input / output controller that can be software-controlled and can be used to receive I / O signals not assigned to a specific role.

[0149] SoC 1404 may also include a wide range of peripheral interfaces to enable communication with peripherals, audio codecs, power management and / or other devices. SoC 1404 can be used to process data from cameras and sensors (e.g., LIDAR sensor 1464, RADAR sensor 1460, etc., which can be connected via Gigabit Multimedia Serial Link and Ethernet), data from bus 1402 (e.g., vehicle 1400 speed, steering wheel position, etc.), and data from GNSS sensor 1458 (connected via Ethernet or CAN bus). SoC 1404 may also include a dedicated high-performance, high-capacity memory controller, which may include its own DMA engine and can be used to free up CPU 1406 from routine data management tasks.

[0150] The SoC 1404 can be an end-to-end platform with a flexible architecture spanning Automation Levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS technologies for diversity and redundancy, along with deep learning tools to deliver a flexible and reliable driving software stack. The SoC 1404 can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, when combined with the CPU 1406, GPU 1408, and data storage 1416, the accelerator 1414 can provide a fast and efficient platform for Level 3-5 autonomous vehicles.

[0151] Therefore, this technology offers capabilities and functionalities that cannot be achieved through conventional systems. For example, computer vision algorithms can be executed on CPUs, which can be configured using high-level programming languages ​​such as C to execute a wide variety of processing algorithms across a diverse range of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs cannot execute complex object detection algorithms in real time, which is a requirement for automotive ADAS applications and practical Level 3-5 autonomous vehicles.

[0152] In contrast to conventional systems, the techniques described in this paper, by providing CPU complexes, GPU complexes, and hardware accelerator clusters, allow multiple neural networks to be executed simultaneously and / or sequentially, and the results combined to achieve Level 3–5 autonomous driving capabilities. For example, a CNN executed on a DLA or dGPU (e.g., GPU 1420) could include text and word recognition, allowing a supercomputer to read and understand traffic signs, including those for which neural networks have not yet been specifically trained. The DLA could also include a neural network capable of recognizing, interpreting, and providing semantic understanding of the signs, and passing that semantic understanding to a path planning module running on the CPU complex.

[0153] As another example, multiple neural networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate icy conditions," along with a light, can be interpreted independently or jointly by several neural networks. The sign itself can be recognized as a traffic sign by a deployed first neural network (e.g., a trained neural network), and the text "Flashing lights indicate icy conditions" can be interpreted by a deployed second neural network, which informs the vehicle's path planning software (preferably executing on a CPU complex) that icy conditions exist when the flashing lights are detected. The flashing lights can be identified by a deployed third neural network operating across multiple frames, which informs the vehicle's path planning software of the presence (or absence) of the flashing lights. All three neural networks can operate simultaneously, for example, within a DLA and / or on a GPU 1408.

[0154] In some examples, the CNN used for facial recognition and owner identification can use data from camera sensors to identify the presence of an authorized driver and / or owner of vehicle 1400. A processing engine always on the sensors can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and in safe mode, to disable the vehicle when the owner leaves. In this way, SoC 1404 provides security against theft and / or carjacking.

[0155] In another example, the CNN used for emergency vehicle detection and identification can use data from microphone 1496 to detect and identify emergency vehicle siren. In contrast to conventional systems that use a general classifier to detect siren and manually extract features, SoC 1404 uses a CNN to classify environmental and urban sounds as well as visual data. In a preferred embodiment, the CNN running on the DLA is trained to identify the relative shut-off rate of emergency vehicles (e.g., by using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the localized area in which the vehicle operates, as identified by GNSS sensor 1458. Thus, for example, when operating in Europe, the CNN will seek to detect European siren, and when operating in the United States, the CNN will seek to identify siren only in North America. Once an emergency vehicle is detected, with the assistance of ultrasonic sensor 1462, the control program can be used to execute emergency vehicle safety routines, causing the vehicle to slow down, pull over to the side of the road, stop, and / or idle until the emergency vehicle passes.

[0156] The vehicle may include a CPU 1418 (e.g., a discrete CPU or dCPU) that can be coupled to the SoC 1404 via a high-speed interconnect (e.g., PCIe). The CPU 1418 may include, for example, an x86 processor. The CPU 1418 can be used to perform any of a wide variety of functions, including, for example, arbitrating the results of potential inconsistencies between ADAS sensors and the SoC 1404, and / or monitoring the status and health of the controller 1436 and / or the infotainment SoC 1430.

[0157] Vehicle 1400 may include a GPU 1420 (e.g., a discrete GPU or dGPU) that can be coupled to SoC 1404 via a high-speed interconnect (e.g., NVIDIA's NVLINK). GPU 1420 may provide additional artificial intelligence capabilities, for example by executing redundant and / or different neural networks, and can be used to train and / or update neural networks based at least in part on inputs (e.g., sensor data) from sensors of vehicle 1400.

[0158] Vehicle 1400 may also include a network interface 1424, which may include one or more wireless antennas 1426 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas, Bluetooth antennas, etc.). Network interface 1424 can be used to enable wireless connectivity via the Internet to the cloud (e.g., with server 1478 and / or other network devices), with other vehicles, and / or with computing devices (e.g., passenger client devices). For communication with other vehicles, a direct link can be established between the two vehicles, and / or an indirect link can be established (e.g., across networks and via the Internet). A direct link can be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide vehicle 1400 with information about vehicles approaching vehicle 1400 (e.g., vehicles in front, to the side, and / or behind vehicle 1400). This functionality may be part of vehicle 1400's cooperative adaptive cruise control function.

[0159] Network interface 1424 may include a SoC that provides modulation and demodulation functions and enables controller 1436 to communicate via a wireless network. Network interface 1424 may include an RF front-end for up-conversion from baseband to RF and down-conversion from RF to baseband. Frequency conversion can be performed using known processes and / or using a superheterodyne process. In some examples, the RF front-end functionality may be provided by a separate chip. The network interface may include wireless functions for communication via LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0160] Vehicle 1400 may also include data storage 1428, which may include off-chip (e.g., off-chip SoC 1404) storage devices. Data storage 1428 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash memory, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0161] Vehicle 1400 may also include a GNSS sensor 1458. The GNSS sensor 1458 (e.g., GPS, assisted GPS sensor, differential GPS (DGPS) sensor, etc.) is used for auxiliary mapping, sensing, occupancy grid generation, and / or path planning functions. Any number of GNSS sensors 1458 can be used, including, for example, but not limited to, GPS using a USB connector with an Ethernet-to-serial (RS-232) bridge.

[0162] Vehicle 1400 may also include a RADAR sensor 1460. The RADAR sensor 1460 can be used by vehicle 1400 for remote vehicle detection even in dark and / or inclement weather conditions. The RADAR functional safety level may be ASIL B. The RADAR sensor 1460 can use CAN and / or bus 1402 (e.g., to transmit data generated by the RADAR sensor 1460) for control and access to object tracking data, and in some examples, Ethernet access for accessing raw data. A wide variety of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor 1460 can be adapted for front, rear, and side RADAR use. In some examples, a pulse Doppler RADAR sensor is used.

[0163] The RADAR sensor 1460 can include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, short-range side coverage, etc. In some examples, the long-range RADAR can be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view (e.g., within 250m) achieved through two or more independent scans. The RADAR sensor 1460 can help distinguish between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. The long-range RADAR sensor can include a single-site multi-mode RADAR with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In an example with six antennas, the four central antennas can create a focused beam pattern designed to record the vehicle 1400's surroundings at higher rates with minimal traffic interference from adjacent lanes. The other two antennas can extend the field of view, enabling rapid detection of vehicles entering or leaving the vehicle 1400's lane.

[0164] As an example, a mid-range RADAR system can include a range of up to 1460m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 1450 degrees (rear). Short-range RADAR systems can include, but are not limited to, RADAR sensors designed to be mounted at both ends of the rear bumper. When mounted at both ends of the rear bumper, such a RADAR sensor system can create two beams that continuously monitor blind spots behind and beside the vehicle.

[0165] Short-range RADAR systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0166] Vehicle 1400 may also include ultrasonic sensors 1462. Ultrasonic sensors 1462, which may be positioned at the front, rear, and / or sides of vehicle 1400, can be used for parking assistance and / or creating and updating occupancy grids. A wide variety of ultrasonic sensors 1462 can be used, and different ultrasonic sensors 1462 can be used for different detection ranges (e.g., 2.5m, 4m). Ultrasonic sensors 1462 can operate at functional safety level ASIL B.

[0167] Vehicle 1400 may include a LIDAR sensor 1464. The LIDAR sensor 1464 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LIDAR sensor 1464 may be of functional safety level ASIL B. In some examples, vehicle 1400 may include multiple LIDAR sensors 1464 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0168] In some examples, the LiDAR sensor 1464 may be able to provide a list of objects and their distances within a 360-degree field of view. Commercially available LiDAR sensors 1464 may have an advertising range of, for example, approximately 1400m, with an accuracy of 2cm-3cm, and support for 1400Mbps Ethernet connectivity. In some examples, one or more non-protruding LiDAR sensors 1464 may be used. In such examples, the LiDAR sensor 1464 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of a vehicle 1400. In such examples, the LiDAR sensor 1464 may provide a horizontal field of view of up to 120 degrees and a vertical field of view of 35 degrees, even for low-reflectivity objects, with a range of 200m. Front-mounted LiDAR sensors 1464 may be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0169] In some examples, LiDAR technologies such as 3D flash LiDAR can also be used. 3D flash LiDAR uses a flash of laser light as the emission source to illuminate the vehicle's surroundings up to approximately 200 meters. A flash LiDAR unit includes a receiver that records the laser pulse propagation time and reflected light on each pixel, which in turn corresponds to the range from the vehicle to the object. Flash LiDAR allows for the generation of highly accurate and distortion-free images of the surrounding environment using each laser flash. In some examples, four flash LiDAR sensors can be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D staring array LiDAR cameras (e.g., non-scanning LiDAR devices) without moving parts other than a fan. Flash LiDAR devices can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-registered intensity data. By using a flash LiDAR, and because a flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 1464 is less susceptible to motion blur, vibration, and / or shock.

[0170] The vehicle may also include an IMU sensor 1466. In some examples, the IMU sensor 1466 may be located at the center of the rear axle of the vehicle 1400. The IMU sensor 1466 may include, for example, but not limited to, an accelerometer, a magnetometer, a gyroscope, a magnetic compass, and / or other sensor types. In some examples, such as in a six-axis application, the IMU sensor 1466 may include an accelerometer and a gyroscope, while in a nine-axis application, the IMU sensor 1466 may include an accelerometer, a gyroscope, and a magnetometer.

[0171] In some embodiments, the IMU sensor 1466 can be implemented as a miniature, high-performance GPS-assisted inertial navigation system (GPS / INS) that combines a microelectromechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filter algorithm to provide estimates of position, velocity, and attitude. Thus, in some examples, the IMU sensor 1466 can enable the vehicle 1400 to estimate heading without input from a magnetic sensor by directly observing and correlating velocity changes from GPS to the IMU sensor 1466. In some examples, the IMU sensor 1466 and the GNSS sensor 1458 can be combined into a single integrated unit.

[0172] The vehicle may include a microphone 1496 placed in and / or around the vehicle 1400. Among other things, the microphone 1496 may be used for emergency vehicle detection and identification.

[0173] The vehicle may also include any number of camera types, including stereo camera 1468, wide-angle camera 1470, infrared camera 1472, surround camera 1474, long-range and / or mid-range camera 1498, and / or other camera types. These cameras can be used to capture image data around the entire perimeter of the vehicle 1400. The types of cameras used depend on the embodiment and the requirements of the vehicle 1400, and any combination of camera types can be used to provide the necessary coverage around the vehicle 1400. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. As an example and without limitation, these cameras may support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet. Each of the cameras is described herein with respect to... Figure 14A and Figure 14B It was described in more detail.

[0174] Vehicle 1400 may also include vibration sensor 1442. Vibration sensor 1442 can measure vibrations of vehicle components such as axles. For example, changes in vibration can indicate changes in the road surface. In another example, when two or more vibration sensors 1442 are used, differences between vibrations can be used to determine friction or slippage on the road surface (e.g., when there is a vibration difference between a power drive shaft and a free-rotating shaft).

[0175] Vehicle 1400 may include ADAS system 1438. In some examples, ADAS system 1438 may include SoC. ADAS system 1438 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward collision warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keeping assist (LKA), blind spot warning (BSW), rear cross traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0176] The ACC system can use RADAR sensor 1460, LIDAR sensor 1464, and / or a camera. The ACC system can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to vehicles immediately in front of vehicle 1400 and automatically adjusts the vehicle speed to maintain a safe distance. Lateral ACC performs distance holding and, if necessary, advises vehicle 1400 to change lanes. Lateral ACC is associated with other ADAS applications such as LCA and CWS.

[0177] CACC uses information from other vehicles, which can be received indirectly from other vehicles via a wireless link or a network connection (e.g., via the Internet) through network interface 1424 and / or wireless antenna 1426. Direct links can be provided by vehicle-to-vehicle (V2V) communication links, while indirect links can be infrastructure-to-vehicle (I2V) communication links. Typically, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles immediately in front of vehicle 1400 and in the same lane), while the I2V communication concept provides information about traffic further ahead. A CACC system can include either or both of these I2V and V2V information sources. Given information about vehicles ahead of vehicle 1400, CACC can be more reliable, and it has the potential to improve traffic flow and reduce road congestion.

[0178] The Forward-Looking Warning (FCW) system is designed to alert the driver to hazards, enabling the driver to take corrective action. The FCW system uses a front-facing camera and / or RADAR sensor 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components. The FCW system can provide warnings in the form of, for example, audible, visual, haptic, and / or rapid braking pulses.

[0179] The AEB system detects an impending forward collision with another vehicle or other object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system can use a front-facing camera and / or RADAR sensor 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first alerts the driver to take corrective action to avoid a collision, and if the driver does not take corrective action, the AEB system can automatically apply the brakes to attempt to prevent or at least mitigate the effects of the predicted collision. The AEB system may include technologies such as dynamic brake support and / or collision proximity braking.

[0180] The Lane Departure Warning (LDW) system provides visual, auditory, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver when the vehicle crosses lane markings at 1400 degrees. When the driver indicates intentional lane departure, the LDW system is deactivated by activating a turn signal. The LDW system can utilize a front-facing camera coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0181] The LKA system is a variation of the LDW system. If vehicle 1400 begins to leave the lane, the LKA system provides steering input or braking to correct vehicle 1400.

[0182] The BSW system detects and warns the driver of vehicles in the vehicle's blind spot. The BSW system can provide visual, auditory, and / or tactile alerts to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses turn signals. The BSW system can utilize a rear-facing camera and / or RADAR sensor 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0183] The RCTW system can provide visual, auditory, and / or tactile notifications when an object is detected outside the range of the rear camera while the vehicle is reversing. Some RCTW systems include AEB to ensure the application of the vehicle's brakes to avoid a collision. The RCTW system may use one or more rear-mounted RADAR sensors 1460 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback such as a display, speaker, and / or vibrating components.

[0184] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for the driver, but typically not catastrophic, as ADAS systems alert the driver and allow them to determine whether a safe condition truly exists and take appropriate action. However, in the autonomous vehicle 1400, in the event of conflicting results, the vehicle 1400 itself must decide whether to heed the results from the main computer or auxiliary computer (e.g., the first controller 1436 or the second controller 1436). For example, in some embodiments, the ADAS system 1438 may be a backup and / or auxiliary computer for providing perception information to a backup computer rationality module. The backup computer rationality monitor may run redundant and diverse software on hardware components to detect faults in perception and dynamic driving tasks. The output from the ADAS system 1438 may be provided to a supervisory MCU. If the outputs from the main computer and the auxiliary computer conflict, the supervisory MCU must determine how to reconcile the conflict to ensure safe operation.

[0185] In some examples, the master computer can be configured to provide a confidence score to the supervisory MCU, indicating the master computer's confidence level in the selected result. If the confidence score exceeds a threshold, the supervisory MCU can follow the master computer's direction regardless of whether the auxiliary computer provides conflicting or inconsistent results. If the confidence score does not meet the threshold and the master and auxiliary computers indicate different results (e.g., conflict), the supervisory MCU can arbitrate between these computers to determine the appropriate result.

[0186] The supervisory MCU can be configured to run a neural network trained and configured to determine the conditions under which the auxiliary computer provides a false alarm, based at least in part on outputs from both the host and auxiliary computers. Thus, the neural network in the supervisory MCU can learn when the output of the auxiliary computer can be trusted and when it cannot. For example, when the auxiliary computer is a RADAR-based FCW system, the neural network in the supervisory MCU can learn when the FCW system is identifying a metallic object that is not actually dangerous, such as a drain grid or manhole cover that triggers an alarm. Similarly, when the auxiliary computer is a camera-based LDW system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest strategy. In embodiments that include a neural network running on the supervisory MCU, the supervisory MCU may include at least one of a DLA or GPU suitable for running the neural network using associated memory. In a preferred embodiment, the supervisory MCU may include components of and / or be included as components of the SoC 1404.

[0187] In other examples, ADAS system 1438 may include an auxiliary computer that performs ADAS functions using conventional computer vision rules. This allows the auxiliary computer to use classic computer vision rules (if-then), and the presence of neural networks in the supervising MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identity make the entire system more fault-tolerant, especially for failures caused by software (or software-hardware interface) functionality. For instance, if a software vulnerability or bug exists in the software running on the host computer and non-identical software code running on the auxiliary computer provides the same overall result, the supervising MCU can be more confident that the overall result is correct and that the vulnerability in the software or hardware on the host computer does not cause a substantial error.

[0188] In some examples, the output of ADAS system 1438 can be fed to the perception block and / or the dynamic driving task block of the main computer. For example, if ADAS system 1438 issues a forward collision warning because an object is immediately in front, the perception block can use this information when identifying the object. In other examples, the assistance computer can have its own neural network, which is trained and thus reduces the risk of false positives as described herein.

[0189] Vehicle 1400 may also include an infotainment SoC 1430 (e.g., an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system may not be an SoC and may include two or more discrete components. The infotainment SoC 1430 may include a combination of hardware and software that can be used to provide vehicle 1400 with audio (e.g., music, personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., TV, movies, streaming media, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, WiFi, etc.) and / or information services (e.g., navigation system, rear parking assistance, radio data system, vehicle-related information such as fuel level, total coverage distance, brake fuel level, fuel level, door opening / closing, air filter information, etc.). For example, the infotainment SoC 1430 may include a radio, disc player, navigation system, video player, USB and Bluetooth connectivity, in-vehicle computer, in-vehicle entertainment, WiFi, steering wheel audio controls, hands-free voice controls, head-up display (HUD), HMI display 1434, telematics device, control panel (e.g., for controlling and / or interacting with various components, features, and / or systems) and / or other components. The infotainment SoC 1430 may further be used to provide information (e.g., visual and / or auditory) to the vehicle's users, such as information from the ADAS system 1438, autonomous driving information such as planned vehicle maneuvers, trajectories, surrounding environment information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0190] The infotainment SoC 1430 may include GPU functionality. The infotainment SoC 1430 can communicate with other devices, systems, and / or components of the vehicle 1400 via a bus 1402 (e.g., CAN bus, Ethernet, etc.). In some examples, the infotainment SoC 1430 may be coupled to a supervisory MCU, allowing the GPU of the infotainment system to perform some autonomous driving functions in the event of a failure of the main controller 1436 (e.g., the primary and / or backup computer of the vehicle 1400). In such an example, the infotainment SoC 1430 may place the vehicle 1400 into a driver-safe parking mode as described herein.

[0191] Vehicle 1400 may also include instrument cluster 1432 (e.g., digital instrument panel, electronic instrument cluster, digital instrument panel, etc.). Instrument cluster 1432 may include a controller and / or supercomputer (e.g., a discrete controller or supercomputer). Instrument cluster 1432 may include a set of instruments such as speedometer, fuel level, oil pressure, tachometer, odometer, turn indicator, shift position indicator, seatbelt warning light, parking brake warning light, engine malfunction indicator, airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between infotainment SoC 1430 and instrument cluster 1432. In other words, instrument cluster 1432 may be included as part of infotainment SoC 1430, or vice versa.

[0192] Figure 14D For cloud-based servers and according to some embodiments of this disclosure Figure 14A The following is a system diagram illustrating communication between example autonomous vehicles 1400. System 1476 may include server 1478, network 1490, and vehicles including vehicle 1400. Server 1478 may include multiple GPUs 1484(A)-1484(H) (collectively referred to herein as GPU 1484), PCIe switches 1482(A)-1482(H) (collectively referred to herein as PCIe switch 1482), and / or CPUs 1480(A)-1480(B) (collectively referred to herein as CPU 1480). GPU 1484, CPU 1480, and PCIe switches may be interconnected with high-speed interconnects and / or PCIe connections 1486, such as, but not limited to, NVLink interface 1488 developed by NVIDIA. In some examples, GPU 1484 is connected via NVLink and / or NVSwitch SoC, and GPU 1484 and PCIe switch 1482 are connected via PCIe interconnect. Although the diagram illustrates eight GPUs 1484, two CPUs 1480, and two PCIe switches, it is not intended to be limiting. Depending on the embodiment, each of the servers 1478 may include any number of GPUs 1484, CPUs 1480, and / or PCIe switches. For example, each of the servers 1478 may include eight, sixteen, thirty-two, and / or more GPUs 1484.

[0193] Server 1478 can receive image data from vehicles via network 1490, representing images of unexpected or altered road conditions such as recently commenced roadworks. Server 1478 can also transmit neural network 1492, updated neural network 1492, and / or map information 1494, including information about traffic and road conditions, to vehicles via network 1490. Updates to map information 1494 may include updates to HD map 1422, such as information about construction sites, potholes, bends, floods, or other obstacles. In some examples, neural network 1492, updated neural network 1492, and / or map information 1494 may have been generated from new training and / or data received from any number of vehicles in the environment, and / or based on experience gained from training performed at a data center (e.g., using server 1478 and / or other servers).

[0194] Server 1478 can be used to train machine learning models (e.g., neural networks) based on training data. Training data can be generated by the vehicle and / or generated in a simulation (e.g., using a game engine). In some examples, the training data is labeled (e.g., where the neural network benefits from supervised learning) and / or undergoes other preprocessing, while in other examples, the training data is not labeled and / or preprocessed (e.g., where the neural network does not require supervised learning). Training can be performed according to any one or more classes of machine learning techniques, including but not limited to: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, joint learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including alternative dictionary learning), rule-based machine learning, anomaly detection, and any variations or combinations thereof. Once the machine learning model is trained, it can be used by the vehicle (e.g., transmitted to the vehicle via network 1490), and / or the machine learning model can be used by server 1478 to remotely monitor the vehicle.

[0195] In some examples, server 1478 can receive data from vehicles and apply that data to state-of-the-art real-time neural networks for real-time intelligent inference. Server 1478 may include a deep learning supercomputer powered by GPU 1484 and / or a dedicated AI computer, such as the DGX and DGX Station machines developed by NVIDIA. However, in some examples, server 1478 may include a deep learning infrastructure in a data center that uses only CPU power.

[0196] The deep learning infrastructure of server 1478 may be capable of rapid real-time inference and can be used to assess and verify the health status of the processor, software, and / or associated hardware in vehicle 1400. For example, the deep learning infrastructure may receive periodic updates from vehicle 1400, such as image sequences and / or objects located in those image sequences by vehicle 1400 (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure may run its own neural network to identify objects and compare them with objects identified by vehicle 1400. If the results do not match and the infrastructure concludes that the AI ​​in vehicle 1400 has malfunctioned, then server 1478 may transmit a signal to vehicle 1400 instructing the vehicle's fail-safe computer to take control, notify passengers, and complete a safe stopping operation.

[0197] For inference, server 1478 may include GPU 1484 and one or more programmable inference accelerators (such as NVIDIA's TensorRT). The combination of a GPU-powered server and inference acceleration enables real-time response. In other examples, such as where performance is less critical, CPU, FPGA, and other processor-powered servers can be used for inference.

[0198] Example computing device

[0199] Figure 15 This is a block diagram of an example computing device 1500 suitable for implementing some embodiments of the present disclosure. The computing device 1500 may include an interconnect system 1502 directly or indirectly coupled to the following devices: a memory 1504, one or more central processing units (CPUs) 1506, one or more graphics processing units (GPUs) 1508, a communication interface 1510, input / output (I / O) ports 1512, input / output components 1514, a power supply 1516, one or more presentation components 1518 (e.g., one or more displays), and one or more logic units 1520. In at least one embodiment, the computing device 1500 may include one or more virtual machines (VMs), and / or any component thereof may include virtual components (e.g., virtual hardware components). For a non-limiting example, one or more GPUs 1508 may include one or more vGPUs, one or more CPUs 1506 may include one or more vCPUs, and / or one or more logic units 1520 may include one or more virtual logic units. Thus, (one or more) computing devices 1500 may include discrete components (e.g., a full GPU dedicated to computing device 1500), virtual components (e.g., a portion of the GPU dedicated to computing device 1500), or a combination thereof.

[0200] although Figure 15 The various blocks are shown as connected via interconnect system 1502 using lines, but this is not intended to be limiting and is merely for clarity. For example, in some embodiments, presentation component 1518 (such as a display device) may be considered I / O component 1514 (e.g., if the display is a touchscreen). As another example, CPU 1506 and / or GPU 1508 may include memory (e.g., memory 1504 may represent a storage device other than the memory of GPU 1508, CPU 1506, and / or other components). In other words, Figure 15 The computing devices described are for illustrative purposes only. No distinction is made between such categories as “workstation,” “server,” “laptop computer,” “desktop computer,” “tablet computer,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types, as all are considered within the scope of… Figure 15 Within the scope of computing devices.

[0201] Interconnect system 1502 may represent one or more links or buses, such as address buses, data buses, control buses, or combinations thereof. Interconnect system 1502 may include one or more bus or link types, such as Industry Standard Architecture (ISA) bus, Extended Industry Standard Architecture (EISA) bus, Video Electronics Standards Association (VESA) bus, Peripheral Component Interconnect (PCI) bus, Fast Peripheral Component Interconnect (PCIe) bus, and / or another type of bus or link. In some embodiments, there is a direct connection between components. As an example, CPU 1506 may be directly connected to memory 1504. Further, CPU 1506 may be directly connected to GPU 1508. In cases where there is a direct or point-to-point connection between components, interconnect system 1502 may include a PCIe link to perform the connection. In these examples, a PCI bus is not required to be included in computing device 1500.

[0202] The memory 1504 may include any computer-readable medium from a variety of computer-readable media. The computer-readable medium may be any available medium accessible by the computing device 1500. The computer-readable medium may include volatile and non-volatile media, as well as removable and non-removable media. By way of example and not limitation, the computer-readable medium may include computer storage media and communication media.

[0203] Computer storage media may include volatile and non-volatile media and / or removable and non-removable media implemented using any method or technology for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 1504 may store computer-readable instructions (e.g., representing (one or more) programs and / or (one or more) program elements, such as an operating system). Computer storage media may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and is accessible by computing device 1500. As used herein, computer storage media does not include the signal itself.

[0204] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transmission mechanisms, and includes any information transmission medium. The term "modulated data signal" can refer to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example and not limitation, computer storage media can include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of the above should also be included within the scope of computer-readable media.

[0205] CPU 1506 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. Each CPU 1506 may contain one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling numerous software threads simultaneously. CPU 1506 may contain any type of processor and may contain different types of processors depending on the type of computing device 1500 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers). For example, depending on the type of computing device 1500, the processor may be an advanced RISC machine (ARM) processor implemented using Reduced Instruction Set Computing (RISC) or an x86 processor implemented using Complex Instruction Set Computing (CISC). In addition to one or more microprocessors or supplementary coprocessors (such as math coprocessors), computing device 1500 may also include one or more CPUs 1506.

[0206] In addition to or in lieu of one or more CPUs 1506, one or more GPUs 1508 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. One or more GPUs 1508 may be integrated GPUs (e.g., having one or more CPUs 1506) and / or one or more GPUs 1508 may be discrete GPUs. In embodiments, one or more GPUs 1508 may be coprocessors of one or more CPUs 1506. GPUs 1508 may be used by computing device 1500 to render graphics (e.g., 3D graphics) or perform general-purpose computing. For example, GPUs 1508 may be used for general-purpose computing on a GPU (GPGPU). GPUs 1508 may contain hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. GPU 1508 can generate pixel data for an output image in response to rendering commands (e.g., rendering commands received from CPU 1506 via a host interface). GPU 1508 may include graphics memory (e.g., display memory) for storing pixel data or any other suitable data (e.g., GPGPU data). Display memory may be included as part of memory 1504. GPU 1508 may include two or more GPUs operating in parallel (e.g., via links). The links may directly connect the GPUs (e.g., using NVLINK) or may connect the GPUs via a switch (e.g., using NVSwitch). When combined, each GPU 1508 may generate pixel data or GPGPU data for a different portion of the output or for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU may include its own memory or may share memory with other GPUs.

[0207] In addition to or in lieu of CPU 1506 and / or GPU 1508, logic unit 1520 may be configured to execute at least some of computer-readable instructions to control one or more components of computing device 1500 to perform one or more of the methods and / or processes described herein. In embodiments, one or more CPUs 1506, one or more GPUs 1508, and / or one or more logic units 1520 may perform any combination of methods, processes, and / or portions thereof discretely or jointly. One or more logic units 1520 may be a portion of one or more CPUs 1506 and / or GPUs 1508 and / or integrated into one or more CPUs 1506 and / or GPUs 1508, and / or one or more logic units 1520 may be discrete components or otherwise external to CPUs 1506 and / or GPUs 1508. In an embodiment, one or more of the logic units 1520 may be coprocessors of one or more of the CPU 1506 and / or one or more of the GPU 1508.

[0208] Examples of logic unit 1520 include one or more processing cores and / or components thereof, such as tensor core (TC), tensor processing unit (TPU), pixel vision core (PVC), vision processing unit (VPU), graphics processing cluster (GPC), texture processing cluster (TPC), streaming multiprocessor (SM), tree lateral unit (TTU), artificial intelligence accelerator (AIA), deep learning accelerator (DLA), arithmetic logic unit (ALU), application-specific integrated circuit (ASIC), floating-point unit (FPU), input / output (I / O) element, peripheral component interconnect (PCI) or fast peripheral component interconnect (PCIe) element, etc.

[0209] The communication interface 1510 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 1500 to communicate with other computing devices via an electronic communication network (including wired and / or wireless communication). The communication interface 1510 may include components and functions that enable communication over any of a plurality of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communication over Ethernet or wirelessband), low-power wide area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.

[0210] I / O port 1512 enables computing device 1500 to be logically coupled to other devices including I / O component 1514, (one or more) presentation component 1518, and / or other components, some of which may be built into (e.g., integrated into) computing device 1500. Illustrative I / O component 1514 includes microphones, mice, keyboards, joysticks, game pads, game controllers, satellite dish antennas, scanners, printers, wireless devices, etc. I / O component 1514 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological input generated by the user. In some cases, input may be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, pen recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures, head and eye tracking, and touch recognition (as described in more detail below) associated with the display of computing device 1500. Computing device 1500 may include depth cameras for gesture detection and recognition, such as stereo camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof. Additionally, the computing device 1500 may include an accelerometer or gyroscope (e.g., as part of an inertial measurement unit (IMU)) that enables motion detection. In some examples, the computing device 1500 may use the output of the accelerometer or gyroscope to render immersive augmented reality or virtual reality.

[0211] Power supply 1516 may include a hardwired power supply, a battery power supply, or a combination thereof. Power supply 1516 may provide power to computing device 1500 to enable the components of computing device 1500 to operate.

[0212] The presentation component 1518 may include a display (e.g., a monitor, touchscreen, television screen, head-up display (HUD), other display types, or combinations thereof), speakers, and / or other presentation components. The presentation component 1518 may receive data from other components (e.g., GPU 1508, CPU 1506, etc.) and output said data (e.g., as images, videos, sounds, etc.).

[0213] Example Data Center

[0214] Figure 16 An example data center 1600 that may be used in at least one embodiment of this disclosure is shown. The data center 1600 may include a data center infrastructure layer 1610, a framework layer 1620, a software layer 1630, and / or an application layer 1640.

[0215] like Figure 16As shown, the data center infrastructure layer 1610 may include a resource coordinator 1612, grouped computing resources 1614, and node computing resources (“nodes CRs”) 1616(1)-1616(N), where “N” represents any complete positive integer. In at least one embodiment, nodes CRs 1616(1)-1616(N) may include, but are not limited to, any number of central processing units (“CPUs”) or other processors (including accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (“NW I / O”) devices, network switches, virtual machines (“VMs”), power modules and / or cooling modules, etc. In some embodiments, one or more node CRs from nodes CRs 1616(1)-1616(N) may correspond to servers having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs 1616(1)-16161(N) may include one or more virtual components, such as vGPU, vCPU, etc., and / or one or more nodes CRs 1616(1)-1616(N) may correspond to virtual machines (VMs).

[0216] In at least one embodiment, the grouped computing resources 1614 may include individual groups of node CRs 1616 housed within one or more racks (not shown), or a plurality of racks housed within a data center in different geographical locations (also not shown). Individual groups of node CRs 1616 within the grouped computing resources 1614 may include grouped computing, networking, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, a plurality of node CRs 1616, including CPUs, GPUs, and / or other processors, may be grouped within one or more racks to provide computing resources to support one or more workloads. One or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0217] Resource coordinator 1622 may be configured or otherwise control one or more nodes CRs 1616(1)-1616(N) and / or grouped computing resources 1614. In at least one embodiment, resource coordinator 1622 may include a Software Design Infrastructure (“SDI”) management entity for data center 1600. Resource coordinator 1622 may include hardware, software, or some combination thereof.

[0218] In at least one embodiment, such as Figure 16As shown, framework layer 1620 may include a job scheduler 1632, a configuration manager 1634, a resource manager 1636, and / or a distributed file system 1638. Framework layer 1620 may include a framework for software 1632 supporting software layer 1630 and / or one or more applications 1642 of application layer 1640. Software 1632 or application 1642 may respectively contain web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. Framework layer 1620 may be, but is not limited to, free and open-source software web application frameworks (such as Apache Spark) that can utilize distributed file system 1638 for large-scale data processing (e.g., "big data"). TM (Hereinafter referred to as "Spark") is a type of resource. In at least one embodiment, the job scheduler 1632 may include Spark drivers to facilitate the scheduling of workloads supported by different layers of data center 1600. The configuration manager 1634 may be able to configure different layers, such as the software layer 1630 and the framework layer 1620 (which includes Spark and a distributed file system 1638 for supporting large-scale data processing). The resource manager 1636 may be able to manage compute resources mapped to or allocated to clusters of distributed file system 1638 and job scheduler 1632 to support distributed file system 1638 and job scheduler 1632. In at least one embodiment, the clustered or grouped compute resources may include grouped compute resources 1614 in the data center infrastructure layer 1610. The resource manager 1036 may coordinate with the resource coordinator 1612 to manage these mapped or allocated compute resources.

[0219] In at least one embodiment, the software 1632 included in software layer 1630 may include software used in at least a portion of the nodes CRs 1616(1)-1616(N), the grouped computing resources 1614, and / or the distributed file system 1638 of framework layer 1620. One or more types of software may include, but are not limited to, internet web search software, email virus scanning software, database software, and streaming video content software.

[0220] In at least one embodiment, the application 1642 included in the application layer 1640 may include one or more types of applications used at least in part by nodes CRs 1616(1)-1616(N), grouped computing resources 1614, and / or the distributed file system 1638 of the framework layer 1620. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in combination with one or more embodiments.

[0221] In at least one embodiment, any of the configuration manager 1634, resource manager 1636, and resource coordinator 1612 can implement any number and type of self-modification actions based on any amount and type of data obtained in any technically feasible manner. Self-modification actions can free the data center operator of data center 1600 from making potentially poor configuration decisions and may prevent underutilization and / or poor performance of the data center.

[0222] According to one or more embodiments described herein, data center 1600 may include tools, services, software, or other resources to train one or more machine learning models or to use one or more machine learning models to predict or infer information. For example, one or more machine learning models can be trained by using the software and / or computing resources described above with respect to data center 1600 to compute weight parameters according to a neural network architecture. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks can be used to infer or predict information using the resources described above with respect to data center 1600 by using weight parameters computed through one or more training techniques (such as, but not limited to, those described herein).

[0223] In at least one embodiment, the data center 1600 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the software and / or hardware resources described above may be configured to allow a user to train or perform services that infer information, such as image recognition, speech recognition, or other artificial intelligence services.

[0224] Example network environment

[0225] A network environment suitable for implementing embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Client devices, servers, and / or other device types (e.g., each device) may be... Figure 15 This is implemented on one or more instances of computing device 1500—for example, each device may include similar components, features, and / or functions of computing device 1500. Furthermore, in the case of implementing backend devices (e.g., servers, NAS, etc.), the backend devices may be included as part of data center 1600, examples of which are described herein. Figure 16 To describe in more detail.

[0226] Components of a network environment can communicate with each other via a network, which can be wired, wireless, or both. A network can include multiple networks or one of multiple networks. For example, a network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks (such as the Internet and / or the Public Switched Telephone Network (PSTN)), and / or one or more private networks. Where the network includes a wireless telecommunications network, components such as base stations, communication towers, or even access points (and other components) can provide wireless connectivity.

[0227] A compatible network environment may include one or more peer-to-peer network environments (in which case the server may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein for the server can be implemented on any number of client devices.

[0228] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, combinations thereof, etc. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework supporting software at the software layer and / or application at the application layer. The software or application may respectively include network-based service software or applications. In embodiments, one or more client devices may use the network-based service software or applications (e.g., by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a free and open-source software network application framework that can use a distributed file system for large-scale data processing (e.g., "big data").

[0229] A cloud-based network environment can provide cloud computing and / or cloud storage for any combination of the computing and / or data storage functions (or one or more portions thereof) described herein. Any of these different functions can be distributed across multiple locations from a central or core server (e.g., distributed across one or more data centers at the state, region, country, global, etc.). The core server may assign at least a portion of the functionality to the edge server if the connection to the user (e.g., a client device) is relatively close to the edge server. A cloud-based network environment can be private (e.g., limited to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0230] (One or more) client devices may include the information described in this article. Figure 15 At least some of the components, features, and functions of the described (one or more) example computing device 1500. By way of example and not limitation, the client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, camera, surveillance equipment or system, vehicle, ship, spacecraft, virtual machine, drone, robot, handheld communication device, hospital equipment, gaming equipment or system, entertainment system, vehicle computer system, embedded system controller, remote control, electrical appliance, consumer electronics device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0231] This disclosure can be described in the general context of machine-usable instructions or computer code, including computer-executable instructions such as program modules, which are executed by a computer or other machine such as a personal digital assistant or other handheld device. Typically, a program module, including routines, programs, objects, components, data structures, etc., refers to code that performs a specific task or implements a specific abstract data type. This disclosure can be practiced in a wide variety of system configurations, including handheld devices, consumer electronics, general-purpose computers, more specialized computing devices, etc. This disclosure can also be practiced in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0232] As used herein, the phrase "and / or" relating to two or more elements should be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" can include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or element A, B, and C. Furthermore, "at least one of element A or element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Further, "at least one of element A and element B" can include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0233] The subject matter of this disclosure is described in detail herein to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the discloser has envisioned that the claimed subject matter may be embodied in other ways to include steps different from or similar combinations of steps described herein in conjunction with other current or future techniques. Moreover, although the terms “step” and / or “block” may be used herein to imply different elements of the method employed, these terms should not be construed as implying any particular order among or between the various steps disclosed herein, unless the order of the steps is explicitly described.

Claims

1. A ground truth data generation method comprising: accessing sensor data corresponding to two or more sensor modalities, the sensor data being obtained using two or more sensors; generating, based at least on alignment of representations of one or more temporal offsets, a representation of a sequence of annotated scenes comprising corresponding frames of the sensor data from each of the two or more sensors; generating, using a labeling tool, a representation of a sequence of annotation tasks ordering annotations of at least a portion of the sensor data corresponding to the two or more sensor modalities based at least on sensor modality, the annotations being in ground truth annotations; accepting, using the labeling tool, input of one or more annotated scenes from the sequence of annotated scenes annotated with a set of the ground truth annotations defined by at least one of the sequence of annotation tasks; and deriving a representation of the set of ground truth annotations.

2. The method of claim 1, wherein the generation of the sequence of annotated scenes comprises identifying a frame of obtained image data that is closest in time to an obtained LiDAR spin and computing a corresponding one of the one or more temporal offsets between a start angle of the obtained LiDAR spin and a perspective represented by the obtained image data.

3. The method of claim 1, wherein the two or more sensors comprise a LiDAR sensor and one or more cameras, and the sequence of annotation tasks comprises: a first annotation task comprising annotation of camera frames in the sequence of annotated scenes; a second annotation task comprising annotation of LiDAR frames in the sequence of annotated scenes in two-dimensional space; and a third annotation task comprising linking of annotations appearing in multiple annotated scenes in the sequence of annotated scenes.

4. The method of claim 3, wherein the sequence of annotation tasks further comprises: a fourth annotation task comprising annotation of the LiDAR frames in the sequence of annotated scenes in three-dimensional space; and a fifth annotation task comprising linking of annotations in both LiDAR frames and camera frames appearing in any individual annotated scene in the sequence of annotated scenes.

5. The method of claim 1, wherein the sequence of annotation tasks separates annotations of different types of objects or different levels of annotation detail into separate annotation tasks.

6. The method of claim 1, further comprising: During at least one individual annotation task in the sequence of annotation tasks, using the labeling tool, initializing an annotation set for at least one individual annotated scene in the sequence of annotated scenes with a previous annotation set of a previous annotated scene in the sequence of annotated scenes during the at least one individual annotation task.

7. The method of claim 1, wherein at least one of the annotation tasks restricts annotations to a specified sensor modality of the two or more sensor modalities.

8. The method of claim 1, wherein the at least one individual annotated scene in the sequence of annotated scenes includes a first frame of the sensor data from a first type of sensor and a second frame of the sensor data from a second type of sensor, the method further comprising: using the labeling tool: determining a portion of the first frame identified by an input probe; projecting coordinates of the portion of the first frame into coordinates of a corresponding portion of the second frame; and causing visualization of the corresponding portion of the second frame.

9. The method of claim 1, further comprising: using the labeling tool, traversing a per-object process during the at least one annotation task, the per-object process comprising, for at least one annotation of an annotation scene in the sequence of annotation scenes, the at least one annotation received during a previous annotation task in the sequence of annotation tasks: causing presentation of a zoomed-in view of the annotation in the annotation scene; and prompting for and accepting input adjusting or confirming the annotation.

10. The method of claim 1, further comprising: using the labeling tool, iteratively traversing the annotation scenes in one or more labeling interfaces of the labeling tool; and prompting for and accepting input annotating the one or more annotation scenes with a corresponding set of ground-truth annotations defined by the at least one annotation task, while causing presentation of the corresponding frames of the sensor data from each of the two or more sensors.

11. One or more processors comprising one or more circuits to: access sensor data resulting during a capture session, the sensor data comprising LiDAR frames from a LiDAR sensor and camera frames from at least one camera; compose a sequence of annotation scenes based at least on a representation of at least one temporal offset between at least one of the LiDAR frames and at least one of the camera frames, the sequence of annotation scenes comprising at least one annotation scene identifying the at least one LiDAR frame and the at least one camera frame; generate, using a labeling tool, a representation of a sequence of annotation tasks ordering annotations of at least a portion of the sequence of annotation scenes based at least on sensor modality, the annotations annotated with ground-truth annotations; accept, using the labeling tool, input annotating one or more annotation scenes in the sequence of annotation scenes with a set of ground-truth annotations defined by at least one annotation task in the sequence of annotation tasks; and export a representation of the set of ground-truth annotations.

12. The one or more processors of claim 11, wherein the one or more circuits are further to generate the at least one annotation scene based at least on determining that the at least one camera frame is temporally closest to the at least one LiDAR frame and computing the at least one temporal offset between a start angle of at least one obtained LiDAR spin represented by the at least one LiDAR frame and a corresponding perspective represented by the at least one camera frame.

13. The one or more processors of claim 11, wherein the sequence of annotation tasks comprises: a first annotation task comprising annotations of a set of camera frames in the sequence of annotation scenes; a second annotation task comprising annotations of a set of LiDAR frames in the sequence of annotation scenes in two-dimensional space; and a third annotation task comprising linking annotations appearing in multiple annotation scenes in the sequence of annotation scenes.

14. The one or more processors of claim 11, wherein the one or more circuits are further to, during at least one individual annotation task of the sequence of annotation tasks, initialize a set of annotations for at least one individual annotation scene of the sequence of annotation scenes with a previous set of annotations of a previous annotation scene of the sequence of annotation scenes during the at least one individual annotation task.

15. The one or more processors of claim 11, wherein the one or more circuits are further to: identify a correspondence between a portion of the at least one LiDAR frame and a portion of the at least one camera frame based at least on projecting information from coordinates of the portion of the at least one LiDAR frame into coordinates of the portion of the at least one camera frame; and cause presentation of a visualization of the correspondence.

16. The one or more processors of claim 11, wherein the one or more circuits are further to: determine a portion of the at least one LiDAR frame identified by an input probe; project coordinates of the portion of the at least one LiDAR frame into coordinates of a corresponding portion of the at least one camera frame; and cause visualization of the corresponding portion of the at least one camera frame.

17. The one or more processors of claim 11, wherein the one or more circuits are further to iterate through a per-object process during the at least one annotation task, the per-object process comprising: for at least one annotation of an annotation scene, the at least one annotation received during a previous annotation task of the sequence of annotation tasks: cause presentation of a zoomed-in view of the at least one annotation in the annotation scene; and prompt for and accept input adjusting or confirming the at least one annotation.

18. A ground truth data generation system comprising one or more processors to: generate, using a labeling tool, a representation of a sequence of annotation scenes comprising at least a portion of the sequence of annotation scenes annotated with ground truth annotations, at least one annotation scene of the sequence of annotation scenes comprising aligned frames of sensor data corresponding to a plurality of sensor modalities, the annotations ordered based on sensor modality using a sequence of annotation tasks; iteratively traverse, using the labeling tool, the sequence of annotation scenes in one or more labeling interfaces of the labeling tool; and accept, using the labeling tool, input annotating the at least one annotation scene with the set of ground truth annotations defined by at least one annotation task of the sequence of annotation tasks while causing presentation of the set of aligned frames of sensor data corresponding to the plurality of sensor modalities; and derive a representation of the set of ground truth annotations.

19. The system of claim 18, wherein at least one annotation scene of the sequence of annotation scenes comprises a first frame of the sensor data from a first type of sensor and a second frame of the sensor data from a second type of sensor, wherein the one or more processors are further to use the labeling tool to: identify a correspondence between a portion of the first frame and a portion of the second frame based at least on projecting information from coordinates of the portion of the first frame into coordinates of the portion of the second frame; and a presentation that causes a visualization of the correspondence.

20. The system of claim 18, wherein the system is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing deep learning operations; a system implemented using edge devices; a system implemented using robots; a system comprising one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources.

Images