Viewing distance estimation using deep learning in autonomous machine applications

Abstract

To provide a system and a method that use one or more machine learning models (MLMs), such as deep neural networks (DNNs).SOLUTION: In various examples, systems and methods are disclosed that use one or more machine learning models (MLMs), such as deep neural networks (DNNs) to compute an output indicating an estimated visible distance corresponding to sensor data generated using one or more sensors of an autonomous or a semi-autonomous machine. After the visible distance is calculated using one or more MLMs, determination of usefulness of the sensor data for one or more downstream tasks of the machine can be evaluated. As such, when the estimated visible distance is low, corresponding sensor data can be relied upon for fewer tasks than when the visible distance is high.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This application relates to U.S. Nonprovisional Patent Application No. 16 / 570,187, filed on 13 September 2019, which is incorporated in its entirety by reference herein. [Background technology]

[0002] Autonomous and semi-autonomous driving systems (e.g., advanced driver assistance systems (ADAS)) can utilize sensors (e.g., cameras, LiDAR sensors, RADAR sensors, etc.) to perform various tasks—for example, blind spot monitoring, automatic emergency braking, lane keeping, object detection, obstacle avoidance, and localization. For example, real-time or near-real-time perception of the vehicle's surroundings may be required for autonomous and ADAS systems to operate independently and efficiently. To accurately and efficiently perceive the vehicle's surroundings, sensors must generate usable, unobstructed sensor data (e.g., images, depth maps, point clouds, etc.). However, the ability of sensors to perceive the surrounding environment can be impaired by various sources—e.g., weather (e.g., rain, fog, snow, hail, smoke, etc.), traffic conditions, sensor interference (e.g., from debris, moisture, etc.), or blurring. As a result, the resulting sensor data may not clearly depict the vehicle, obstacles, and / or other objects in the environment.

[0003] Conventional systems for addressing impaired visibility have employed feature-level techniques to detect individual visual evidence and then stitch these features together to determine the presence of impaired visibility. These conventional methods primarily rely on computer vision techniques such as analyzing the absence of sharp edge features (e.g., gradients, abrupt changes in color or intensity) in areas of an image, detecting potential visibility problems using color-based pixel analysis or other low-level feature analysis, and / or binary support vector machine classification with blind vs. non-blind outputs. However, such feature-based computer vision techniques require separate analysis of each feature, such as whether each feature is relevant to visibility, and analysis of how to combine different features of specific sensor visibility degradation conditions, thereby limiting the scalability of such techniques due to the complexity inherent in the wide variety of conditions and events that can impair data observed using sensors in real-world situations. For example, the computational cost of performing these conventional techniques renders them ineffective for real-time or near-real-time deployment.

[0004] Furthermore, conventional systems may rely on classifying the causes of reduced sensor visibility—e.g., rain, snow, fog, glare—but this may not provide accurate indication of the usefulness of the sensor data. For example, identifying rain in an image may not be feasible for the system to determine whether the corresponding image—or part thereof—is usable for various autonomous or semi-autonomous tasks. In such instances, an image may clearly depict the environment within 100 meters of a vehicle, but if it is raining, the image may be considered unusable by conventional systems. As such, instead of relying on the image for one or more tasks within the visible range, the image may be mistakenly discarded, and one or more tasks may be rendered unusable. In this way, by treating each type of impaired sensor visibility equally, less severe or less harmful types of sensors may result in instances of sensor data being considered unusable, even though this determination may not be entirely accurate (for example, an image of an environment with light rain may be usable for one or more operations, while an image of an environment with dense fog may be unusable). [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] U.S. Patent Application No. 16 / 101,232 [Overview of the Initiative]

[0006] Embodiments of this disclosure relate to deep neural network processing for estimating line of sight distance in autonomous machine applications—for example, the furthest distance from a sensor at which an object or element can be identified. Systems and methods are disclosed for calculating an output indicating the estimated line of sight distance (for example, in the form of a calculated distance or distance bin, including a range of distances) using machine learning models such as one or more deep neural networks (DNNs) corresponding to one or more sensors of an autonomous or semi-autonomous machine. For example, by predicting the estimated line of sight distance, the machine's reliance on relevant sensor data for one or more downstream tasks—for example, object detection, object tracking, obstacle avoidance, path planning, control decision, and / or similar—can be adjusted. For example, if the estimated sight distance is low—for instance, less than 20 meters—the corresponding sensor data may only be relied upon for Level 0 (no automation) or Level 1 (driver assistance) tasks (for example, according to the Society of Automotive Engineers (SAE) automation level), or only for predictions of the machine within 20 meters (for example, and predictions beyond 20 meters may be ignored or have lower relevant reliability). Similarly, as another example, if the estimated sight distance is high—for example, more than 1000 meters—the corresponding sensor data may only be relied upon for the full performance of Level 3 (conditional driving automation) and Level 4 (high driving automation) tasks, or only for predictions corresponding to the machine's position within 1000 meters.

[0007] Thus, in contrast to conventional systems such as those described above, the systems and methods of this disclosure can be used not only to determine the usefulness of sensor data, but also to determine the level or degree of usefulness of sensor data as defined using sight distance—or sight distance bins having a range of relevant sight distances. To train a machine learning model, such as a DNN, to accurately compute an output representing sight distance, the DNN may be trained using real-world data, augmented reality data, and / or synthetic data representing sensor data representations (e.g., images, LiDAR point clouds, etc.) that include changing weather, lighting, and / or other conditions. Each sensor data instance may include corresponding ground truth data representing sight distance and / or sight distance bins. In some embodiments, the ground truth data may be automatically generated using one or more trained models such that there exists a known or estimated sight distance for a given parameter—e.g., fog density, rainwater moisture content, rain intensity, etc. As such, a robust training set may be generated using one or more models (for example, different models may correspond to different sensor data types such as real-world, augmented, and / or synthetic), and the machine learning model may be trained using a combination of training data and associated ground truth data.

[0008] The system and method for deep neural network processing of sight distance estimation in autonomous machine applications are described in detail below with reference to the attached drawings. [Brief explanation of the drawing]

[0009] [Figure 1] This data flow diagram illustrates the process for training a machine learning model to calculate estimated sight distance according to some embodiments of the present disclosure. [Figure 2A]This is an exemplary visualization of sensor data with different viewing distances, according to some embodiments of the present disclosure. [Figure 2B] This is an exemplary visualization of sensor data with different viewing distances, according to some embodiments of the present disclosure. [Figure 2C] This is an exemplary visualization of sensor data with different viewing distances, according to some embodiments of the present disclosure. [Figure 3] This flowchart illustrates a method for training a machine learning model to calculate estimated sight distance, according to some embodiments of the present disclosure. [Figure 4] This data flow diagram illustrates the process for deploying a machine learning model to calculate estimated sight distance according to some embodiments of the present disclosure. [Figure 5] This is an exemplary visualization of sensor data and corresponding sight distance output of various objects according to some embodiments of the present disclosure. [Figure 6] This flowchart illustrates a method for deploying a machine learning model to calculate estimated sight distance according to some embodiments of the present disclosure. [Figure 7A] This is an illustration of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 7B] Figure 7A shows examples of camera positions and fields of view of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 7C] Figure 7A is a block diagram of an exemplary system architecture of an exemplary autonomous vehicle according to some embodiments of the present disclosure. [Figure 7D] This is a system diagram of communication between a cloud-based server and the exemplary autonomous vehicle shown in Figure 7A, according to some embodiments of the present disclosure. [Figure 8] This is a block diagram of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure. [Figure 9] This is an exemplary data center block diagram suitable for use in implementing some embodiments of the present disclosure.

Best Mode for Carrying Out the Invention

[0010] Systems and methods are disclosed in connection with deep neural network processing for visual range estimation in autonomous machine applications. Although the present disclosure may be described with respect to an exemplary autonomous vehicle 700 (or also referred to herein as "vehicle 700" or "ego vehicle 700", examples of which are described with respect to FIGS. 7A - 7D), this is not intended to be limiting. For example, the systems and methods described herein may be used by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more advanced driver assistance systems (ADAS)), manned and unmanned robots or robot platforms, warehouse vehicles, off-road vehicles, vehicles connected to one or more trailers, flying vessels, boats, shuttles, emergency response vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, submarines, drones, and / or other vehicle types, but are not limited thereto. Additionally, although the present disclosure may be described with respect to visual range estimation in autonomous or semi-autonomous machine applications, this is not intended to be limiting, and the systems and methods described herein may be used in augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications, and / or any other technical space where sensor data conditions and usefulness can be analyzed.

[0011] (Training of a Machine Learning Model for Calculating Visual Range) Referring to Figure 1, Figure 1 is a data flow diagram corresponding to an exemplary process 100 for training a machine learning model for sight distance estimation, according to several embodiments of the present disclosure. It should be understood that this and other configurations described herein are merely illustrative. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be performed using similar components, features, and / or functionalities in the exemplary autonomous vehicle 700 of Figures 7A–7D, the exemplary computing device 800 of Figure 8, and / or the exemplary data center 900 of Figure 9.

[0012] The machine learning model 104 can be trained using training data 102 - for example, sensor data 102A, augmented data 102B, and / or synthetic data 102C. For example, the sensor data 102A can correspond to real-world sensor data generated using one or more sensors of the ego machine 700, such as a stereo camera 768, a RADAR sensor 760, an ultrasonic sensor 762, a LiDAR sensor 764, a wide-view camera 770, a surround camera 774, other sensors of the ego machine 700, and / or other sensor types. For example, the sensor data 102A can be collected using one or more data collection vehicles that collect various types of sensor data 102A under various conditions - for example, changing weather, lighting, occlusion, and / or other conditions. In embodiments, generating a sufficiently diverse training data set can be impractical and / or very expensive using only the sensor data 102A, so the augmented data 102B and / or synthetic data 102C can be generated in addition to or instead of the real-world sensor data 102A.

[0013] Extended data 102B can correspond to real-world sensor data 102A by simulating sensor data captured under various conditions other than the actual conditions under which sensor data 102A was initially captured—for example, to include various weather conditions (fog, snow, rain, sleet, etc.), lighting conditions (darkness, sunlight, sun shining towards the sensor, etc.), occlusion, and / or other conditions. For example, if sensor data 102A was captured under clear skies, sensor data 102A may be extended using rain, fog, snow, and / or other conditions to generate extended sensor data 102B. As another example, if sensor data 102A was captured under light rain, sensor data 102A may be extended to include heavier rain and / or fog in addition to the light rain. To do this, in embodiments, different levels of rain (e.g., drizzle, heavy rain, etc.), fog (e.g., dense fog, light fog, etc.), snow (e.g., heavy snow, light snow, snow showers, etc.), and / or other conditions (e.g., different sun positions that produce different lighting conditions for the sensor) may be applied to generate augmented data 102B. As such, values ​​corresponding to parameters that control one or more conditions may be determined, for example, manually, automatically, and / or randomly, to generate augmented data 102B. For example, values ​​for a fog density parameter, a rain intensity parameter, and / or other parameters for another condition may be determined, and sensor data 102A may be augmented based on these values ​​to generate augmented data 102B.

[0014] The synthesized data 102C may correspond to sensor data generated using one or more virtual sensors (e.g., cameras, LiDAR sensors, RADAR sensors, etc.) in one or more virtual machines (e.g., virtual instances of vehicle 700) within a virtual environment (e.g., virtual cameras of a virtual vehicle in a virtual or simulated environment). For example, a simulation or game engine may be used to generate a simulated environment for virtual machines, and the virtual sensors may generate the synthesized data 102C from within the simulated environment for use as training data 102. To generate a simulated or virtual environment that corresponds to various conditions—e.g., weather, lighting, occlusion, etc.—the values ​​of parameters corresponding to various conditions may be set manually, automatically, and / or randomly to generate diverse synthesized data 102C. In addition, road layout in the virtual environment (e.g., number of lanes, curvature, elevation changes, etc.), traffic conditions, surrounding environmental conditions, scenery, object locations and / or types, and / or other factors within the virtual environment may be selected or randomized to further diversify the training data 102 using the synthesized data 102C.

[0015] To ensure consistency between the augmented data 102B and the composite data 102C, and thus more accurate training data 102, the augmented and simulation parameters may be calibrated. This can increase the likelihood that data generated using augmentation and simulation with the same sight distance labels are visually identical or similar. To perform calibration, in embodiments, instances of sight augmented data 102B and instances of composite data 102C may be curated to include objects at the same distance corresponding to the sight distance. For each instance, one pair of augmented instances and one pair of composite instances may be generated at a distance of +2 meters (as an example and not limited to) and at a distance of -2 meters (as an example and not limited to). The assessor can then determine whether the target object is visible in both instances of augmented data 102B and both instances of composite data 102C. According to embodiments under such exemplary scenarios, the parameters may then be adjusted until the object is visible at a distance of -2 meters and invisible at a distance of +2 meters. For the same viewing distance, parameters such as rain, snow, and fog can be adjusted so that the resulting training data of the composite or augmented data are visually similar.

[0016] In embodiments, the training data 102 may include the initial data 102A, 102B, and / or 102C, downsampled data, upsampled data, cropped or region of interest (ROI) data, inverted or rotated data, other augmented data, and / or combinations thereof, in order to further diversify the training dataset and to increase the robustness of the training dataset.

[0017] In addition to the corresponding ground truth data generated using the ground truth generator 116, training data 102 may be used to train a machine learning model 104 to compute outputs 106 (e.g., sight distance 108, distance bin 110, and / or sightability classification and / or attribute 112). Examples have been described herein with respect to the use of deep neural networks (DNNs), and more specifically convolutional neural networks (CNNs), as the machine learning model 104, but this is not intended to be an limitation. For example, and not limited to, machine learning models 104 may include any type of machine learning model, such as linear regression, logistic regression, decision trees, support vector machines (SVMs), naive Bayes, k-nearest neighbors (Knn), K-means clustering, random forests, dimensionality reduction algorithms, gradient boosting algorithms, neural networks (e.g., autoencoders, convolutions, recurrent, perceptrons, long / short-term memory / LSTMs, Hopfield, Boltzmann, deep belief, deconvolution, adversarial generative, liquid state machines, etc.), machine learning models using computer vision algorithms, and / or other types of machine learning models.

[0018] As an example, if the machine learning model 104 includes a CNN, the machine learning model 104 may include any number of layers. One or more of these layers may include an input layer. The input layer may hold values ​​related to the training data 102 (or, during deployment, sensor data 402) (e.g., before or after post-processing). For example, if the training data represents an image, the input layer may hold values ​​representing the raw pixel values ​​of the image as a volume (e.g., width, height, and color channels (e.g., RGB), such as 32x32x3).

[0019] One or more layers may include convolutional layers. A convolutional layer can compute the outputs of neurons connected to local regions in the input layer, and each neuron computes the dot product (inner product) of its weights and the smaller regions to which it is connected in the input volume. The results of the convolutional layer may be another volume having one of the dimensions based on the number of filters applied (e.g., width, height, and number of filters; for example, 32x32x12 if 12 is the number of filters).

[0020] One or more layers may include a deconvolutional layer (or a transposed deconvolutional layer). For example, the result of a deconvolutional layer may be another volume having a higher dimension than the input dimension of the data received in the deconvolutional layer.

[0021] One or more of the layers may include a ReLU (rectified linear unit) layer. The ReLU layer may apply, for example, a zero-threshold element-wise activation function, such as max(0,x). The resulting volume of the ReLU layer may be the same as the volume of the input to the ReLU layer.

[0022] One or more of the layers may include a pooling layer. The pooling layer may perform downsampling operations along spatial dimensions (e.g., height and width), resulting in a smaller volume than the pooling layer's input volume (e.g., from a 32x32x12 input volume to 16x16x12).

[0023] One or more of the layers may contain one or more fully connected layers. Each neuron in a fully connected layer may be connected to each of the neurons in the previous volume. A fully connected layer may compute class scores, and the resulting volume may be 1x1xn, where n is equal to the number of classes. In some examples, a CNN may contain fully connected layers such that the output of one or more of the layers of the CNN may be provided as input to a fully connected layer of the CNN. In some examples, one or more convolutional streams may be implemented by a machine learning model 104, and some or all of the convolutional streams may contain their respective fully connected layers.

[0024] In some non-limiting embodiments, the machine learning model 104 may include a series of convolutional and max-pooling layers for facilitating image feature extraction, followed by multiscale augmented convolutional and upsampling layers for facilitating global context feature extraction.

[0025] Input layers, convolutional layers, pooling layers, ReLU layers, and fully connected layers are discussed herein in relation to machine learning model 104, but this is not intended to be limiting. Additional or alternative layers, such as normalization layers, SoftMax layers, and / or other layer types, may be used in machine learning model 104.

[0026] In addition, some layers, such as convolutional layers and fully connected layers, may include parameters (e.g., weights and / or biases), while other layers, such as ReLU layers and pooling layers, may not. In some examples, parameters may be learned by the machine learning model 104 during training. Furthermore, some layers, such as convolutional layers, fully connected layers, and pooling layers, may include additional hyperparameters (e.g., learning rate, stride, epoch, etc.), while other layers, such as ReLU layers, may not. In an embodiment where the machine learning model 104 regresses on sight distance, the activation function of the last layer of the CNN may include a ReLU activation function. In an embodiment where the machine learning model 104 classifies instances of sensor data into distance bins 110, the activation function of the last layer of the CNN may include a SoftMax activation function. Parameters and hyperparameters may vary depending on the embodiment, without limitation.

[0027] In embodiments where the machine learning model 104 includes a CNN, different orders and numbers of layers in the CNN may be used depending on the embodiment. In other words, the order and number of layers in the CNN are not limited to any single architecture.

[0028] For example, in one or more embodiments, the CNN may include an encoder-decoder architecture and / or one or more output heads. For example, the CNN may include one or more layers corresponding to the CNN's feature detection trunk, and the output of the feature detection trunk (e.g., a feature map) may be processed using one or more output heads. For example, a first output head (including one or more first layers) may be used to calculate the viewability distance 108, a second output head (including one or more second layers) may be used to calculate the distance bin 110, and / or a third output head (including one or more third layers) may be used to calculate the viewability classification / attribute 112. When two or more heads are used as such, the two or more heads can process data from the trunk in parallel, and each head may be trained to accurately predict the corresponding output of the output head. However, in other embodiments, a single trunk may be used without separate heads.

[0029] The ground truth generator 116 may be used to generate real-world labels or annotations 118 as ground truth data corresponding to real-world sensor data 102A. For example, for sensor data 102A generated in a real-world environment—without extensions—the real-world labels or annotations may correspond to view distance labels indicating the view distance for each instance of the sensor data 102A. When real-world labels or annotations 118 are used to generate ground truth data, in some examples the annotations or labels may be generated within a drawing program (e.g., an annotation program), a computer-aided design (CAD) program, a labeling program, another type of program suitable for generating annotations, and / or handwritten. In any given instance, ground truth data may be synthetically generated (e.g., from computer models or renderings), created in reality (e.g., designed and created from real-world data), automated by machines (e.g., using feature analysis and learning to extract features from the data and then generate labels), annotated by humans (e.g., a labeler or annotation expert defines the location of the labels), and / or a combination of these (e.g., a person identifies visible objects and a computer determines the distance to the objects and the corresponding viewing distance).

[0030] The real-world labels 118 may correspond to actual sight distances—e.g., meters, feet, etc.—and / or to distance bins (e.g., each distance bin may contain a range of sight distance values). Without limitation, if distance bins are used, for example, they may include a first bin for very low visibility (e.g., <10 meters), a second bin for low visibility (e.g., between 10 meters and 100 meters), a third bin for medium visibility (e.g., between 100 meters and 1000 meters), a fourth bin for high visibility (e.g., between 1000 meters and 4000 meters), and a fifth bin for clear visibility (e.g., greater than 4000 meters). While five bins are listed here with their corresponding ranges, this is not intended to be limiting, and several distance bins with values ​​in any range may be used without departing the scope of this disclosure. As such, for a given instance of sensor data 102A, the sensor data 102A may be labeled with sight distance and / or distance bins, and these values ​​may be used during training to compare with the output 106 of a machine learning model 104 using, for example, a loss function 114. In some instances, the machine learning model 104 may also be trained to compute, for example, a visibility classification and / or attribute 112 corresponding to the cause of the decrease in visibility, if present. In other instances, in addition to or instead of the sight distance classification, the machine learning model 104 may be trained to compute a sensor visibility loss classification corresponding to impaired sensor data (e.g., raindrops on the camera lens, snow blocking the sensor, etc.). In such instances, the ground truth data and output may be analogous to U.S. Nonprovisional Patent Application No. 16 / 570,187, filed September 13, 2019, which is incorporated herein by reference in its entirety.

[0031] Static and / or dynamic object positions may be used to determine the viewing distance and / or distance bin of a given instance of sensor data 102A. For example, in some instances, one or more depth or distance values ​​may be determined for an object in the environment using one or more machine learning, computer vision, depth sensors, and / or other outputs, and these depth or distance values ​​may be used to determine the viewing distance information for the instance of sensor data 102A. With respect to Figure 2A, assuming that visualization 200A (including heavy rain) corresponds to an instance of sensor data 102A (e.g., an image from a camera on a data collection vehicle), depth values ​​may be known for vehicle 202A based on outputs from depth sensors (e.g., LiDAR, RADAR, etc.), outputs from machine learning models or neural networks, outputs from computer vision algorithms, and / or other output types. In such an example, when a data acquisition vehicle is generating an instance of sensor data 102A that includes vehicle 202A, the data acquisition vehicle may perform one or more basic processes to determine depth or distance information to objects in the environment. For example, if vehicle 202A was the furthest visible object from the data acquisition vehicle, and the depth to vehicle 202A is known, the sight distance of the instance of sensor data 102A may correspond to the depth or distance to vehicle 202A (for example, in an embodiment, an additional distance may be added if the environment is visible beyond vehicle 202A, or some distance may be subtracted if vehicle 202A is only partially visible or blurred). Similarly, for visualization 200B (including light rain conditions), vehicle 202B may be visible, and therefore the sight distance and / or distance bin may be determined using the depth or distance value corresponding to vehicle 202B.

[0032] In some embodiments, in addition to or instead of using depth output from sensors, machine learning models, computer vision algorithms, etc., high-resolution (HD) maps may be used to determine the distance or depth to static objects in the environment. For example, identifiable or visible objects in the environment that have a known distance from the localized position of the data acquisition vehicle, using a localization technique to localize the data acquisition vehicle with respect to an HD map, may be used to determine the viewing distance of a given instance of sensor data 102A. For example, if traffic signs, trees, streetlights, intersections, buildings, and / or static features are visible in the instance of sensor data 102A, and the distance to the object or feature is known from the HD map after localization, the distance or depth may be used as the viewing distance and / or to determine the distance bin.

[0033] In some embodiments, this determination of the ground truth label for the viewing distance and / or distance bin can be performed manually. For example, an annotator may determine the furthest object visible in the instance of sensor data 102A, determine the associated distance or depth value of that object, and generate the ground truth accordingly. In other embodiments, the determination of the ground truth label for the viewing distance and / or distance bin can be performed automatically, such that the largest depth or distance value of an object identified using a depth sensor output, machine learning model, DNN, computer vision algorithm, HD map, etc. (e.g., the furthest distance from the data acquisition vehicle) can be used as the viewing distance value and / or used to determine the distance bin for the instance of sensor data 102A.

[0034] In the embodiment, a similar process may be used with respect to sensor data 102A to generate ground truth data for augmented data 102B. For example, known distance or depth values ​​of objects in the environment may be used to determine the sight distance and / or distance bin. Additionally or instead, a model may be trained to determine the correspondence or mapping between augmentation parameter values ​​and sight distance and / or distance bin. For example, fog parameter values ​​(e.g., density, height, etc.) may be used to augment sensor data 102A to generate augmented data 102B. Instances of augmented data 102B may then be analyzed to determine the sight distance and / or distance bin (e.g., using known depth values ​​of visible objects after augmentation). This process may be repeated for any number of instances of augmented data 102B until the model is trained to calculate the sight distance and / or distance bin based on parameter values ​​for augmentation. After the model has been trained, it may be used to automatically generate ground truth for the automatically generated augmented data. For example, the values ​​of parameters (e.g., rain, snow, fog, sleet, lighting, occlusion, etc.) may be randomized to generate augmented data 102B, and these values ​​(or combinations thereof) may have known correspondences to sight distances and / or distance bins, which may be used as ground truth for instances of augmented data 102B.

[0035] As an example of training a model or creating a mapping between parameter values ​​and visibility distance and / or distance bins, with respect to Figures 2A-2C, visualization 200A may correspond to the values ​​of the rain parameter used to generate augmented data 102B under heavy rain. An annotator or labeler can then determine the distance to vehicle 202A and label the augmented data 102B as such, which may correspond to a mapping between the values ​​of one or more parameters for augmenting the data and visibility distance and / or distance bins. This process may be repeated for visualization 200B using the parameter values ​​that result in distance to vehicle 202B and light rain, and for visualization 200C using the parameter values ​​that result in distance to vehicle 202C and clear conditions (for example, all rain values ​​are 0 when there is no rain). While rain is illustrated and described in relation to Figures 2A-2C, this is not intended to be limiting, and other conditions such as fog, snow, sleet, hail, illumination, occlusion, and combinations thereof may be used to generate the mapping between the extended data 102B and the parameter values ​​(or combinations thereof) and the ground truth visibility distance and / or distance bins.

[0036] A similar process can be used, as with sensor data 102A, to generate ground truth data for synthetic data 102C, except that known depth information may originate from the simulation engine, when the state data corresponding to the simulation may include precise depth or distance information of objects in the environment. For example, an annotator can identify the furthest visible object in an instance of synthetic data 102C generated according to varying parameter values ​​in the simulation, and the depth or distance from the virtual sensor of the virtual machine to the furthest visible object may be used to determine the distance bin as the viewing distance and / or ground truth. Additionally or instead, in embodiments, a model may be trained to determine the correspondence or mapping between simulation parameter values ​​and viewing distances and / or distance bins. For example, values ​​of fog parameters (e.g., density, height, etc.) may be used to generate the simulated environment of the simulation, and synthetic data 102C may be captured from within the simulated environment. An instance of synthetic data 102C may then be analyzed to determine the viewing distance and / or distance bin (e.g., using known depth values ​​of visible objects in the simulation). This process may be repeated for any number of instances of the synthetic data 102C until the model is trained to calculate the visibility distance and / or distance bin based on the parameter values ​​for the simulation. In some embodiments, the tool may be used to allow the user to identify when a particular object is visible or invisible for a particular set of parameters. For example, a vehicle may be placed in a first position, and the user may indicate that the vehicle is visible. The vehicle may then be moved further away, and the user may indicate that the vehicle is still visible. The vehicle may then be moved further away in the simulated environment, and the user may indicate that the vehicle is no longer visible, and this distance of the vehicle may be used as the ground truth visibility distance and / or to determine the distance bin.This process may be repeated for various parameters and different types of objects until the model is trained. After the model is trained, it can be used to automatically generate ground truth for automatically generated synthetic data. For example, the values ​​of parameters (e.g., rain, snow, fog, sleet, lighting, occlusion, etc.) may be randomized to generate the simulation, and those values ​​(or combinations thereof) may have a known correspondence to visibility distances and / or distance bins, which can be used as ground truth for instances of the synthetic data 102C.

[0037] As an example of training a model or creating a mapping between parameter values ​​and visibility distance and / or distance bins, with respect to Figures 2A-2C, visualization 200A may correspond to the rain parameter values ​​used to generate synthetic data 102C where heavy rain is falling. An annotator or labeler can then determine the distance to vehicle 202A and label synthetic data 102C as such, which may correspond to a mapping between one or more parameter values ​​and visibility distance and / or distance bins for generating a simulation. This process may be repeated for visualization 200B using the distance to vehicle 202B and parameter values ​​that result in light rain, and for visualization 200C using the distance to vehicle 202C and parameter values ​​that lead to clear conditions (e.g., all rain values ​​are 0 when there is no rain). Rain is illustrated and described in relation to Figures 2A-2C, but this is not intended to be limiting, and other conditions such as fog, snow, sleet, hail, illumination, occlusion, and combinations thereof may be used to generate the composite data 102C and, by extension, the mapping between the parameter (or combination thereof) values ​​and the ground truth visibility distance and / or distance bins.

[0038] Once the ground truth data has been generated to correspond to the output 106 of the machine learning model 104 as calculated using the training data 102—using the ground truth generator 116—one or more loss functions 114 may be used to determine the accuracy of the machine learning model 104 and to update the machine learning model 104 in each iteration (e.g., parameters, weights, and biases) until an acceptable level of accuracy is achieved. If the machine learning model 104 is trained to regress on sight distance values, the regression loss may be used as the loss function 114. In such an example, the regression loss may include a normalized L1 loss that may constitute a higher error at farther sight distances and a lower error at shorter sight distances. In such an example, the loss may be calculated according to the following equation (1):

number

[0039] In other examples, normalized L2 loss may be used, or direction-normalized L1 loss may be used. Direction-normalized L1 loss may be similar to normalized L1 loss, but the slope of the loss curve may be steeper on the positive side to incentivize errors on the positive side (for example, to vaporize recall over precision at the loss level). The minimum value of this loss function may still be 0, but predictions below ground truth may be penalized more heavily.

[0040] In other examples, Gaussian distance loss may be used for regression channels to penalize more distances further from ground truth and less predictions closer to ground truth (rather than requiring perfect accuracy). In such examples, overestimating distance may incur a greater penalty than underestimating it.

[0041] In embodiments where the machine learning model 104 outputs distance bins 110 (for example, as classification outputs), a classification loss may be used. For example, a multiclass cross-entropy loss function and / or a directional multiclass cross-entropy loss function may be used to train the machine learning model 104 to compute the distance bins 110.

[0042] During the training of the machine learning model 104, one or more key performance indicators (KPIs) may be used to determine an acceptable level of accuracy. For example, network (or machine learning model) level KPIs may be used, or module level KPIs may be used. A network-level KPI may measure the relative visibility distance prediction error (RDE) for each instance of the training data 102 according to the following equation (2):

number

[0043] Module-level KPIs can measure the error in classifying predicted viewing distances to various bins. The error in classification (Δclass ) To calculate, the following equation (3) can be used: Δ class =c pred -c GT (3) c is a distance bin identifier (ID) (e.g., 1 corresponding to very low, 5 corresponding to distinct), and c pred is the visual recognition distance bin ID predicted by the machine learning model 104, and c GT is the ground truth visual bin ID. As such, Δ class can be 0 for an accurate prediction, positive if the predicted distance bin is higher than expected, and negative if the predicted distance bin is lower than expected. In this way, Δ class =0 is a true positive, Δ class >0 is a false negative, and Δ class <0 can be a false positive. This can be explained by false positives making the ego machine 700 overly conservative (which can be unpleasant for the occupant), and false negatives making the ego machine 700 under - conservative (which can be dangerous for the occupant). Using these definitions of true positive, false positive, and false negative, the module can be adjusted to achieve excellent precision and recall, and may prefer recall if necessary (e.g., preferring to be overly conservative, etc.). Additionally, |Δ class | can be used as an indicator of the predicted error, and a loss function can be used to reduce the spread of |Δ class |.

[0044] Referring here to Figure 3, each block of the method 300 described herein includes a computation process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 300 may also be performed as computer-usable instructions stored on a computer storage medium. Method 300 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 300 is described as an example with respect to process 100 in Figure 1 and ego machine 700 in Figures 7A-7D. However, Method 300 may be performed additionally or instead in any one system and / or any one system, or any combination of processes and systems, including but not limited to those described herein.

[0045] Figure 3 is a flowchart illustrating a method 300 for training a machine learning model to calculate an estimated sight distance, according to some embodiments of the present disclosure. Method 300 includes, in block B302, receiving values ​​corresponding to one or more parameters for adjusting the sight to correspond to an instance of training sensor data. For example, values ​​for one or more parameters (e.g., rain intensity, rain moisture content, fog density, lighting conditions, etc.) may be received.

[0046] Method 300 includes generating instances of training sensor data in block B304 based at least in part on those values. For example, sensor data 102A may be augmented using the values ​​of one or more parameters to generate augmented data 102B and / or a virtual simulation, and synthetic data 102C may be captured using a virtual sensor of a virtual machine.

[0047] Method 300 includes determining the sight distance corresponding to an instance of training sensor data in block B306 based at least in part on those values. For example, one or more trained models may be used to determine the correspondence between one or more values ​​of one or more parameters and the sight distance 108 and / or sight distance bin 110.

[0048] Method 300 includes training a machine learning model using instances of training sensor data and sight distance as ground truth data in block B308. For example, the machine learning model 104 may be trained using one or more loss functions 114 to compare the output 106 with ground truth data (corresponding to, for example, ground truth sight distance, ground truth distance bins, and / or ground truth visibility classification / attributes) generated using the ground truth generator 116. Any number of instances of the training data 102 (e.g., thousands, millions, etc.) may be used to train the machine learning model 104 until the machine learning model 104 reaches an acceptable level of accuracy (as determined in an embodiment using one or more KPIs) and is validated for deployment (e.g., in process 400 in Figure 4).

[0049] (Calculation of visibility distance during deployment) Referring to Figure 4, Figure 4 is a data flow diagram corresponding to an exemplary process 400 for deploying a machine learning model for sight distance estimation according to several embodiments of the present disclosure. It should be understood that this and other configurations described herein are merely illustrative. Other configurations and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those illustrated, and some elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. In some embodiments, the systems, methods, and processes described herein may be performed using similar components, features, and / or functionalities in the exemplary autonomous vehicle 700 of Figures 7A–7D, the exemplary computing device 800 of Figure 8, and / or the exemplary data center 900 of Figure 9.

[0050] During deployment, process 400 may include generating and / or receiving sensor data 402. Sensor data 402 may include sensor data generated using one or more sensors of the ego machine—for example, the ego machine 700 described herein with respect to Figures 7A-7D. For example, sensor data 402 may be similar to sensor data 102A described with respect to Figure 1. As such, sensor data 402 may represent the field of view and / or perceptual field of one or more sensors of the ego machine 700—for example, in the form of images, LiDAR range images, point clouds, etc.

[0051] The machine learning model 104 can receive sensor data 402 as input and process the sensor data 402 to compute an output 106, which may include a viewing distance 108, a distance bin 110, and / or a visibility classification / attribute 112. In embodiments in which the machine learning model 104 computes the viewing distance 108 (rather than the distance bin 110), a post-processor 404 may be used to threshold the viewing distance 108 to a distance bin. For example, if the distance bin is between 10 meters and 100 meters and the viewing distance 108 is 78 meters, the post-processor 404 may label or classify instances of the sensor data 402 as corresponding to a distance bin between 10 meters and 100 meters, and this information may be analyzed for usefulness 406.

[0052] In any given instance, whether the distance bin 110 is directly calculated as the output 106 of the machine learning model 104 or determined using the post-processor 404, there may be any number of distance bins, and each distance bin may contain any range of view distance values. In addition, each bin may correspond to a different number of operations that can be performed. For example, if the sensor data 402 is not visually impaired in any way, there may be a set of operations on which the ego machine 700 can rely on the sensor data 402. However, if the sensor data 402 is visually impaired in any way (e.g., the view distance is less than the maximum), an indicator may be provided that disables one or more operations from that set of operations, or causes one or more instances of the sensor data 402 that rely on the sensor data 402 to be ignored because they are unsuitable or unusable for the functioning of the sensor data 402. In addition, in the embodiment, when the visibility classification / attribute 112 is calculated, this information may be used as an additional factor in the usefulness determination 406 so that a particular visibility classification combined with a particular distance bin may correspond to a different usefulness than other visibility classifications combined with other distance bins.

[0053] As a non-restrictive example, distance bins may include a first bin for very low visibility (e.g., <10 meters), a second bin for low visibility (e.g., between 10 meters and 100 meters), a third bin for moderate visibility (e.g., between 100 meters and 1000 meters), a fourth bin for high visibility (e.g., between 1000 meters and 4000 meters), and a fifth bin for clear visibility (e.g., above 4000 meters). In this example, the first bin may correspond to a visual range correlated with extreme fog or blizzard, or high rainfall or heavy drizzle. The second bin may correspond to a visual range correlated with dense fog weather. The third bin may correspond to a visual range correlated with moderate fog weather or moderate rainfall or moderate drizzle. The fourth bin may correspond to a visual range correlated with hazy weather conditions or light rainfall or drizzle. The fifth bin may correspond to a visual range that correlates with the maximum visual range of the particular sensor generating sensor data 402.

[0054] Continuing with this non-limiting example, each of the five bins could correspond to a variety of tasks that the ego machine could perform. For example, for the first bin (e.g., very low visibility distance), as long as the ego machine is moving below a threshold speed (e.g., less than 25 kilometers per hour), the full capabilities of Level 0, Level 1, and Level 2 low-speed active safety functions (e.g., lane assist, automatic lane keeping, automatic cruise control, automatic emergency braking, etc.) may be available. However, if the vehicle is moving above the threshold speed, these functions may be disabled (at least with respect to instances of sensor data 402 with very low invalid distance). For the second bin (e.g., low visibility distance), as long as the ego machine is moving below a threshold speed, the full capabilities of Level 0, Level 1, and Level 2 parking functions (e.g., proximity detection, automatic parallel parking alignment, etc.) and low-speed active safety functions may be available. With respect to the third bin (e.g., medium visibility), the full performance of Level 0, Level 1, Level 2, and Level 2+ driving functions may be available. With respect to the fourth bin (e.g., high visibility), the full performance of Level 3 and Level 4 parking functions (e.g., automatic parallel parking (APP), MPP, VVP) may be available in addition to the Level 0, Level 1, Level 2, and Level 2+ functions of the first, second, and third bins. With respect to the fourth bin (e.g., clear visibility), the full performance of Level 3 autonomous highway driving functions may be available in addition to the functionality of the first, second, third, and fourth bins.

[0055] In some embodiments, utility 406 may correspond to a portion of the information from sensor data 402 that can be used. For example, if distance bins are known and the distance bins correspond to a range of 100 to 1000 meters, any output of the system within 1000 meters of the egomachine 700 corresponding to sensor data 402 may be used, while any output corresponding to distances greater than 1000 meters may be ignored. In such an example, if a first vehicle is detected at a distance of 200 meters from the egomachine 700 using an object detection algorithm, the detection may be relied upon by the downstream system of the drive stack 408. However, if a second vehicle is detected at a distance of 1200 meters from the egomachine 700 using an object detection algorithm, the detection may not be relied upon. A similar process may be used for other tasks, such as object tracking, obstacle in path analysis (OIPA), object-to-lane assignment, and localization.

[0056] In any embodiment, after usefulness 406 has been determined, instances of sensor data 402 may be tagged with the usefulness 406 instructions or transmitted to an autonomous or semi-autonomous software driving stack ("Drive Stack") so that one or more features, functionalities, and / or components of the drive stack 408 may be disabled, instances of sensor data 402 may be ignored, and / or other uses of usefulness 406 may be made. The drive stack 408 may include one or more layers, for example, a world state manager layer that manages the world state using one or more maps (e.g., 3D maps), localization components, perception components, and / or similar. In addition, the autonomous driving software stack may include planning components (e.g., as part of a planning layer), control components (e.g., as part of a control layer), actuator components (e.g., as part of an actuator layer), obstacle avoidance components (e.g., as part of an obstacle avoidance layer), and / or other components (e.g., as part of one or more additional or substitute layers). As such, utility 406 may provide instructions to any downstream task of the drive stack 408 that relies on the sensor data 402, so that the appropriate use of the sensor data 402 is managed.

[0057] As an example, with respect to the visualization 500 in Figure 5, an instance of sensor data 402 representing an image may be generated using the camera of the ego machine 700. The instance of sensor data 402 may be applied to a machine learning model 104, which may compute one or more of the outputs 106. If the output 106 includes a viewing distance 108, the post-processor 404 may threshold the viewing distance 108 into a distance bin. The usefulness 406 may then be determined based on the distance bin (and / or visibility classification / attribute 112), and this information may be used by the drive stack 408. For example, if the distance bin corresponds to very low visibility, a safety feature such as automatic emergency braking (AEB) may be used to help the ego machine 700 stop for vehicle 502A, but the object detection result corresponding to vehicle 502B may not be relied upon due to the reduced usefulness 406 of the sensor data 402 corresponding to the visualization 500.

[0058] Referring here to Figure 6, each block of the Method 600 described herein includes a computation process that can be executed using any combination of hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory. Method 600 may also be performed as computer-usable instructions stored on a computer storage medium. Method 600 may be provided, to name a few, as a standalone application, a service or a hosted service (standalone or in combination with another hosted service), or as a plug-in to another product. In addition, Method 600 is described as an example with respect to Process 400 in Figure 4 and the Ego Machine 700 in Figures 7A-7D. However, Method 600 may be performed additionally or instead within any single process and / or in any single system, or any combination of processes and systems, including but not limited to those described herein.

[0059] Figure 6 is a flowchart illustrating a method 600 for deploying a machine learning model to calculate an estimated sight distance, according to some embodiments of the present disclosure. In block B602, the method 600 includes using a machine learning model to calculate data indicating the sight distance corresponding to sensor data, at least in part on sensor data generated using one or more sensors of an ego machine. For example, a machine learning model 104 may use sensor data 402 as input to calculate one or more of the outputs 106.

[0060] Method 600 includes determining the usefulness of sensor data for one or more operations of an ego machine, at least in part, based on the viewing distance, in block B604. For example, the usefulness 406 of sensor data 402 may be determined based on the viewing distance 108, the distance bin 110, and / or the visibility classification / attribute 112.

[0061] Method 600 includes, in block B606, performing at least one of one or more actions based at least in part on the usefulness of the sensor data. For example, if the viewing distance is unclear, one or more actions that would have relied on the sensor data 402 if the sensor data had been clear may be deactivated and / or signaled to ignore certain instances of the sensor data 402.

[0062] (Example autonomous vehicle) Figure 7A shows an exemplary autonomous vehicle 700 according to some embodiments of the present disclosure. The autonomous vehicle 700 (or referred to herein as "vehicle 700") may include, but is not limited to, passenger vehicles such as cars, trucks, buses, first responder vehicles, shuttle buses, electric or motorized bicycles, motorcycles, fire engines, police vehicles, ambulances, boats, construction vehicles, submarines, drones, trailer-mounted vehicles, and / or other types of vehicles (e.g., unmanned and / or carrying one or more passengers). Autonomous vehicles are generally described in terms of automation levels as defined by the National Highway Traffic Safety Administration (NHTSA), departments of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicle" (standard number J3016-201806, published June 15, 2018; standard number J3016-201609, published September 30, 2016; and previous and future versions of this standard). Vehicle 700 may have functions at one or more levels of autonomous driving from Level 3 to Level 5. Vehicle 700 may have functions at one or more levels of autonomous driving from Level 1 to Level 5. For example, depending on the embodiment, the vehicle 700 may have driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5). In this specification, the term “autonomous” may include any and / or all types of autonomy of the vehicle 700 or any other machine, such as being fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing auxiliary autonomy, semi-autonomous, primarily autonomous, or other designations.

[0063] Vehicle 700 may include components such as the vehicle's chassis, body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other components. Vehicle 700 may include a propulsion system 750, such as an internal combustion engine, a hybrid power unit, a fully electric engine, and / or another propulsion system type. The propulsion system 750 may include a transmission and be connected to the drivetrain of vehicle 700 to enable the propulsion of vehicle 700. The propulsion system 750 may be controlled in response to receiving signals from a throttle / accelerator 752.

[0064] A steering system 754, which may include a steering wheel, may be used to steer the vehicle 700 (for example, along a desired course or route) when the propulsion system 750 is operating (for example, when the vehicle is moving). The steering system 754 may receive signals from the steering actuator 756. The steering wheel may be optional for fully automated (level 5) functionality.

[0065] The brake sensor system 746 may be used to operate the vehicle brakes in response to receiving signals from the brake actuator 748 and / or the brake sensor.

[0066] The controller 736, which may include one or more System-on-Chip (SoC) 704 (Figure 7C) and / or GPUs, can provide signals (e.g., expressions of commands) to one or more components and / or systems of the vehicle 700. For example, the controller can send signals to actuate the vehicle brakes via one or more brake actuators 748, actuate the steering system 754 via one or more steering actuators 756, and actuate the propulsion system 750 via one or more throttle / accelerators 752. The controller 736 may include one or more onboard (e.g., integrated) computing devices (e.g., supercomputers) that process sensor signals and output operational commands (e.g., signals representing commands) to enable autonomous driving and / or assist the driver in driving the vehicle 700. The controller 736 may include a first controller 736 for autonomous driving functions, a second controller 736 for functional safety functions, a third controller 736 for artificial intelligence functions (e.g., computer vision), a fourth controller 736 for infotainment functions, a fifth controller 736 for redundancy in emergency situations, and / or other controllers. In some examples, a single controller 736 may handle two or more of the aforementioned functions, and two or more controllers 736 may handle a single function and / or any combination thereof.

[0067] The controller 736 can provide signals for controlling one or more components and / or systems of the vehicle 700 in response to sensor data (e.g., sensor inputs) received from one or more sensors. Sensor data may be received from, for example, but not limited to, global navigation satellite system sensors 758 (e.g., global positioning system sensors), RADAR sensors 760, ultrasonic sensors 762, LIDAR sensors 764, inertial measurement unit (IMU) sensors 766 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 796, stereo cameras 768, wide-view cameras 770 (e.g., fisheye cameras), infrared cameras 772, surround cameras 774 (e.g., 360-degree cameras), long-range and / or medium-range cameras 798, speed sensors 744 (e.g., for measuring the speed of a vehicle 700), vibration sensors 742, steering sensors 740, brake sensors (e.g., as part of a brake sensor system 746), and / or other sensor types.

[0068] One or more of the controllers 736 may receive inputs (represented, for example, by input data) from the instrument cluster 732 of the vehicle 700 and provide outputs (represented, for example, by output data, display data, etc.) via a human-machine interface (HMI) display 734, an audible annunciator, a loudspeaker, and / or other components of the vehicle 700. The outputs may include information such as vehicle speed, speed, time, map data (e.g., HD map 722 in Figure 7C), location data (e.g., the location of the vehicle 700, such as on a map), direction, the location of other vehicles (e.g., occupied grid), and information about objects and the status of objects as perceived by the controller 736. For example, the HMI display 734 may display information regarding the presence of one or more objects (e.g., road signs, warning signs, changes in traffic signals, etc.) and / or information regarding driving operations that the vehicle has performed, is performing, or will perform (e.g., changing lanes now, exiting exit 34B within 3.22 km (2 miles), etc.).

[0069] Vehicle 700 further includes a network interface 724 that can communicate over one or more networks using one or more wireless antennas 726 and / or a modem. For example, the network interface 724 may have communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, etc. The wireless antennas 726 can also enable communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area networks such as Bluetooth®, Bluetooth® LE, Z-Wave, ZigBee, and / or low-power wide-area networks (LPWAN) such as LoRaWAN, SigFox.

[0070] Figure 7B shows examples of camera positions and fields of view of the exemplary autonomous vehicle 700 of Figure 7A according to several embodiments of the present disclosure. The cameras and their respective fields of view are exemplary embodiments and are not intended to limit the scope. For example, additional and / or alternative cameras may be included, and / or cameras may be placed in different locations on the vehicle 700.

[0071] The camera type may include, but is not limited to, a digital camera that can be adapted for use with the components and / or systems of the vehicle 700. The camera may operate at Automotive Safety Integrity Level (ASIL) B and / or other ASILs. Depending on the embodiment, the camera type may have any image capture rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The camera may have the ability to use a roll shutter, a global shutter, another type of shutter, or a combination thereof. In some examples, the color filter array may include an RCCC (red clear clear clear) color filter array, an RCCB (red clear clear blue) color filter array, an RBGC (red blue green clear) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear pixel cameras, such as cameras having RCCC, RCCB, and / or RBGC color filter arrays, may be used in efforts to increase light sensitivity.

[0072] In some applications, one or more cameras may be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe design). For example, a multi-function mono-camera may be installed to provide functions including lane departure warning, traffic sign assist, and intelligent headlamp control. One or more cameras (e.g., all cameras) may simultaneously record and provide image data (e.g., video).

[0073] One or more of the cameras may be mounted in custom-designed (3D-printed) mounting parts to eliminate stray light and reflections from inside the vehicle (e.g., reflections from the dashboard reflected in the windshield mirror) that may interfere with the camera's image data capture capability. Referring to side mirror mounting parts, the side mirror parts may be custom 3D-printed so that the camera mounting plate conforms to the shape of the side mirror. In some examples, the camera may be integrated within the side mirror. For side-view cameras, the camera may also be integrated within four struts located at each corner of the cabin.

[0074] A camera having a field of view that includes a portion of the environment in front of the vehicle 700 (e.g., a forward-facing camera) may be used for surround view, which helps identify the forward path and obstacles and, with the help of one or more controllers 736 and / or control SoCs, helps provide information essential for generating an occupied grid and / or determining a preferred vehicle path. The forward-facing camera may also be used to perform many of the same ADAS functions as LIDAR, including emergency braking, pedestrian detection, and collision avoidance. The forward-facing camera may also be used for ADAS functions and systems, including other functions such as lane departure warning (LDW), autonomous cruise control (ACC), and / or traffic sign recognition.

[0075] Various cameras may be used in forward-facing configurations, including, for example, a monocular camera platform that includes a CMOS (complementary metal oxide semiconductor) color imaging device. Another example may be a wide-view camera 770, which can be used to capture objects that come into view from the surroundings (e.g., pedestrians, intersecting traffic, or bicycles). Although only one wide-view camera is shown in Figure 7B, any number of wide-view cameras 770 may be present in the vehicle 700. In addition, long-range cameras 798 (e.g., a long-view stereo camera pair) may be used for depth-based object detection, particularly for objects for which the neural network has not yet been trained. Long-range cameras 798 may also be used for object detection and classification, as well as basic object tracking.

[0076] One or more stereo cameras 768 may also be included in a forward-facing configuration. The stereo camera 768 may include an integrated control unit with an expandable processing unit that may provide a programmable logic (FPGA) and a multi-core microprocessor with an integrated CAN or Ethernet® interface on a single chip. Such a unit may be used to generate a 3D map of the vehicle's environment, including distance estimates for all points in the image. An alternative stereo camera 768 may include a compact stereo vision sensor comprising two camera lenses (one on the left and one on the right) and an image processing chip capable of measuring the distance from the vehicle to an object and activating autonomous emergency braking and lane departure warning functions using the generated information (e.g., metadata). Other types of stereo cameras 768 may be used in addition to or instead of those described herein.

[0077] A camera having a field of view including a portion of the environment on the sides of the vehicle 700 (e.g., a side-view camera) may be used for surround view, providing information used to create and update the occupancy grid and generate side impact collision warnings. For example, surround cameras 774 (e.g., four surround cameras 774 as shown in Figure 7B) may be positioned on the vehicle 700. The surround cameras 774 may include wide-view cameras 770, fisheye cameras, 360-degree cameras, and / or similar. For example, four fisheye cameras may be positioned in front of, behind, and on the sides of the vehicle. In an alternative configuration, the vehicle may use three surround cameras 774 (e.g., left, right, and rear) and utilize one or more other cameras (e.g., forward-facing cameras) as a fourth surround view camera.

[0078] A camera having a field of view that includes a portion of the environment behind the vehicle 700 (e.g., a rear-view camera) may be used for parking assistance, surround view, rear collision warning, and creation and updating of the occupancy grid. A wide variety of cameras may be used, including, but not limited to, cameras also suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 798, stereo cameras 768, infrared cameras 772, etc.) as described herein.

[0079] Figure 7C is a block diagram of an exemplary system architecture of the exemplary autonomous vehicle 700 of Figure 7A, according to some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, functions, sequences, groupings of functions, etc.) may be used in addition to or instead of those shown, and some elements may be omitted collectively. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and in any appropriate combination and location. Various functions described herein as being performed by entities may be performed by hardware, firmware, and / or software. For example, various functions may be performed by a processor that executes instructions stored in memory.

[0080] Each component, feature, and system of the vehicle 700 in Figure 7C is illustrated as being connected via a bus 702. Bus 702 may include a Controller Area Network (CAN) data interface (or referred to as the "CAN bus"). CAN may also be a network within the vehicle 700 used to help control various features and functions of the vehicle 700, such as the operation of brakes, acceleration, braking, steering, and windshield wipers. The CAN bus may be configured to have dozens or hundreds of nodes, each having its own unique identifier (e.g., CAN ID). The CAN bus may be read to find steering angle, ground speed, engine revolutions per minute (RPM), button position, and / or other vehicle condition indicators. The CAN bus may be ASIL B compliant.

[0081] Bus 702 is described herein as a CAN bus, but this is not intended to limit it. For example, FlexRay and / or Ethernet® may be used in addition to, or instead of, a CAN bus. In addition, a single line is used to represent bus 702, but this is not intended to limit it. There may be any number of buses 702, which may include, for example, one or more CAN buses, one or more FlexRay buses, one or more Ethernet® buses, and / or one or more other types of buses using different protocols. In some examples, two or more buses 702 may be used to perform different functions and / or for redundancy. For example, a first bus 702 may be used for collision avoidance functions, and a second bus 702 may be used for operation control. In any example, each bus 702 may communicate with any of the components of the vehicle 700, and two or more buses 702 may communicate with the same component. In some examples, each SoC 704, each controller 736, and / or each computer within the vehicle may have access to the same input data (e.g., input from sensors in the vehicle 700) and may be connected to a common bus such as a CAN bus.

[0082] The vehicle 700 may include one or more controllers 736, such as those described herein with respect to Figure 7A. The controllers 736 may be used for a variety of functions. The controllers 736 may be connected to any of the various other components and systems of the vehicle 700 and may be used for the control of the vehicle 700, the artificial intelligence of the vehicle 700, infotainment for the vehicle 700, and / or the like.

[0083] Vehicle 700 may include a system-on-a-chip (SoC) 704. The SoC 704 may include a CPU 706, a GPU 708, a processor 710, a cache 712, an accelerator 714, a data store 716, and / or other components and features not shown. The SoC 704 may be used to control vehicle 700 in various platforms and systems. For example, the SoC 704 may be coupled in a system (e.g., a system of vehicle 700) that has an HD map 722 that can obtain map refreshes and / or updates via a network interface 724 from one or more servers (e.g., server 778 in Figure 7D).

[0084] The CPU 706 may include a CPU cluster or CPU complex (or, as referred to herein, "CCPLEX"). The CPU 706 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU 706 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU 706 may include four dual-core clusters, each cluster having its own dedicated L2 cache (e.g., 2MBL2 cache). The CPU 706 (e.g., CCPLEX) may be configured to support concurrent cluster operation, which allows any combination of the clusters of the CPU 706 to be active at a given time.

[0085] The CPU706 can implement power management capabilities that include one or more of the following features: individual hardware blocks may be automatically clock-gated when idle to conserve dynamic power; each core clock may be gated when a core is not actively executing instructions by executing WFI / WFE instructions; each core may be independently power-gated; each core cluster may be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster may be independently power-gated when all cores are power-gated. The CPU706 can further implement enhanced algorithms for managing power states, where acceptable power states and expected wake-up times are specified, and the hardware / microcode determines the best power state to input to the cores, clusters, and CCPLEX. The processing core may support a simplified power state input sequence in software where the work is offloaded to the microcode.

[0086] The GPU708 may include an integrated GPU (or, as referred to herein, "iGPU"). The GPU708 may be programmable and efficient for parallel workloads. In some embodiments, the GPU708 may be able to use an enhanced tensor instruction set. The GPU708 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a storage capacity of at least 96KB), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512KB). In some embodiments, the GPU708 may include at least eight streaming microprocessors. The GPU708 may be able to use a Computation Application Programming Interface (API). In addition, the GPU708 may be able to use one or more parallel computing platforms and / or programming models (e.g., NVIDIA® CUDA).

[0087] The GPU708 can be power-optimized for best performance in automotive and embedded use cases. For example, the GPU708 can be manufactured on a FinFET (Fin field-effect transistor). However, this is not intended to be an limitation, and the GPU708 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can incorporate several mixed-precision processing cores divided into multiple blocks. Not limited to, for example, 64 PF32 cores and 32 PF64 cores may be divided into four processing blocks. In such an example, each processing block may be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, 2 mixed-precision NVIDIA TENSOR COREs for deep learning matrix operations, an L0 instruction cache, a warp scheduler, a dispatch unit, and / or a 64KB register file. In addition, the streaming microprocessor may include independent parallel integer and floating-point data paths to provide efficient execution of workloads with a mixture of computation and addressing operations. A streaming microprocessor may include independent thread scheduling capabilities to enable finer-grained synchronization and coordination between concurrent threads. A streaming microprocessor may also include a combined L1 data cache and shared memory unit to simplify programming while improving performance.

[0088] In some examples, the GPU708 may include high-bandwidth memory (HBM) and / or a 16GB HBM2 memory subsystem to provide a peak memory bandwidth of 900 GB / s. In some examples, in addition to or instead of HBM memory, synchronous graphics random-access memory (SGRAM), such as graphics double data rate type five synchronous random-access memory (GDDR5), may be used.

[0089] The GPU708 can incorporate unified memory technology, including access counters, to enable more precise movement of memory pages to the processor that most frequently accesses them, thereby improving the efficiency of shared memory ranges between processors. In some examples, address translation service (ATS) support may be used to allow the GPU708 to directly access the CPU706 page table. In such examples, when the GPU708 memory management unit (MMU) experiences a miss, an address translation request may be sent to the CPU706. In response, the CPU706 can look up its page table for virtual-to-real-address mapping and send the translation back to the GPU708. As such, unified memory technology can enable a single, unified virtual address space for both the CPU706 and GPU708 memory, thereby simplifying GPU708 programming and porting of applications to the GPU708.

[0090] In addition, the GPU708 may include an access counter that can record how often the GPU708 accesses the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses that page most frequently.

[0091] The SoC704 may include any number of caches 712, including those described herein. For example, the cache 712 may include an L3 cache available to both the CPU 706 and the GPU 708 (e.g., connected to both the CPU 706 and the GPU 708). The cache 712 may include a write-back cache that can record line states, for example, by using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache may include 4 MB or more, depending on the embodiment, although a smaller cache size may be used.

[0092] The SoC704 may include an arithmetic logic unit (ALU) that can be used to perform processing for any of the various tasks or operations of the vehicle 700 (for example, a processing DNN). In addition, the SoC704 may include a floating-point unit (FPU) (or other mass coprocessor or numerical coprocessor type) for performing mathematical operations within the system. For example, the SoC704 may include one or more FPUs integrated as execution units within the CPU706 and / or GPU708.

[0093] The SoC704 may include one or more accelerators 714 (e.g., a hardware accelerator, a software accelerator, or a combination thereof). For example, the SoC704 may include a hardware acceleration cluster that may include an optimized hardware accelerator and / or a large on-chip memory. The large on-chip memory (e.g., 4MB of SRAM) may enable the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster may be used to complement the GPU708 and to offload some of the GPU708's tasks (e.g., to free up more cycles of the GPU708 to perform other tasks). As an example, accelerator 714 may be used for target workloads that are sufficiently stable to be suitable for acceleration (e.g., perception, convolutional neural networks (CNNs), etc.). In this specification, the term "CNN" may include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (for example, as used for object detection).

[0094] The accelerator 714 (e.g., a hardware acceleration cluster) may include a deep learning accelerator (DLA). The DLA may include one or more tensor processing units (TPUs) that can be configured to provide an additional 10 trillion operations per second for deep learning applications and inference. The TPU may also be an accelerator configured and optimized to perform image processing functions (e.g., CNN, RCNN, etc.). The DLA may further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA can provide more performance per millisecond than a general-purpose GPU and significantly exceed the performance of a CPU. The TPU can perform several functions, including, for example, single-instance convolutional functions supporting INT8, INT16, and FP16 data types for both features and weights, as well as post-processing functions.

[0095] DLA can rapidly and efficiently run neural networks, particularly CNNs, on processed or unprocessed data for any of a variety of functions, including, but not limited to, the following: CNNs for object recognition and detection using data from camera sensors; CNNs for distance estimation using data from camera sensors; CNNs for emergency vehicle detection, identification, and detection using data from microphones; CNNs for facial recognition and vehicle owner identification using data from camera sensors; and / or CNNs for security and / or safety-related events.

[0096] DLA can perform any function of GPU708, and by using inference accelerators, for example, a designer can target either DLA or GPU708 for any function. For example, a designer can focus on CNN and floating-point arithmetic processing on DLA, and leave other functions to GPU708 and / or other accelerators 714.

[0097] The accelerator 714 (for example, a hardware accelerator cluster) may include a programmable vision accelerator (PVA), which may be referred to herein as a computer vision accelerator. A PVA may be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving, and / or augmented reality (AR) and / or virtual reality (VR) applications. A PVA can provide a balance between performance and flexibility. For example, each PVA may, but is not limited to, any number of reduced instruction set computer (RISC) cores, direct memory access (DMA), and / or any number of vector processors.

[0098] A RISC core can interact with an image sensor (for example, the image sensor of one of the cameras described herein), an image signal processor, and / or similar devices. Each RISC core may include any amount of memory. Depending on the embodiment, a RISC core may use one of several protocols. In some examples, a RISC core can run a real-time operating system (RTOS). A RISC core may be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, a RISC core may include an instruction cache and / or tightly coupled RAM.

[0099] DMA can enable PVA components to access system memory independent of the CPU 706. DMA can support any number of features used to bring optimizations to the PVA, including but not limited to supporting multidimensional addressing and / or circular addressing. In some examples, DMA can support up to six or more dimensions of addressing, which may include block width, block height, block depth, horizontal block stepping, vertical block stepping, and / or depth stepping.

[0100] A vector processor may also be a programmable processor that can be designed to efficiently and flexibly execute the programming of computer vision algorithms and provide signal processing capabilities. In some examples, a PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, a DMA engine (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem can act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). The VPU core may include a digital signal processor, such as a single-instruction, multiple-data (SIMD), or very-long instruction word (VLIW) digital signal processor. A combination of SIMD and VLIW can increase throughput and speed.

[0101] Each vector processor may include an instruction cache and be linked to dedicated memory. As a result, in some examples, each vector processor may be configured to run independently of other vector processors. In other examples, the vector processors included in a particular PVA may be configured to use data parallelism. For example, in some embodiments, multiple vector processors included in a single PVA may execute the same computer vision algorithm, but on different regions of an image. In other examples, the vector processors included in a particular PVA may execute different computer vision algorithms simultaneously on the same image, or even execute different algorithms sequentially on the image or parts of an image. In particular, any number of PVAs may be included in a hardware acceleration cluster, and any number of vector processors may be included in each PVA. In addition, a PVA may include additional error correction code (ECC) memory to enhance overall system safety.

[0102] The accelerator 714 (for example, a hardware accelerator cluster) may include a computer vision network on-chip and SRAM to provide high-bandwidth, low-latency SRAM for the accelerator 714. In some examples, the on-chip memory may include at least 4 MB of SRAM consisting of eight field-configurable memory blocks, which may be accessible by both the PVA and DLA, for example, but not limited to. Each pair of memory blocks may include an advanced peripheral bus (APB) interface, configuration circuitry, a controller, and a multiplexer. Any type of memory may be used. The PVA and DLA can access the memory via a backbone that provides the PVA and DLA with high-speed access to the memory. The backbone may include a computer vision network on-chip that interconnects the PVA and DLA to the memory (for example, using an APB).

[0103] A computer vision network on-chip may include an interface that determines whether both the PVA and DLA are activatable and enable signals before any control signals / addresses / data are transmitted. Such an interface can provide separate phases and separate channels for transmitting control signals / addresses / data, as well as burst-type communication for continuous data transfer. This type of interface may conform to ISO 26262 or IEC 61508 standards, but other standards and protocols may be used.

[0104] In some embodiments, the SoC704 may include a real-time ray tracing hardware accelerator, such as the one described in Patent Document 1, filed August 10, 2018. The real-time ray tracing hardware accelerator may be used to quickly and efficiently determine the location and size of objects (e.g., in a world model) to generate real-time visualization simulations for RADAR signal interpretation, acoustic propagation synthesis and / or analysis, SONAR system simulation, general wave propagation simulation, comparison to LIDAR data for localization and / or other functions, and / or other uses. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray tracing-related operations.

[0105] The accelerator 714 (e.g., a hardware accelerator cluster) has diverse applications for autonomous driving. The PVA may also be a programmable vision accelerator that can be used in critical processing stages in ADAS and autonomous vehicles. The PVA's capabilities are suitable for areas of algorithms requiring predictable processing at low power and low latency. In other words, the PVA performs well in semi-high density or high density typical computations, even on small data sets, where predictable execution time is required along with low latency and low power. Therefore, because the PVA is efficient in object detection and integer computation, in relation to a platform for autonomous vehicles, the PVA is designed to run classic computer vision algorithms.

[0106] For example, according to one embodiment of this technology, PVA is used to perform computer stereo vision. Semi-global matching-based algorithms may be used in some examples, but this is not intended to be limiting. Numerous applications for Level 3-5 autonomous driving require motion estimation / stereo matching on the fly (e.g., SFM (structure from motion), pedestrian recognition, lane detection, etc.). PVA can perform computer stereo vision functions with input from two monocular cameras.

[0107] In some applications, PVA can be used to perform high-density optical flow, providing processed RADAR data through processing of raw RADAR data (e.g., using 4D Fast Fourier Transform). In other applications, PVA is used for time-of-flight depth processing, for example, by processing raw time-of-flight data to provide processed time-of-flight data.

[0108] DLA can be used to run any type of network to enhance control and driving safety, for example, a neural network that outputs a confidence value for each object detection. Such confidence values ​​can be interpreted as probabilities or as providing the relative "weight" of each detection compared to other detections. This confidence value allows the system to make other decisions about which detections should be considered true positives rather than false positives. For example, the system can set a confidence threshold and consider only detections that exceed the threshold as true positives. In an automatic emergency braking (AEB) system, a false positive detection would cause the vehicle to automatically apply the emergency brakes, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for the AEB. DLA can run a neural network that devolves the confidence values. The neural network can accept at least a subset of parameters as its input, such as bounding box dimensions, ground plane estimation acquired (e.g., from another subsystem), vehicle orientation, distance, inertial measurement unit (IMU) sensor output correlated with 3D position estimation of an object acquired from the neural network and / or other sensors (e.g., LIDAR sensor 764 or RADAR sensor 760), and others.

[0109] The SoC704 may include a data store 716 (for example, memory). The data store 716 may also be the on-chip memory of the SoC704 and can store neural networks that will run on the GPU and / or DLA. In some examples, the data store 716 may have a capacity large enough to store multiple instances of the neural network for redundancy and safety. The data store 712 may comprise an L2 or L3 cache 712. References to the data store 716 may include references to memory associated with the PVA, DLA, and / or other accelerators 714, as described herein.

[0110] The SoC704 may include one or more processors 710 (e.g., integrated processors). The processors 710 may include a boot and power management processor, which may be a dedicated processor and subsystem for handling boot power and management capabilities and associated security enforcement. The boot and power management processor may also be part of the SoC704 boot sequence and can provide runtime power management services. The boot power and management processor can provide clock and voltage programming, assistance with system low-power state transitions, management of SoC704 thermal and temperature sensors, and / or management of SoC704 power states. Each temperature sensor may be implemented as a ring oscillator whose output frequency is proportional to temperature, and the SoC704 may use the ring oscillators to detect the temperatures of the CPU 706, GPU 708, and / or accelerator 714. If the temperature is determined to have exceeded a threshold, the boot and power management processor may enter a temperature fault routine, placing the SoC704 into a lower power state and / or putting the vehicle 700 into a substitute driving mode for safe shutdown (for example, safely shutting down the vehicle 700).

[0111] The processor 710 may further include a set of integrated processors that can perform the functions of an audio processing engine. The audio processing engine may also be an audio subsystem that enables full hardware support for multi-channel audio through multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core having a digital signal processor with dedicated RAM.

[0112] The processor 710 may further include an always-on processor engine that can provide the necessary hardware features to support low-power sensor management and wake use cases. The always-on processor engine may include a processor core, tightly coupled RAM, support peripherals (e.g., timer and interrupt controllers), various I / O controller peripherals, and routing logic.

[0113] The processor 710 may further include a safety cluster engine, which includes a dedicated processor subsystem for handling safety management in automotive applications. The safety cluster engine may include two or more processor cores, tightly coupled RAM, supporting peripherals (e.g., timers, interrupt controllers, etc.), and / or routing logic. In safety mode, the two or more cores may operate in lockstep mode and function as a single core with comparison logic to detect any differences between their operations.

[0114] The processor 710 may further include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0115] The processor 710 may further include a high dynamic range signal processor, which may include an image signal processor, a hardware engine that is part of the camera processing pipeline.

[0116] The processor 710 may include a video image synthesizer, which may also be a processing block (for example, implemented on a microprocessor) that implements post-video processing functions required by the video playback application to produce the final image for the player window. The video image synthesizer can perform lens distortion correction on the wide-view camera 770, the surround camera 774, and / or the in-cabin surveillance camera sensors. The in-cabin surveillance camera sensors are preferably monitored by a neural network running on another instance of the advanced SoC, configured to identify and appropriately respond to in-cabin events. The in-cabin system can perform lip-reading to activate cellular services and make phone calls, transcribe emails, change the vehicle's destination, activate or change the vehicle's infotainment system and settings, or provide voice-activated web surfing. Certain functions are available to the driver only when operating in autonomous mode and are otherwise disabled.

[0117] A video image synthesizer may include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in the video, noise reduction reduces the weight of information provided by adjacent frames and appropriately weights the spatial information. If the image or part of the image does not contain motion, the temporal noise reduction performed by the video image synthesizer can use information from previous images to reduce noise in the current image.

[0118] The video image synthesizer can also be configured to perform stereo rectification on the input stereo lens frame. Furthermore, the video image synthesizer can be used for user interface compositing when the operating system desktop is in use, so that the GPU708 is not required to continuously render new faces. Even when the GPU708 is powered on and actively performing 3D rendering, the video image synthesizer can be used to offload the GPU708 to improve performance and responsiveness.

[0119] The SoC704 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions to receive video and input from a camera. The SoC704 may further include an input / output controller that can be controlled by software and can be used to receive I / O signals that are not committed to a specific role.

[0120] The SoC704 may further include a wide range of peripheral interfaces to enable communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC704 may be used to process data from cameras (connected, for example, via Gigabit Multimedia Serial Link and Ethernet®), sensors (e.g., Lidar sensor 764, Radar sensor 760, etc., which may be connected via Ethernet®), data from bus 702 (e.g., vehicle speed, steering wheel position, etc.), and data from GNSS sensor 758 (connected, for example, via Ethernet® or CAN bus). The SoC704 may further include a dedicated high-performance mass storage controller, which may include its own DMA engine and may be used to free up CPU 706 from routine data management tasks.

[0121] The SoC704 may also be an inter-terminal platform with a flexible architecture that spans automation levels 3-5, thereby providing a comprehensive functional safety architecture that leverages and efficiently uses computer vision and ADAS techniques for diversity and redundancy, and, together with deep learning tools, provides a platform for a flexible, reliable driving software stack. The SoC704 can be faster, more reliable, more energy-efficient, and more space-efficient than conventional systems. For example, when the accelerator 714 is coupled with the CPU 706, the GPU 708, and the data store 716 can provide a fast and efficient platform for autonomous vehicles at levels 3-5.

[0122] Therefore, this technology brings capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be executed on a CPU, which can be configured using high-level programming languages ​​such as the C programming language to execute a wide variety of processing algorithms across a wide variety of visual data. However, CPUs often cannot meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. Specifically, many CPUs cannot execute real-time complex object detection algorithms, which are required for in-vehicle ADAS applications and actual Level 3-5 autonomous vehicles.

[0123] In contrast to conventional systems, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, the technologies described herein enable multiple neural networks to run simultaneously and / or sequentially, and the results to be combined to enable Level 3–5 autonomous driving capabilities. For example, a DLA or a CNN running on a dGPU (e.g., GPU720) may include text and word recognition, enabling a supercomputer to read and understand traffic signs, including signs for which the neural network has not been specifically trained. The DLA may further include a neural network capable of identifying, interpreting, and providing a semantic understanding of signs, and passing that semantic understanding to a route planning module running on the CPU complex.

[0124] As another example, multiple neural networks may run simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of a flashing light and the text "Caution: Flashing light indicates frozen conditions" may be interpreted independently or collectively by several neural networks. The sign itself may be identified as a traffic sign by a first deployed neural network (e.g., a trained neural network), and the text "Flashing light indicates frozen conditions" may be interpreted by a second deployed neural network, informing the vehicle's route planning software (preferably running on a CPU complex) that frozen conditions are present when flashing light is detected. The flashing light may be identified by informing the vehicle's route planning software of the presence (or absence) of the flashing light, and by operating a third deployed neural network through multiple frames. All three neural networks can run simultaneously within the DLA and / or on the GPU708, for example.

[0125] In some applications, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of the legitimate driver and / or owner of vehicle 700. An always-on sensor processing engine may be used to unlock the vehicle and turn on the lights when the owner approaches the driver's side door, and in security mode, to stop the vehicle when the owner leaves the vehicle. In this way, SoC704 provides security against theft and / or vehicle hijacking.

[0126] In another example, a CNN for emergency vehicle detection and identification can use data from microphone 796 to detect and identify emergency vehicle sirens. In contrast to conventional systems that use a general classifier to detect sirens and manually extract features, SoC 704 uses a CNN for classifying environmental and urban sounds, as well as for classifying visual data. In one preferred embodiment, a CNN running on DLA is trained to identify the relative terminal velocity of emergency vehicles (for example, by using the Doppler effect). The CNN may also be trained to identify emergency vehicles specific to the local area in which the vehicle is operating, as identified by GNSS sensor 758. Thus, for example, when operating in Europe, the CNN would attempt to detect European sirens, and when in the United States, the CNN would attempt to identify only North American sirens. After an emergency vehicle is detected, a control program may be used, with the assistance of ultrasonic sensor 762, to perform emergency vehicle safety routines such as slowing down the vehicle, stopping it at the side of the road, parking the vehicle, and / or idling the vehicle until the emergency vehicle has passed.

[0127] The vehicle may include a CPU 718 (e.g., a discrete CPU, or dCPU) which can be connected to the SoC 704 via a high-speed interconnect (e.g., PCIe). The CPU 718 may include, for example, an x86 processor. The CPU 718 may be used to perform a variety of arbitrary functions, including, for example, mediating the consequences of potential mismatches between ADAS sensors and the SoC 704, and / or monitoring the status and condition of the controller 736 and / or the infotainment SoC 730.

[0128] Vehicle 700 may include a GPU 720 (e.g., a discrete GPU, or dGPU) which can be connected to SoC 704 via a high-speed interconnect (e.g., NVIDIA NVLINK). The GPU 720 can provide additional artificial intelligence capabilities, such as by running redundant and / or different neural networks, and may be used to train and / or update neural networks based on input from sensors in Vehicle 700 (e.g., sensor data).

[0129] Vehicle 700 may further include a network interface 724 which may include one or more wireless antennas 726 (e.g., one or more wireless antennas for different communication protocols, such as cellular antennas and Bluetooth® antennas). The network interface 724 may be used to enable wireless connectivity to a cloud via the Internet (e.g., a server 778 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). To communicate with other vehicles, a direct link may be established between two vehicles, and / or an indirect link may be established (e.g., via a network and via the Internet). The direct link may be provided using a vehicle-to-vehicle communication link. The vehicle-to-vehicle communication link can provide information about vehicle 700 regarding vehicles in close proximity to vehicle 700 (e.g., vehicles in front of, beside, and / or behind vehicle 700). This functionality may also be part of the vehicle 700's cooperative adaptive cruise control functionality.

[0130] The network interface 724 may include an SoC that provides modulation and demodulation functions and enables the controller 736 to communicate over a wireless network. The network interface 724 may include a radio frequency front end for baseband-to-radio frequency upconversion and radio frequency-to-baseband downconversion. Frequency conversion can be performed through well-known processes and / or using superheterodyne processes. In some examples, the radio frequency front end functionality may be provided by a separate chip. The network interface may include wireless functionality for communication over LTE, WCDMA®, UMTS, GSM, CDMA2000, Bluetooth®, Bluetooth® LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0131] The vehicle 700 may further include a data store 728 which may include storage outside the chip (for example, outside the SoC 704). The data store 728 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disk, and / or other components and / or devices capable of storing at least one bit of data.

[0132] The vehicle 700 may further include GNSS sensors 758. The GNSS sensors 758 (e.g., GPS, assisted GPS sensors, differential GPS (DGPS) sensors, etc.) assist in mapping, perception, occupy grid generation, and / or route planning functions. Any number of GNSS sensors 758 may be used, including, but not limited to, GPS using a USB connector with Ethernet® to a serial (RS-232) bridge.

[0133] Vehicle 700 may further include a RADAR sensor 760. The RADAR sensor 760 may be used by vehicle 700 for long-range vehicle detection, even in darkness and / or severe weather conditions. The RADAR functional safety level may be ASIL B. In some examples, the RADAR sensor 760 may use CAN and / or bus 702 for control and to access object tracking data (for example, to transmit data generated by the RADAR sensor 760) using Ethernet® access for accessing raw data. A wide variety of RADAR sensor types may be used. For example, and without limitation, the RADAR sensor 760 may be suitable for front, rear, and side RADAR use. In some examples, a pulsed Doppler RADAR sensor may be used.

[0134] The RADAR sensor 760 may include different configurations, such as long-range with a narrow field of view, short-range with a wide field of view, and short-range side coverage. In some examples, the long-range RADAR may be used for adaptive cruise control functions. The long-range RADAR system can provide a wide field of view achieved by two or more independent scans, such as within a range of 250m. The RADAR sensor 760 can help distinguish between static and moving objects and may be used by ADAS systems for emergency brake assist and forward collision warning. The long-range RADAR sensor may include monostatic multimodal RADARs with multiple (e.g., six or more) fixed RADAR antennas and high-speed CAN and FlexRay interfaces. In one example with six antennas, the four central antennas may create a focused beam pattern designed to record the area around the vehicle 700 at high speed with minimal interference from traffic in adjacent lanes. The other two antennas can widen the field of view, enabling rapid detection of vehicles entering or leaving the lane of the vehicle 700.

[0135] As an example, a medium-range RADAR system may include a range of up to 760m (front) or 80m (rear) and a field of view of up to 42 degrees (front) or 750 degrees (rear). A short-range RADAR system may include, but is not limited to, RADAR sensors designed to be mounted on both ends of the rear bumper. When mounted on both ends of the rear bumper, such a RADAR sensor system can create two beams that constantly monitor the blind spots behind and beside the vehicle.

[0136] Short-range radar systems can be used in ADAS systems for blind spot detection and / or lane change assistance.

[0137] The vehicle 700 may further include ultrasonic sensors 762. Positioned on the front, rear, and / or sides of the vehicle 700, the ultrasonic sensors 762 may be used for parking assistance and / or for creating and updating the occupancy grid. A wide variety of ultrasonic sensors 762 may be used, and different ultrasonic sensors 762 may be used for detection at different ranges (e.g., 2.5m, 4m). The ultrasonic sensors 762 may operate at a functional safety level of ASIL B.

[0138] The vehicle 700 may include a LiDAR sensor 764. The LiDAR sensor 764 may be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor 764 may also have a functional safety level of ASIL B. In some examples, the vehicle 700 may include multiple LiDAR sensors 764 (e.g., two, four, six, etc.) that can use Ethernet® (for example, to provide data to a Gigabit Ethernet® switch).

[0139] In some examples, the LIDAR sensor 764 may have the ability to provide a list of objects and their distances within a 360-degree field of view. A commercially available LIDAR sensor 764 may have an advertised range of approximately 700m, for example, with an accuracy of 2cm to 3cm and support for 700Mbps Ethernet® connectivity. In some examples, one or more non-protruding LIDAR sensors 764 may be used. In such examples, the LIDAR sensor 764 may be implemented as a small device that can be incorporated into the front, rear, side, and / or corners of a vehicle 700. In such examples, the LIDAR sensor 764 may have a range of 200m even for low-reflection objects and can provide a field of view up to 120 degrees horizontal and 35 degrees vertical. A front-mounted LIDAR sensor 764 may be configured for a horizontal field of view between 45 and 135 degrees.

[0140] In some applications, LiDAR technologies such as 3D flash LiDAR may also be used. 3D flash LiDAR uses a laser flash as a source to illuminate the area around the vehicle up to approximately 200m. The flash LiDAR unit includes a receptor that records the laser pulse travel time and reflected light on each pixel, sequentially corresponding to the range from the vehicle to the object. Flash LiDAR can enable the generation of high-precision and distortion-free images of the surroundings with every laser flash. In some applications, four flash LiDAR sensors may be deployed, one on each side of the vehicle. Available 3D flash LiDAR systems include solid-state 3D steering array LiDAR cameras (e.g., non-scanning LiDAR devices) that have no moving parts other than a blower. The flash LiDAR device can use 5 nanosecond Class I (eye-safe) laser pulses per frame and can capture reflected laser light in the form of a 3D range point cloud and co-documented intensity data. By using flash LiDAR, and because flash LiDAR is a solid-state device with no moving parts, the LiDAR sensor 764 can be made less susceptible to motion blur, vibration, and / or shock.

[0141] The vehicle may further include an IMU sensor 766. In some examples, the IMU sensor 766 may be positioned in the center of the rear axle of the vehicle 700. The IMU sensor 766 may include, but is not limited to, an accelerometer, magnetometer, gyroscope, magnetic compass, and / or other sensor types. In some examples, such as in a 6-axis application, the IMU sensor 766 may include an accelerometer and a gyroscope, while in a 9-axis application, the IMU sensor 766 may include an accelerometer, a gyroscope, and a magnetometer.

[0142] In some embodiments, the IMU sensor 766 may be implemented as a compact, high-performance GPS-aided inertial navigation system (GPS / INS) that combines a micro-electro-mechanical system (MEMS) inertial sensor, a high-sensitivity GPS receiver, and an advanced Kalman filtering algorithm to provide position, velocity, and attitude estimates. As such, in some examples, the IMU sensor 766 may enable the vehicle 700 to estimate its direction of travel without requiring input from a magnetic sensor by directly observing and correlating changes in velocity from the GPS to the IMU sensor 766. In some examples, the IMU sensor 766 and the GNSS sensor 758 may be combined in a single integrated unit.

[0143] The vehicle may include a microphone 796 placed inside and / or around the vehicle 700. The microphone 796 may, among other things, be used for emergency vehicle detection and identification.

[0144] The vehicle may further include any number of camera types, including a stereo camera 768, a wide-view camera 770, an infrared camera 772, a surround camera 774, a long-range and / or medium-range camera 798, and / or other camera types. The cameras may be used to capture 360-degree image data of the vehicle 700. The type of camera used will depend on the embodiment and requirements of the vehicle 700, and any combination of camera types may be used to achieve the required coverage around the vehicle 700. In addition, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or another number of cameras. The cameras may, as an example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet®. Each camera is described in more detail herein in relation to Figures 7A and 7B.

[0145] The vehicle 700 may further include a vibration sensor 742. The vibration sensor 742 can measure vibrations of vehicle components, such as axles. For example, a change in vibration may indicate a change in the road surface. In another example, when two or more vibration sensors 742 are used, the difference in vibration may be used to determine friction or slippage of the road surface (for example, when the difference in vibration is between a power-driven axle and a free-rotating axle).

[0146] Vehicle 700 may include ADAS system 738. In some examples, ADAS system 738 may include SoC. ADAS system 738 may include autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warning (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning system (CWS), lane centering (LC), and / or other features and functions.

[0147] The ACC system may utilize a radar sensor 760, a lithium-ion sensor 764, and / or cameras. The ACC system may include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of vehicle 700 and automatically adjusts the vehicle speed to maintain a safe distance from the vehicle ahead. Lateral ACC performs distance maintenance and advises vehicle 700 to change lanes when necessary. Lateral ACC is related to other ADAS applications such as LCA and CWS.

[0148] CACC uses information from other vehicles that can be received from other vehicles via a wireless link through a network interface 724 and / or a wireless antenna 726, or indirectly via a network connection (e.g., via the Internet). Direct links may be provided by vehicle-to-vehicle (V2V) communication links, while indirect links may be infrastructure-to-vehicle (I2V) communication links. Generally, the V2V communication concept provides information about the vehicle immediately ahead (e.g., a vehicle in the same lane as vehicle 700, immediately in front of vehicle 700), while the I2V communication concept provides information about traffic further ahead. A CACC system may include either or both I2V and V2V information sources. Given information about vehicles ahead of vehicle 700, CACC can be more reliable, and CACC has the potential to make traffic flow smoother and reduce road congestion.

[0149] The FCW system is designed to warn the driver of hazards so that the driver can take corrective action. The FCW system uses a forward-facing camera and / or radar sensor 760, coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback such as a display, speaker, and / or vibration components. The FCW system can provide warnings in the form of audible, visual, vibration, and / or quick brake pulses.

[0150] An AEB system can detect an imminent forward collision with another vehicle or object and automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. The AEB system may use a forward-facing camera and / or radar sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it typically first warns the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system may automatically apply the brakes as part of an effort to prevent, or at least mitigate, the impact of the anticipated collision. The AEB system may include techniques such as dynamic brake support and / or emergency collision braking.

[0151] The LDW system warns the driver when the vehicle 700 crosses a lane marking by providing visual, audible, and / or tactile warnings, such as vibration of the steering wheel or seat. The LDW system does not activate when the driver indicates an intentional lane departure by activating the turn signal. The LDW system may use a forward-facing camera connected to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration components.

[0152] The LKA system is a modified version of the LDW system. The LKA system provides steering input or braking to correct the vehicle 700 if it begins to drift out of its lane.

[0153] The BSW system detects and warns the driver of a vehicle in its blind spots. The BSW system can provide visual, audible, and / or tactile warnings to indicate that merging or changing lanes is unsafe. The system can provide additional warnings when the driver uses the turn signal. The BSW system can use a rear-facing camera and / or radar sensor 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0154] The RCTW system can provide visual, audible, and / or haptic notifications when an object is detected outside the range of the rear camera while the vehicle 700 is reversing. Some RCTW systems include AEB to ensure that the vehicle brakes are applied to avoid a collision. The RCTW system may use one or more rear-facing RADAR sensors 760 coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which are electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0155] Conventional ADAS systems warn the driver, prompting them to determine whether a safe condition truly exists and to act accordingly. However, conventional ADAS systems have a tendency to produce misjudgments that, while not usually catastrophic, can frustrate and distract the driver. In the autonomous vehicle 700, however, if the results are contradictory, the vehicle 700 itself must decide whether to heed the results from the primary computer or the secondary computer (e.g., the first controller 736 or the second controller 736). For example, in some embodiments, the ADAS system 738 may also be a backup and / or secondary computer for providing perceptual information to a backup computer rationality module. The backup computer rationality monitor can run redundant and diverse software on hardware components to detect failures in perceptual and dynamic driving tasks. The output from the ADAS system 738 may be provided to the supervisory MCU. If the outputs from the primary and secondary computers are contradictory, the supervisory MCU must decide how to reconcile the contradiction to ensure safe operation.

[0156] In some implementations, the primary computer may be configured to provide the supervising MCU with a reliability score indicating the reliability of the primary computer in the selected outcome. If the reliability score exceeds a threshold, the supervising MCU may follow the primary computer's instructions, regardless of whether the secondary computer gives conflicting or inconsistent results. If the reliability score does not meet the threshold, and if the primary and secondary computers produce different results (e.g., conflicting results), the supervising MCU may mediate between the computers to determine an appropriate outcome.

[0157] The supervisory MCU may be configured to run a neural network trained and configured to determine, based on the outputs from the primary and secondary computers, when a secondary computer provides a false alarm. Thus, the neural network in the supervisory MCU can learn when the output of the secondary computer is reliable and when it is not. For example, when the secondary computer is a radar-based forward crossing (FCW) system, the neural network in the supervisory MCU can learn when the FCW identifies a metal object that is not actually dangerous, such as a sewer grate or manhole cover, which triggers an alarm. Similarly, when the secondary computer is a camera-based lane departure warning (LDW) system, the neural network in the supervisory MCU can learn to ignore the LDW when a cyclist or pedestrian is present and lane departure is actually the safest operation. In embodiments including a neural network running on the supervisory MCU, the supervisory MCU may include at least one DLA or GPU suitable for running a neural network with associated memory. In a preferred embodiment, the supervisory MCU may comprise and / or be included as a component of the SoC704.

[0158] In other examples, ADAS system 738 may include a secondary computer that performs ADAS functions using conventional rules of computer vision. As such, the secondary computer may use classical computer vision rules (if-then), and the presence of a neural network within the supervisory MCU can improve reliability, safety, and performance. For example, diverse implementations and intentional non-identities make the entire system more fault-tolerant, particularly to failures caused by software (or software-hardware interface) functions. For instance, if a software bug or error exists in the software running on the primary computer, and non-identical software code running on the secondary computer produces the same overall result, the supervisory MCU may have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer did not cause a critical error.

[0159] In some examples, the output of the ADAS system 738 may be supplied to the perception block and / or the dynamic driving task block of the primary computer. For example, if the ADAS system 738 indicates a forward collision warning due to an object immediately ahead, the perception block can use this information when identifying the object. In other examples, the secondary computer may have its own neural network, which is trained as described herein and therefore reduces the risk of misjudgment.

[0160] Vehicle 700 may further include an infotainment SoC 730 (for example, an in-vehicle infotainment system (IVI)). Although illustrated and described as an SoC, the infotainment system does not have to be an SoC and may include two or more discrete components. The infotainment SoC 730 may include a combination of hardware and software that can be used to provide vehicle 700 with audio (e.g., music, personal digital assistant, navigation commands, news, radio, etc.), video (e.g., TV, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation system, rear parking assist, radio data system, fuel level, total mileage, brake fuel level, oil level, door open / close, air filter information, and other vehicle-related information). For example, the infotainment SoC 730 may include wireless, disc player, navigation system, video player, USB and Bluetooth® connectivity, car computer, in-car entertainment, Wi-Fi, steering wheel audio control unit, hands-free voice control, heads-up display (HUD), HMI display 734, telematics device, control panel (for example, for controlling and / or interacting with various components, features, and / or systems), and / or other components. The infotainment SoC 730 may be further used to provide information (for example, visual and / or audible) to the vehicle user, such as information from the ADAS system 738, autonomous driving information such as planned vehicle operation, trajectory, surrounding environment information (for example, intersection information, vehicle information, road information, etc.), and / or other information.

[0161] The infotainment SoC 730 may include GPU functionality. The infotainment SoC 730 can communicate with other devices, systems, and / or components of the vehicle 700 via bus 702 (e.g., CAN bus, Ethernet®, etc.). In some examples, the infotainment SoC 730 may be coupled to a supervisory MCU so that the infotainment system's GPU can perform certain self-drive functions in the event of a primary controller 736 (e.g., the vehicle 700's primary and / or backup computer) failure. In such examples, the infotainment SoC 730 can put the vehicle 700 into a driver-assistant mode for safe stopping, as described herein.

[0162] Vehicle 700 may further include an instrument cluster 732 (e.g., a digital dash, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 732 may include a controller and / or a supercomputer (e.g., a discrete controller or supercomputer). The instrument cluster 732 may include a set of instruments such as a speedometer, fuel level indicator, oil pressure indicator, tachometer, odometer, turn signals, gear shift position indicator, seat belt warning light, parking brake warning light, engine fault light, airbag (SRS) system information, lighting control device, safety system control device, and navigation information. In some real forms, information may be displayed and / or shared between the infotainment SoC 730 and the instrument cluster 732. In other words, the instrument cluster 732 may be included as part of the infotainment SoC 730, and vice versa.

[0163] Figure 7D is a system diagram of communication between the cloud-based server in Figure 7A and an exemplary autonomous vehicle 700, according to several embodiments of the present disclosure. System 776 may include a server 778, a network 790, and a vehicle including the vehicle 700. Server 778 may include a plurality of GPUs 784(A) to 784(H) (collectively referred to herein as GPU 784), PCIe switches 782(A) to 782(H) (collectively referred to herein as PCIe switch 782), and / or CPUs 780(A) to 780(B) (collectively referred to herein as CPU 780). The GPUs 784, CPUs 780, and PCIe switches may be interconnected by high-speed interconnects, such as, for example, NVLink interfaces 788 and / or PCIe connections 786 developed by NVIDIA. In some examples, the GPU784 is connected via NVLink and / or NVSwitch SoCs, and the GPU784 and PCIe switch 782 are connected via PCIe interconnects. Eight GPU784s, two CPU780s, and two PCIe switches are illustrated, but this is not intended to be limiting. Depending on the embodiment, each server 778 may include any number of GPU784s, CPU780s, and / or PCIe switches. For example, each server 778 may include eight, sixteen, thirty-two, and / or more GPU784s.

[0164] Server 778 can receive image data from vehicles via network 790, representing images showing unexpected or altered road conditions, such as recently started road construction. Server 778 can transmit to vehicles via network 790 the neural network 792, updated neural network 792, and / or map information 794, which includes information about traffic and road conditions. Updates to map information 794 may include updates to HD map 722, such as information about construction sites, potholes, detours, floods, and / or other obstacles. In some instances, the neural network 792, updated neural network 792, and / or map information 794 may have arisen from new training and / or experience represented in data received from any number of vehicles in the environment, and / or based on training performed in a data center (for example, using server 778 and / or other servers).

[0165] Server 778 may be used to train a machine learning model (e.g., a neural network) based on training data. The training data may be generated by a vehicle and / or in a simulation (e.g., using a game engine). In some instances, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or otherwise pre-processed, while in other instances, the training data is not tagged and / or pre-processed (e.g., if the neural network does not require supervised learning). Training may be performed according to any one or more classes of machine learning techniques, including but not limited to the following: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, associative learning, transfer learning, feature learning (including key component and cluster analysis), multilinear subspace learning, manifold learning, representation learning (including pre-dictionary learning), rule-based machine learning, anomaly detection, and variations or combinations thereof. After the machine learning model has been trained, it may be used by the vehicle (for example, transmitted to the vehicle via network 790), and / or the machine learning model may be used by server 778 to remotely monitor the vehicle.

[0166] In some examples, Server 778 can receive data from vehicles and apply it to state-of-the-art real-time neural networks for real-time intelligent inference. Server 778 may include deep learning supercomputers and / or dedicated AI computers powered by GPU 784, such as the DGX and DGX Station Machines developed by NVIDIA. However, in some examples, Server 778 may include deep learning infrastructure that uses only CPU-powered data centers.

[0167] The deep learning infrastructure of server 778 can have the capability for high-speed real-time inference, which can be used to evaluate and verify the condition of the processor, software, and / or associated hardware within vehicle 700. For example, the deep learning infrastructure can receive periodic updates from vehicle 700, such as images of a sequence and / or objects located within images of that sequence (e.g., via computer vision and / or other machine learning object classification techniques). The deep learning infrastructure can run its own neural network to identify objects and compare them with objects identified by vehicle 700, and if the results do not match and the infrastructure concludes that the AI ​​within vehicle 700 is not functioning properly, server 778 can send a signal to vehicle 700 instructing the vehicle's fail-safe computer to infer control, notify passengers, and complete a safe parking operation.

[0168] For inference, server 778 may include GPU 784 and one or more programmable inference accelerators (e.g., NVIDIA TensorRT). The combination of a GPU-powered server and inference accelerator can enable real-time responsiveness. In other examples, such as when high performance is not required, a server powered by a CPU, FPGA, and other processors may be used for inference.

[0169] (Example computing device) Figure 8 is a block diagram of an example of a computing device 800 suitable for use in implementing some embodiments of the present disclosure. The computing device 800 may include an interconnection system 802 that directly or indirectly connects the following devices: memory 804, one or more central processing units (CPUs) 806, one or more graphics processing units (GPUs) 808, a communication interface 810, input / output (I / O) ports 812, input / output components 814, a power supply device 816, one or more presentation components 818 (e.g., a display), and one or more logical units 820. In at least one embodiment, the computing device 800 may include one or more virtual machines (VMs), and / or any of its components may include virtual components (e.g., virtual hardware components). As an unrestricted example, one or more of the GPUs 808 may include one or more vGPUs, one or more of the CPUs 806 may include one or more vCPUs, and / or one or more of the logical units 820 may include one or more virtual logical units. As such, the computing device 800 may include discrete components (e.g., an entire GPU dedicated to the computing device 800), virtual components (e.g., a portion of a GPU dedicated to the computing device 800), or a combination thereof.

[0170] The various blocks in Figure 8 are shown connected by lines via the interconnection system 802, but this is not intended to be limiting and is simply for illustrative purposes. For example, in some embodiments, a presentation component 818, such as a display device, could be considered an I / O component 814 (for example, if the display is a touchscreen). As another example, the CPU 806 and / or GPU 808 may include memory (for example, memory 804 may represent a storage device in addition to the memory of the GPU 808, CPU 806, and / or other components). In other words, the computing devices in Figure 8 are merely illustrative. Categories such as “workstation,” “server,” “laptop,” “desktop,” “tablet,” “client device,” “mobile device,” “handheld device,” “game console,” “electronic control unit (ECU),” “virtual reality system,” and / or other device or system types are all intended to fall within the scope of the computing devices in Figure 8 and are therefore not distinguished.

[0171] The interconnection system 802 may represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The interconnection system 802 may include one or more bus or link types, such as an industry standard architecture (ISA) bus, an extended industry standard architecture (EISA) bus, a VESA (video electronics standards association) bus, a peripheral component interconnect (PCI) bus, a peripheral component interconnect express (PCIe) bus, and / or other types of buses or links. In some embodiments, direct connections exist between components. For example, the CPU 806 may be directly connected to the memory 804. Furthermore, the CPU 806 may be directly connected to the GPU 808. Where direct or point-to-point connections exist between components, the interconnection system 802 may include PCIe links to implement the connections. In these examples, the PCI bus does not need to be included in the computing device 800.

[0172] Memory 804 may include any of various computer-readable media. The computer-readable media may be any available media accessible by the computing device 800. The computer-readable media may include both volatile and non-volatile media, as well as removable and non-removable media. In practice, but not limited to, the computer-readable media may include computer storage media and communication media.

[0173] Computer storage media may include both volatile and non-volatile media, and / or removable and non-removable media, implemented in any method or technique for storing information such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory 804 may store computer-readable instructions (e.g., representing programs and / or program elements), such as an operating system. Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory, or other memory technologies, CD-ROM, digital versatile disk (DVD), or other optical disk storage, magnetic cassette, magnetic tape, magnetic disk storage, or other magnetic storage devices, or any other media that can be used to store desired information and can be accessed by computing device 800. In this specification, computer storage media does not include signals themselves.

[0174] Computer storage media include any information distribution medium that can implement computer-readable instructions, data structures, program modules, and / or other data types in modulated data signals such as carrier waves or other transfer mechanisms. The term “modulated data signal” may refer to a signal that has been modified in a manner that has one or more of its characteristic sets or encodes information within the signal. By example, but not limited to, computer storage media may include wired media such as wired networks or direct wired connections, and wireless media such as acoustic, RF, infrared, and other wireless media. Any combination of the foregoing should also be included in the scope of computer-readable media.

[0175] The CPU 806 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to execute one or more of the methods and / or processes described herein. The CPU 806 may include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) each capable of processing a large number of software threads concurrently. The CPU 806 may include any type of processor, and depending on the type of computing device 800 in which it is implemented, it may include different types of processors (e.g., a processor with fewer cores for mobile devices and a processor with more cores for servers). For example, depending on the type of computing device 800, the processor may be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 800 may include one or more CPUs 806 in addition to one or more microprocessors or auxiliary coprocessors, such as a computing coprocessor.

[0176] In addition to or instead of the CPU 806, the GPU 808 may be configured to execute at least some computer-readable instructions to control one or more components of the computing device 800 to execute one or more of the methods and / or processes described herein. One or more of the GPU 808 may be an integrated GPU (for example, with one or more of the CPU 806), and / or one or more of the GPU 808 may be a discrete GPU. In embodiments, one or more of the GPU 808 may be a coprocessor of one or more of the CPU 806. The GPU 808 may be used by the computing device 800 to render graphics (for example, 3D graphics) or to perform general-purpose computing. For example, the GPU 808 may be used for GPU-based general-purpose computing (GPGPU). It can be used for a GPU. The GPU808 may include hundreds or thousands of cores capable of processing hundreds or thousands of software threads simultaneously. The GPU808 can generate pixel data for an output image in response to rendering commands (for example, rendering commands from CPU806 received via the host interface). The GPU808 may include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. Display memory may be included as part of memory 804. The GPU808 may include two or more GPUs operating in parallel (for example, via a link). The link can connect directly to the GPUs (for example, using NVLINK) or via a switch (for example, using NVSwitch). When coupled together, each GPU808 can generate pixel data or GPGPU data for different parts of an output or for different outputs (for example, the first GPU for the first image and the second GPU for the second image). Each GPU may have its own memory or may share memory with other GPUs.

[0177] In addition to or instead of the CPU 806 and / or GPU 808, the logic unit 820 may be configured to execute at least some computer-readable instructions to control one or more of the computing devices 800 to execute one or more of the methods and / or processes described herein. In embodiments, the CPU 806, GPU 808, and / or logic unit 820 can execute any combination of methods, processes, and / or parts thereof discretely or congruently. One or more of the logic units 820 may be part of and / or integrated with one or more of the CPU 806 and / or GPU 808, and / or one or more of the logic units 820 may be discrete components of the CPU 806 and / or GPU 808 or otherwise external to them. In embodiments, one or more of the logic units 820 may be coprocessors of one or more of the CPU 806 and / or one or more of the GPU 808.

[0178] Examples of logical unit 820 include one or more processing cores and / or components thereof, such as a Data Processing Unit (DPU), Tensor Core (TC), Tensor Processing Unit (TPU), Pixel Visual Core (PVC), Vision Processing Unit (VPU), Graphics Processing Cluster (GPC), Texture Processing Cluster (TPC), Streaming Multiprocessor (SM), Tree Traversal Unit (TTU), Artificial Intelligence Accelerator (AIA), and Deep Learning Accelerator (DLA). This includes an Accelerator, a Logical Unit (ALU), an Application-Specific Integrated Circuit (ASIC), a Floating-Point Unit (FPU), input / output (I / O) elements, a Peripheral Component Interconnect (PCI) or Peripheral Component Interconnect Express (PCIe) element, and / or similar.

[0179] The communication interface 810 may include one or more receivers, transmitters, and / or transceivers that enable the computing device 800 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 810 may include components and functions to enable communication over any of several different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth®, Bluetooth® LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet® or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet. In one or more embodiments, the logic unit 820 and / or the communication interface 810 may include one or more data processing units (DPUs) to transmit data received via the network and / or through the interconnection system 802 directly to one or more GPUs 808 (e.g., their memory).

[0180] I / O port 812 can enable the computing device 800 to be logically connected to other devices, including I / O components 814, presentation components 818, and / or other components, some of which can be built into (e.g., integrated into) the computing device 800. Exemplary I / O components 814 include microphones, mice, keyboards, joysticks, gamepads, game controllers, satellite dishes, scanners, printers, wireless devices, etc. I / O components 814 can provide a natural user interface (NUI) that processes air gestures, voice, or other physiological inputs generated by the user. In some cases, the input may be transmitted to appropriate network elements for further processing. The NUI may implement any combination of voice recognition, stylus recognition, face recognition, biometric recognition, on-screen and beside-screen gesture recognition, air gestures, head and target tracking, and touch recognition related to the display of the computing device 800 (as described in more detail below). The computing device 800 may include depth cameras, such as stereoscope camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof, for gesture detection and recognition. Additionally, the computing device 800 may include accelerometers or gyroscopes to enable motion detection (for example, as part of an inertia measurement unit (IMU)). In some examples, the output of the accelerometer or gyroscope may be used by the computing device 800 to render immersive augmented reality or virtual reality.

[0181] The power supply device 816 may include a hardwired power supply device, a battery power supply device, or a combination thereof. The power supply device 816 can provide power to the computing device 800 to enable the components of the computing device 800 to operate.

[0182] The presentation component 818 may include a display (e.g., a monitor, touch screen, television screen, head-up display device (HUD), other display types, or a combination thereof), a speaker, and / or other presentation components. The presentation component 818 can receive data from other components (e.g., GPU 808, CPU 806, DPU, etc.) and output data (e.g., as images, videos, sounds, etc.).

[0183] (Example data center) Figure 9 shows an exemplary data center 900 that may be used in at least one embodiment of the present disclosure. The data center 900 may include a data center infrastructure layer 910, a framework layer 920, a software layer 930, and / or an application layer 940.

[0184] As shown in Figure 9, the data center infrastructure layer 910 may include a resource orchestrator 912, grouped computing resources 914, and node computing resources ("node CRs") 916(1) to 916(N), where "N" represents any positive integer. In at least one embodiment, the node CRs 916(1) to 916(N) may include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processors or graphics processing units (GPUs), etc.), memory devices (e.g., dynamic read-only memory), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules, and / or cooling modules. In some embodiments, one or more of the nodes CRs916(1) to 916(N) may correspond to a server having one or more of the aforementioned computing resources. In addition, in some embodiments, nodes CRs916(1) to 916(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or similar, and / or one or more of the nodes CRs916(1) to 916(N) may correspond to a virtual machine (VM).

[0185] In at least one embodiment, the grouped computing resources 914 may include a separate group of nodes CRs916 housed in one or more racks (not shown), or a number of racks housed in data centers in various geographical locations (also not shown). The separate group of nodes CRs916 within the grouped computing resources 914 may include grouped compute, network, memory, or storage resources that can be configured or allocated to support one or more workloads. In at least one embodiment, several nodes CRs916, including CPUs, GPUs, DPUs, and / or other processors, may be grouped in one or more racks to provide computing resources to support one or more workloads. The one or more racks may also include any number of power modules, cooling modules, and / or network switches in any combination.

[0186] The resource orchestrator 912 can configure or otherwise control one or more node CRs 916(1) to 916(N) and / or grouped computing resources 914. In at least one embodiment, the resource orchestrator 912 may include a software design infrastructure ("SDI") management entity of the data center 900. The resource orchestrator 912 may include hardware, software, or any combination thereof.

[0187] In at least one embodiment, as shown in Figure 9, the framework layer 920 may include a job scheduler 932, a configuration manager 934, a resource manager 936, and / or a distributed file system 938. The framework layer 920 may include a framework to support software 932 of the software layer 930 and / or one or more applications 942 of the application layer 940. The software 932 or application 942 may include web-based service software or applications, such as those provided by Amazon Web Services®, Google Cloud, and Microsoft Azure, respectively. The framework layer 920 may be, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark® ("Spark"), which can use the distributed file system 938 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 932 may include a Spark driver to facilitate scheduling of workloads supported by various layers of the data center 900. The configuration manager 934 may have the ability to configure different layers, for example, a software layer 930 and a framework layer 920 including Spark and a distributed file system 938 to support large-scale data processing. The resource manager 936 may have the ability to manage clustered or grouped computing resources mapped or allocated for support of the distributed file system 938 and the job scheduler 932. In at least one embodiment, the clustered or grouped computing resources may include computing resources 914 grouped in the data center infrastructure layer 910. The resource manager 936 can coordinate with the resource orchestrator 912 to manage these mapped or allocated computing resources.

[0188] In at least one embodiment, the software 932 included in the software layer 930 may include software used by at least a portion of the nodes CRs 916(1) to 916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of software may include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0189] In at least one embodiment, the application 942 included in the application layer 940 may include one or more types of applications used by at least a portion of the nodes CRs 916(1) to 916(N), the grouped computing resources 914, and / or the distributed file system 938 of the framework layer 920. One or more types of applications may include, but are not limited to, any number of genomics applications, cognitive computing, and machine learning applications, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0190] In at least one embodiment, any of the configuration manager 934, resource manager 936, and resource orchestrator 912 may implement any number and type of self-rewriting actions based on any amount and type of data obtained in any technically possible manner. Self-rewriting actions may free the data center operator of data center 900 from making potentially poor configuration decisions and possibly avoiding underutilized and / or underperforming parts of the data center.

[0191] The data center 900 may include tools, services, software, or other resources for training one or more machine learning models or for predicting or inferring information using one or more machine learning models according to one or more embodiments described herein. For example, a machine learning model may be trained by calculating weight parameters by a neural network architecture using the software and / or computing resources described herein with respect to the data center 900. In at least one embodiment, a trained or deployed machine learning model corresponding to one or more neural networks may be used to infer or predict information using the resources described herein with respect to the data center 900 by using weight parameters calculated via one or more training techniques, not limited to those described herein.

[0192] In at least one embodiment, the data center 900 may use a CPU, application-specific integrated circuit (ASIC), GPU, FPGA, and / or other hardware (or corresponding virtual computing resources) to perform training and / or inference using the aforementioned resources. Furthermore, one or more of the aforementioned software and / or hardware resources may be configured as services that enable users to train or perform inference of information, such as image recognition, speech recognition, or other artificial intelligence services.

[0193] (Example network environment) A network environment suitable for use in implementing embodiments of the embodiments of this disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. Each client device, server, and / or other device type (e.g., each device) may be implemented as one or more instances of the computing device 800 in Figure 8, for example, each device may include similar components, features, and / or functionalities of the computing device 800. In addition, if backend devices (e.g., servers, NAS, etc.) are implemented, they may be included as part of the data center 900, examples of which are further detailed herein with respect to Figure 9.

[0194] Components of a network environment may communicate with one another via the network, either wired, wirelessly, or both. A network may comprise multiple networks, or one of multiple networks. Examples include one or more wide-area networks (WANs), one or more local-area networks (LANs), one or more public networks, such as the Internet and / or the Public Switched Telephone Network (PSTN), and / or one or more private networks. If a network includes a wireless telecommunications network, its components, such as base stations, towers, or access points (and other components), may provide wireless connectivity.

[0195] Compatible network environments may include one or more peer-to-peer network environments (in which case servers may not be included in the network environment) and one or more client-server network environments (in which case one or more servers may be included in the network environment). In a peer-to-peer network environment, the functionality described herein with respect to the server can be implemented on any number of client devices.

[0196] In at least one embodiment, the network environment may include one or more cloud-based network environments, distributed computing environments, or a combination thereof. The cloud-based network environment may include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more of the servers, which may include one or more core network servers and / or edge servers. The framework layer may include a framework to support the software in the software layer and / or one or more applications in the application layer. The software or applications may each include web-based service software or applications. In embodiments, one or more of the client devices may use the web-based service software or applications (for example, by accessing the service software and / or applications via one or more application programming interfaces (APIs)). The framework layer may be, but is not limited to, a type of free and open-source software web application framework that may use a distributed file system for, for example, large-scale data processing (e.g., “big data”).

[0197] A cloud-based network environment may provide cloud computing and / or cloud storage that implements any combination of the computing and / or data storage functions (or one or more of them) described herein. Any of these various functions may be distributed across multiple locations from a central or core server (such as one or more data centers that may be distributed across states, territories, countries, or the world). If the connection to the user (e.g., a client device) is relatively close to the edge server, the core server may designate at least a portion of its functionality to the edge server. The cloud-based network environment may be private (e.g., limited to a single organization), public (e.g., available to multiple organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0198] A client device may include at least some of the components, features, and functionalities of the exemplary computing device 800 described herein with respect to Figure 8. As an example, and not limited to, a client device may be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or device, video player, video camera, surveillance device or system, vehicle, boat, airship, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, instrument, consumer electronic device, workstation, edge device, any combination of these depicted devices, or any other suitable device.

[0199] This disclosure may be described in general terms with computer code or machine-usable instructions, including computer-executable instructions such as program modules, which are executed by computers or other machines, such as personal digital assistants or other handheld devices. Generally, a program module, including routines, programs, objects, components, and data structures, refers to code that performs a specific task or implements a specific abstract data type. This disclosure may be implemented in a variety of configurations, including handheld devices, consumer electronics, general-purpose computers, and more specialized computing devices. This disclosure may also be implemented in a distributed computing environment where tasks are performed by remote processing devices linked over a communication network.

[0200] In this specification, any “and / or” statement relating to two or more elements should be interpreted as meaning only one element or a combination of elements. For example, “element A, element B, and / or element C” may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. In addition, “at least one of element A or element B” may include at least one element A, at least one element B, or at least one element A and at least one element B. Furthermore, “at least one of element A and element B” may include at least one element A, at least one element B, or at least one element A and at least one element B.

[0201] The inventions of this disclosure are described in a manner that is specific in order to satisfy statutory requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors intend that the inventions described in the claims may be carried out in other ways, including different steps or combinations of steps similar to those described herein, in conjunction with other current or future technologies. Furthermore, the terms “step” and / or “block” may be used herein to imply different elements of the way in which they are used, but these terms should not be construed as implying any particular order among the various steps disclosed herein unless the order of the individual steps is expressly stated and, when so, is explicitly stated.

Claims

1. Using a machine learning model, data indicating the viewing distance corresponding to the sensor data is calculated, at least partially based on sensor data generated using one or more sensors of the ego machine. Based at least partially on the aforementioned viewing distance, the usefulness of the sensor data for the operation of one or more of the ego machines is determined. Based at least partially on the usefulness of the sensor data, at least one of the one or more operations is performed. Includes one or more circuits, A processor in which the calculation of the aforementioned data is regressed by the value of the viewing distance.

2. The processor according to claim 1, further comprising deciding to refrain from performing at least one of the one or more operations autonomous or semi-autonomous operations based at least in part on the usefulness of the sensor data.

3. Determining the viewing distance bin from a plurality of viewing distance bins corresponding to the sensor data, based at least in part on the aforementioned value of the viewing distance. The processor according to claim 1, further comprising the following: the viewing distance bin indicates the usefulness of the sensor data for the one or more operations of the ego machine.

4. The processor according to claim 1, wherein the data indicates a viewing distance bin among a plurality of viewing distance bins, and further, the viewing distance bin indicates the usefulness of the sensor data for one or more operations of the ego machine.

5. The processor according to claim 1, wherein one or more of the operations include at least one of object tracking, object detection, path planning, obstacle avoidance, or advanced driver assistance system (ADAS) operations.

6. The processor according to claim 1, wherein the machine learning model includes a deep neural network (DNN), and the DNN is trained using a combination of real-world data, augmented reality data, and synthetic data.

7. The aforementioned processor, Control systems for autonomous or semi-autonomous machines Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The processor according to claim 1, which is included in at least one of the following.

8. One or more sensors, One or more processing units for determining the usefulness of an instance of sensor data generated using one or more sensors, at least in part, based on the relevant viewing distance calculated using a machine learning model. A system comprising, wherein the machine learning model, Receiving values ​​corresponding to one or more parameters for adjusting visibility corresponding to instances of training sensor data, To generate instances of the training sensor data based at least partially on the aforementioned values, The viewing distance corresponding to the instance of the training sensor data is determined based at least partially on the aforementioned value, and Training a machine learning model using instances of the training sensor data and the line of sight distance as ground truth data. A system that is at least partially trained by [someone / something].

9. The system according to claim 8, wherein the parameters correspond to weather conditions, lighting conditions, or occlusion conditions.

10. The system according to claim 8, wherein determining the viewing distance includes calculating the viewing distance using the value.

11. The system according to claim 10, wherein the aforementioned values ​​are applied to a trained model that determines the correspondence between the values ​​of each of the one or more parameters and the viewing distance.

12. The system according to claim 11, wherein the trained model is trained using annotations or labels indicating the viewing distance corresponding to instances of training sensor data generated using the associated values ​​of the one or more parameters.

13. The system according to claim 8, wherein the instance of training sensor data corresponds to an instance of real-world sensor data, and generating the instance of training sensor data includes extending the instance of real-world sensor data using the values ​​corresponding to one or more parameters in order to generate the instance of training sensor data.

14. The instance of the training sensor data corresponds to the composite instance of the sensor data, and the generation of the composite instance of the sensor data is Applying the values ​​corresponding to one or more of the aforementioned parameters to the simulation engine to generate a simulation, and Capturing a composite instance of the sensor data using one or more virtual sensors within the aforementioned simulation. The system according to claim 8, including the following:

15. The system according to claim 8, wherein the machine learning model is trained to compute data indicating sight distance, and the training includes using a loss function that penalizes overestimating the sight distance more than underestimating it.

16. The aforementioned system Control systems for autonomous or semi-autonomous machines Cognitive systems for autonomous or semi-autonomous machines A system for performing simulation operations. A system for performing deep learning operations. Systems implemented using edge devices, Systems implemented using robots, A system that incorporates one or more virtual machines (VMs). A system that is at least partially implemented in a data center, or A system that is at least partially implemented using cloud computing resources. The system according to claim 8, which is included in at least one of the following.

Images