Deep Image Analysis and Correction for Machine Learning Systems and Applications

Abstract

Various examples provide depth image analysis and correction for stereo image machine learning systems and applications. A depth image anomaly identification stage of a depth image anomaly processor can perform functions to evaluate a depth image in order to identify anomalous segments where the values ​​of pixels predicted by a stereo depth perception model appear anomalous in terms of accuracy compared to other regions of the depth image. A correction stage adjusts the depth image based on the identified anomalous segments using one or more correction techniques that can compute estimates or predictions for depth values ​​within the identified anomalous segments. The correction can be based on detected structural characteristics within the spatial volume corresponding to the identified anomalous segments and / or on characteristics of local phenomena (e.g.,lighting, reflections, glare, etc.) are based on factors that have been identified as causing inaccuracies in depth values.

Description

BACKGROUND

[0001] The representation of three-dimensional (3D) information in a scene is often captured as a depth or disparity image. A depth or disparity image is essentially similar to a traditional two-dimensional color raster image (2D color raster image), except that each pixel in a depth image represents information about a depth (e.g., depth to an object) that allows for the reconstruction of the scene's 3D geometry. Stereoscopic vision techniques, time-of-flight (ToF) cameras, structured light projection techniques, and light detection and ranging (LiDAR) systems are all technologies that can be modified to gather depth information from a scene.Depth information can be used in various applications, such as 3D modeling, augmented reality, vehicle safety, autonomous systems, and robotics, where understanding the spatial relationship between objects is crucial. For example, depth information can be used by autonomous or semi-autonomous vehicles or robots to help an ego machine detect hazards (such as foreign material, road defects, other vehicles, wild or free-roaming animals, and / or pedestrians) within its path. In other cases, depth information can be used within a vehicle interior by an occupant monitoring system (OMS).For example, an OMS—using data generated or obtained by vehicle or machine sensors—can be used to track a driver's gaze direction, head posture, or blinking (for example, to detect drowsiness, fatigue, and / or distraction), for hand position and / or gesture detection, child and / or pet presence detection, and / or in conjunction with the operation of features such as, but not limited to, seatbelt checks, seat heating, and / or intelligent airbag deployment. In other applications, depth images can be used in machine learning applications, such as in conjunction with machine learning models trained to predict 3D information from images captured by cameras and to verify the accuracy of such models. OVERVIEW

[0002] The invention is defined by the claims. To illustrate the invention, aspects and embodiments that may or may not be within the scope of the claims are described herein.

[0003] This document discloses depth image analysis and correction for stereo image machine learning systems and applications. A depth image anomaly identification stage of a depth image anomaly processor can perform functions to evaluate a depth image in order to identify anomalous segments where the values ​​of pixels predicted by a stereo depth perception model appear anomalous in terms of accuracy compared to other regions of the depth image. A correction stage adjusts the depth image based on the identified anomalous segments using one or more correction techniques that can compute estimates or predictions for depth values ​​within the identified anomalous segments. The correction can be based on detected structural characteristics within the spatial volume corresponding to the identified anomalous segments and / or on characteristics of local phenomena (e.g.,lighting, reflections, glare, etc.) are based on factors that have been identified as causing inaccuracies in depth values.

[0004] Embodiments of the present disclosure relate to depth image analysis and correction for stereo image machine learning systems and applications. Systems and methods for evaluating and correcting machine learning model depth images generated from stereo image pairs are disclosed, which can be used in computer vision and perception-based systems.

[0005] In contrast to conventional systems, embodiments of the depth image analysis and correction systems described herein can generate ground-truth depth images based on depth images generated by a stereo depth perception model from a pair of stereo images of a scene. As described herein, in some embodiments, a process may include a depth image anomaly identification stage, which may be followed by a depth image correction stage. In some embodiments, a stereo image pair (e.g., left and right stereo images acquired by a pair of cameras with at least partially overlapping fields of view, and / or left and right stereo images acquired at offset locations and / or timestamps by a single monocular camera) is fed as input to a stereo depth perception model, which then makes a prediction for a depth image (e.g.,a disparity map) – taking into account ego movement between frames in time-shifted depth image execution forms. Each pixel of the depth image has a value representing information about a depth measurement from the stereo image pair to a surface in the scene represented by the pixel. A depth image anomaly identification stage of a depth image anomaly processor can perform functions to evaluate a depth image in order to identify regions (e.g., surfaces of segments) where the values ​​of pixels (predicted by a stereo depth perception model) appear anomalous in accuracy compared to other regions of the depth image. For example, in a visual rendering of a depth image (e.g.,(when pixel depth values ​​are translated into pixel color values) an object sitting on a surface may appear visually normal, but contain numerous anomalies with respect to intrinsic and / or contextual consistencies if the colors are understood to represent depth data.

[0006] A depth image anomaly identification stage of a depth image anomaly processor can employ image segmentation techniques, such as one or more image segmentation machine learning models. Image segmentation is a computer vision technique (which can be performed, for example, using image segmentation models and / or other deep learning models) used in object detection tasks, partitioning regions of pixels in an image into distinctive image segments according to prominent features. The depth image anomaly identification stage can then apply one or more feature classification models to distinctive image segments to infer one or more image space classifications for each segment, indicating whether a segment contains features known to contribute to depth image anomalies.Segments containing potentially anomalous depth values ​​can be evaluated based on the image space classification(s) and / or the depth values ​​of one or more adjacent (unsuspected) segments. For example, if a surface in a potentially anomalous segment is classified as a feature that is physically attached to and / or touches one or more surfaces of adjacent segments, and there is a significant discontinuity in depth values ​​at pixels where these surfaces intersect, the potentially anomalous segment of the depth image can be relabeled as an identified anomalous segment. In some embodiments, an image segmentation model can aid in identifying those segments that exhibit potentially anomalous depth values.The identification of anomalous segments in a predicted depth image can be aided by specifying depth value inconsistencies, which can be provided to the anomaly detection system via human input into a human-machine interface (HMI). The identification of regions (e.g., segments) with anomalous depth values ​​can be used to calculate an accuracy score or rating to assess the quality of the stereo depth perception model, or to otherwise quantify the quality of the depth image output for use, for example, as ground truth for training machine learning models and / or other purposes.

[0007] In some embodiments, the process can progress from the identification stage to a correction stage that adjusts the depth image based on the identified anomalous segments. The anomaly detection system can employ one or more correction techniques that can calculate estimates or predictions for depth values ​​within identified anomalous segments, for example, based on contexts provided by depth values ​​from non-anomalous segments, based on detected structural characteristics within the spatial volume corresponding to the identified anomalous segments, and / or based on characteristics of local phenomena (e.g., illumination, reflections, glare, etc.) that have been identified as causing the inaccuracies in depth values.In some embodiments, the surface fitting of depth values ​​to correct anomalies may be subject to one or more structural constraints based on the geometry of the overall scene, as illustrated in the stereo image pair. The correction stage of the anomaly detection system can output a corrected depth image, where the anomalous depth values ​​of one or more identified anomalous segments are corrected using adjusted depth values ​​calculated based on interpolation and / or fitting algorithms as described herein. The corrected depth image can then be included with the stereo image pair as a training data sample (e.g., for training and / or evaluating a machine learning model). The corrected depth image can establish a ground-truth depth image, which can be used to assess the accuracy of a predicted depth image generated by a (e.g.,A machine-learned stereo depth perception model is generated.

[0008] Further features of the disclosure are characterized by the independent and dependent claims.

[0009] Any feature in one aspect of the disclosure can be applied to other aspects of the disclosure in any suitable combination. In particular, procedural aspects can be applied to device or system aspects and vice versa.

[0010] Furthermore, features implemented in hardware can also be implemented in software, and vice versa. Any reference to software and hardware features herein should be interpreted accordingly.

[0011] Each system or device feature as described herein can also be provided as a process feature, and vice versa. Functionally described system and / or device aspects (including means plus functional features) can alternatively be expressed in terms of their corresponding structure, such as a suitably programmed processor and associated memory.

[0012] It should also be understandable that certain combinations of the various features described and defined in any aspect of the revelation can be implemented and / or provided and / or used independently of one another.

[0013] This disclosure also provides computer programs and computer program products comprising software code adapted, when executed on a data processing device, to perform any of the procedures described herein and / or to embody any of the device and system features described herein, including any or all of the partial steps of a procedure.

[0014] The disclosure also provides a computer or computing system (including networked or distributed systems) that includes an operating system supporting a computer program for performing any of the procedures described herein and / or embodying any of the device or system features described herein.

[0015] The revelation also provides a computer-readable medium on which one or more of the aforementioned computer programs are stored.

[0016] The revelation also provides a signal that carries one or more of the aforementioned computer programs.

[0017] The disclosure extends to processes and / or devices and / or systems as described herein with reference to the accompanying drawings.

[0018] Aspects and embodiments of the disclosure will now be described exclusively by way of example with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The present systems and methods for depth image analysis and correction for stereo image machine learning are described in detail below with reference to the attached drawings, wherein: Fig. 1 an exemplary data flow representation of a process for stereo image-based depth value anomaly detection in accordance with some embodiments of the present disclosure; Fig. 2A is a data flow representation illustrating a depth image anomaly identification stage for a depth image anomaly processor in accordance with some embodiments of the present disclosure; Fig. 2B is a data flow diagram illustrating an exemplary implementation of a depth image anomaly correction stage for a depth image anomaly processor in accordance with some embodiments of the present disclosure; Fig. 3A-3C illustrate an exemplary process for depth image anomaly correction in accordance with some embodiments of the present disclosure; Fig. 4 is a representation illustrating examples of a model-restricted surface adaptation in accordance with some embodiments of the present disclosure; Fig. 5 illustrates an exemplary training process for training a stereo depth perception model in accordance with some embodiments of the present disclosure; Fig. Figures 6A-6B illustrate an exemplary process for evaluating the accuracy of one or more stereo depth perception models using one or more depth image anomaly processors in accordance with some embodiments of the present disclosure; Fig. 7 is a flow diagram illustrating an exemplary method 700 for depth image anomaly processing in accordance with some embodiments of the present disclosure; Fig. 8 is a flow diagram illustrating an exemplary method 800 for generating a depth image from a stereo image pair in accordance with some embodiments of the present disclosure; Fig. 9A is an illustration of an exemplary autonomous vehicle in accordance with some embodiments of the present disclosure; Fig. 9B is an example of camera locations and fields of view for the exemplary autonomous vehicle of Fig. 9A in accordance with some embodiments of the present disclosure; Fig. 9C a block diagram of an exemplary system architecture for the exemplary autonomous vehicle of Fig. 9A in accordance with some embodiments of the present disclosure; Fig. 9D a system representation in accordance with some embodiments of the present disclosure for communication between a cloud-based server(s) and the exemplary autonomous vehicle of Fig. 9A is; Fig. 10 is a block representation of an exemplary computing device suitable for use in implementing some embodiments of the present disclosure; and Fig. 11 is a block representation of an exemplary data center suitable for use in implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0020] This document discloses systems and methods relating to depth image analysis and correction for stereo image machine learning systems and applications. Although the present disclosure relates to an exemplary autonomous or semi-autonomous vehicle or machine 900 (hereinafter referred to alternatively as "Vehicle 900" or "Ego-Machine 900"), an example of which is given with reference to Fig. The fact that the descriptions may be limited to (9A-9D) is not intended to be restrictive. For example, the systems and procedures described herein may be used without restriction by non-autonomous vehicles or machines, semi-autonomous vehicles or machines (e.g., in one or more Advanced Driver Assistance Systems (ADAS)), autonomous vehicles or machines, guided and unguided robots or robotic platforms, warehouse vehicles, all-terrain vehicles, vehicles coupled to one or more trailers, flying vehicles, boats, shuttle vehicles, emergency vehicles, motorcycles, electric or motorized bicycles, airplanes, construction vehicles, trains, remotely controlled vehicles such as drones, and / or other vehicle types.Furthermore, although the present disclosure may be described with reference to stereovision-based depth perception for autonomous driving, this is not intended to be limiting, and the systems and methods described herein may be used for augmented reality, virtual reality, mixed reality, robotics, security and surveillance, autonomous or semi-autonomous machine applications and / or any other technology spaces where stereovision-based depth perception can be used.

[0021] This disclosure relates to stereo-image-based computer vision technologies. In particular, the systems and methods provided in this disclosure provide technologies for evaluating and correcting machine learning model depth images generated from stereo image pairs, which can be used in computer vision and perception-based systems.

[0022] Camera-based technologies for generating depth images are typically less expensive than 3D LiDAR-based technologies, and while they typically provide less accurate geometric information than 3D LiDAR, cameras provide a large amount of semantically relevant data that can be used, for example, to improve navigation and provide feature classification. However, vision-based perception using camera-captured images presents a significant challenge, partly due to the complexity of the algorithms involved and partly due to the computational resources required to run them.

[0023] Stereo disparity, or binocular disparity, is a geometric term used in the field of computer vision that refers to the difference in the image pixel location of an object as seen in left and right stereo images captured by a pair of cameras. The numerical value of a calculated disparity reflects the object's depth away from the cameras, with the greater the disparity, the closer the object. For example, a pair of image sensors O, O' on an ego machine can be set up with their respective optical axes aligned parallel to capture a forward-facing stereo image pair of the ego machine's path. An object point X captured by the pair of sensors will have a pixel location at (x, y) in the image captured by O and a pixel location at (x', y') in the image captured by O'. The images can be rectified to the point where the image lines are aligned such that y = y'.The disparity, d, for a pixel is then defined by the pixel position difference: d = x - x'. Furthermore, according to the similar triangulation principle, the disparity is proportional to the sensor focal length, f, and the basilin length, b, between the two sensors and is inversely proportional to the object depth, Z, which can be expressed as: d = b * f / Z. As such, a disparity map can be created for the image pair, in which each element (e.g., each pixel) of the disparity map represents a disparity for a corresponding element of the image captured by O. The resulting image space disparity map can be of the same size in terms of rows and columns, since the captured images represent a depth image of a scene as captured by a pair of image sensors.

[0024] State-of-the-art (SotA) deep learning-based machine learning models for stereo-based depth estimation (stereo depth perception models) represent an emerging computer vision technology that largely avoids the computational complexities of calculating image space disparity maps by training a deep learning model to predict depth reconstruction for a scene from a pair of stereo images. Nevertheless, training such models requires large sets of training data, and collecting and curating a collection of depth image ground truths for an adequate training dataset is a challenging task.

[0025] Several methods for creating an accurate depth image ground truth for training a set of data samples include LiDAR, structured and unstructured lighting techniques, and multi-view scene reconstruction techniques.

[0026] LiDAR uses light beams to produce point cloud data and is capable of obtaining long-range depth measurements. However, depth images produced from LiDAR measurements can exhibit small fill factors, errors related to parallax and timing differences, errors related to motion (both static and dynamic objects), and pixel-plane alignment accuracy. Structured lighting techniques generate depth data based on projecting a known pattern of light (e.g., a grid) onto a scene. However, this technique typically only works for short-range measurements and indoor environments with controlled lighting and is susceptible to errors caused by real-world effects, such as very dark objects or glossy surfaces. This technique also typically has a long acquisition time and therefore cannot capture moving scenes.Unstructured lighting techniques project arbitrary patterns onto the scene and then rely on another computational technique (such as triangulation) to determine depth. Unstructured lighting techniques also typically only work for short-range and / or indoor measurements and have resolutions limited by the projected pattern resolution. Both structured and unstructured lighting techniques can be susceptible to errors caused by real-world effects, such as very dark objects or glossy surfaces. Multi-view 3D scene reconstruction, such as neural radiance fields (NeRF), 3D Gaussian splatting, and / or other techniques for rendering 3D volumes from 2D images, is typically limited to static scenes and can be inconsistent in terms of accuracy and resolution.Furthermore, an intrinsic part of the problem is not only the accuracy provided by these different technologies, but also the scale at which ground-truth depth images can be collected with them. For example, if data acquisition using these earlier techniques takes a long time, even if it is very accurate, it remains a challenge to scale up enough to collect a sufficient corpus of ground-truth data samples for training a stereo depth perception model.

[0027] In contrast to previous techniques for generating depth images for ground-truth samples, embodiments of the depth image analysis and correction systems described herein can generate ground-truth depth images based on depth images generated by a stereo depth perception model from a pair of stereo images of a scene. As described herein, in some embodiments, a process may include a depth image anomaly identification stage, which may be followed by a depth image correction stage. Furthermore, in some embodiments, the techniques described herein can be used for objectively evaluating (e.g., scoring) the accuracy of depth images produced by different stereo depth perception models, thus enabling a comparison of the quality of the different stereo depth perception models.

[0028] In some embodiments, a stereo image pair (e.g., left and right stereo images captured by a pair of cameras, and / or left and right stereo images captured at offset locations by a single monocular camera) is fed as input to a stereo depth perception model, which then outputs a prediction for a depth image (e.g., a disparity map). Each pixel of the depth image has a value representing information about a depth measurement from the stereo image pair to a surface in the scene represented by the pixel. The resulting depth image can have the same size in terms of resolution and / or row and column coordinates as the captured images of the stereo image pair, so that 3D depth information about a feature appearing at a given pixel in the stereo image pair can be determined based on the value of the corresponding pixel in the predicted depth image.

[0029] In some embodiments, a depth image anomaly identification stage of a depth image anomaly processor can perform functions to evaluate a depth image in order to identify regions (e.g., surfaces of a segment) where the values ​​of pixels (predicted by a stereo depth perception model) appear anomalous in accuracy compared to other regions of the depth image. For example, during a visual rendering of a depth image (e.g., when pixel depth values ​​are translated into pixel color values), an object sitting on a surface may appear visually normal but contain numerous anomalies with respect to intrinsic and / or contextual consistencies when the colors are understood to represent depth data.This means that in a depth image, the pixel values ​​of the base of an object placed on a surface should be approximately the same as the adjacent pixel values ​​of the area of ​​the surface where the base touches the object (e.g., because they are both approximately equidistant from the stereo image sensors used to generate the depth image). Anomalies may appear as occurrences of depth value change rate discontinuities along a surface that are otherwise easily distinguishable from a stereo image pair with a smooth, continuous surface. Other anomalies may appear as inconsistencies and / or discontinuities in depth values ​​along transitions between connected and / or parallel surfaces (e.g., discontinuities in depth values ​​of regions near a point where a ceiling (or floor) and a wall meet).In many cases, pixels corresponding to a smooth surface extending to a focal point (e.g., a horizon) and appearing relatively uniform in image space can provide low disparity between image pairs for a stereo depth perception model to work with. Such surfaces can therefore appear in depth image prediction as a relatively uniform field of pixel values, as opposed to a gradient of pixel values ​​that vary in the direction of the focal point. In some cases, surface characteristics, reflections from images of surfaces, and / or illumination features present in the image space (e.g., the real-world scene represented by optical image data) of a stereo image pair are factors known to cause depth prediction errors in depth images generated with stereo depth perception models.As an example of another type of accuracy anomaly, depth values ​​associated with a feature may not truly be consistent with each other, such as significant discontinuities in depth between a base, side, and top of an object, which would imply that these regions of the object are not connected and / or immediately adjacent to each other.

[0030] As such, an understanding of the physical structure of a scene captured by the stereo image pair can be informative for the task of assessing the accuracy of a depth image produced from the stereo image pair. Based on an understanding of the types of features (e.g., objects, surfaces, geometries, structures, etc.) present in a scene as captured by a stereo image pair, features and factors known to contribute to depth image anomalies can be identified and segmented from other regions, allowing for a more efficient assessment of pixel values ​​in depth images corresponding to these features.

[0031] For example, a depth image anomaly identification stage of a depth image anomaly processor can use image segmentation techniques, such as employing one or more image segmentation machine learning models. Image segmentation is a computer vision technique (which can be performed, for example, using image segmentation models and / or other deep learning models) used in object detection tasks, partitioning regions of pixels in an image into distinctive image segments according to prominent features. The depth image anomaly identification stage can then apply one or more feature classification models to distinctive image segments to infer one or more image space classifications for each segment, indicating whether a segment contains features known to contribute to depth image anomalies.A feature can be classified, for example, using a lighting-based artifact image space clarification, such as an optical reflection or a light source, or as a feature associated with curved or planar surfaces (e.g., a floor, a wall, a ceiling, a staircase, a corner, a hallway, etc.) that extend back to a focal point in an image space, and the depth image anomaly identification stage then locates (and / or labels) the corresponding segment of pixels in the depth image for further evaluation with respect to the accuracy of their depth values.

[0032] In some embodiments, segments containing potentially anomalous depth values ​​can be evaluated based on the image space classification(s) and / or the depth values ​​of one or more adjacent (unsuspected) segments. For example, if a surface in a potentially anomalous segment is classified as a feature that is physically attached to and / or touches one or more surfaces of adjacent segments, and there is a significant discontinuity in depth values ​​at pixels where these surfaces intersect, the potentially anomalous segment of the depth image can be relabeled as an identified anomalous segment.

[0033] In some implementations, an image segmentation model can help identify segments that potentially exhibit anomalous depth values. For a depth image in a 3D scene, there should generally be a gradient in pixel depth values, where depth values ​​gradually increase towards an optical focal point. For example, for a ceiling or floor extending away from the camera, the depth image should show a continuous increase in depth. Therefore, if a segment lacks such an expected gradient in depth values ​​and can be associated with a segment of the optical image where a gradient should exist, that segment of the depth image can be labeled as an identified anomalous segment.In some embodiments, the identification of anomalous segments in a predicted depth image can be aided by specifying depth value inconsistencies, which can be provided to the anomaly detection system via human input into a human-machine interface (HMI). For example, the anomaly detection system can display one or both images of a stereo image pair and a visual rendering of the depth image predicted by the stereo depth perception model on the HMI. In some embodiments, segmentation images of the stereo image pair and a visual rendering of the depth image can be displayed. In some embodiments, the conversion of the depth image to a point cloud can be displayed, since certain features are easier to detect in 3D than in 2D.A user viewing images on the HMI can select one or more regions of pixels that they perceive as potentially anomalous, and this indication of one or more regions can be included as input to the anomaly detection system for evaluating and / or defining identified anomaly segments. As illustrated by these non-restrictive examples, segments that include depth values ​​lacking value patterns consistent with the image space classification of that segment, and / or when evaluated in the context of neighboring features, can be identified (and labeled) as exhibiting inconsistencies caused by inaccuracies in the depth predictions of the stereo depth perception model.

[0034] For certain use cases aimed at machine learning model development, the identification of regions (e.g., segments) of anomalous depth values ​​can be used to calculate an accuracy score or rating to assess the quality of the stereo depth perception model or otherwise to quantify the quality of the depth image output for use, for example, as ground truth for training machine learning models and / or other purposes. For instance, the distribution and / or severity of anomalous depth values ​​in a depth image produced by a model (e.g., by a depth image quality algorithm) can be quantified into an objective accuracy score.Identified anomaly-segment-based accuracy scores can be iteratively calculated between stereo depth perception model training sessions to determine whether continued training improves the accuracy of predictions made by the stereo depth perception model. In some embodiments, a set of stereo image pair evaluation samples can be fed to a set of different stereo depth perception models undergoing evaluation, and the depth image outputs of each model can be compared with respect to identified anomalous segments and / or anomaly-segment-based accuracy scores to rank the respective accuracy of the models.

[0035] In some embodiments, the process can progress from the identification stage to a correction stage that adjusts the depth image based on the identified anomalous segments. The anomaly detection system can employ one or more correction techniques that can calculate estimates or predictions for depth values ​​within identified anomalous segments, for example, based on contexts provided by depth values ​​from non-anomalous segments, based on detected structural characteristics within the spatial volume corresponding to the identified anomalous segments, and / or based on characteristics of local phenomena (e.g., illumination, reflections, glare, etc.) that have been identified as causing the inaccuracies in depth values.

[0036] In some embodiments, for certain use cases aimed at inference applications, the identification of regions (e.g., segments) of anomalous depth values ​​in the depth image output from a stereo depth perception model can be used to mask the identified anomalous segment so that inaccurate depth value data is not provided to downstream processes (e.g., where a lack of depth data is less detrimental to the downstream process than receiving poor depth data).

[0037] As another example, an identified anomalous segment of the depth image may be associated with an image space segment that includes one or more illumination-based artifacts, such as (but not limited to) light cast by illumination sources, shadows, and instances of reflections and / or glare appearing in a stereo image pair. Illumination-based artifacts in the stereo image pair may include the validity of surface information for flat surface geometries available to the stereo depth perception models, potentially resulting in inaccuracies in depth predictions (discontinuities, inconsistencies, inaccurate depth values, etc.) in segments affected by the artifacts.As such, in some embodiments, the anomaly detection system can apply infill and / or interpolation functions to adjust inaccurate depth values ​​based on depth values ​​from one or more adjacent non-anomalous segments. For example, a depth-value infill algorithm can use a set of depth-value key points selected from non-anomalous segments near the identified anomalous segment and interpolate these depth values ​​inward to fill the anomalous segment with interpolated depth values ​​that blend seamlessly with the non-anomalous segments. In some embodiments, the selected depth-value key points can include three or more key points that define a polygon which at least partially overlaps the anomalous segment. Depth value interpolation can then be performed based on at least a multi-point infill of the polygon.

[0038] In some embodiments, other corrective functions can utilize structural geometries identified within the spatial volume corresponding to the identified anomalous segments and fit interpolated depth values ​​to surfaces to generate the corrected depth values ​​applied to the anomalous segment. In some embodiments, a surface shape classification can be determined for an identified anomalous segment, and subsequently, a corresponding shape can be selected for fitting based on the classification. For example, if an identified anomalous segment falls within the boundaries of a flat surface (such as a floor, ceiling, wall, tabletop, or the like), the corrected depth values ​​can then be calculated based on linear interpolation and / or extrapolation from the selected non-analogous depth value key points.In some embodiments, if an identified anomalous segment corresponds to a curved surface, the interpolation and / or extrapolation of depth values ​​from the selected non-analogous depth value key points can be adjusted to fit a curved surface, for example, using a polynomial-based fitting algorithm.

[0039] Corrected depth values ​​based on surface fitting to individual surfaces can provide locally consistent depth values, but in some cases, they can result in misalignments between features when pixels of a corrected depth image are projected into 3D space. That is, in some cases, when corrections are applied solely based on fitting extrapolations to planes, there is a lack of enforcement of certain constraints—such as enforcing constraints when points on a first surface are actually in contact with points on a second surface, or constraints on surface positions where one plane intersects another along a line.As such, in some embodiments, surface adjustments of depth values ​​to correct anomalies may be subject to one or more structural constraints based on the geometry of the overall scene, as illustrated in the stereo image pair. For example, for a hallway scene, adjustments may be constrained based on a generalized architectural model of a hallway comprising five levels: a floor level, a ceiling level, a left-wall level, a right-wall level, and an end-of-hallway back level. In such an architectural model of a hallway, the surfaces of the left and right walls may be defined as parallel to each other and intersecting with the floor and / or ceiling levels, and the floor and / or ceiling levels may be defined as parallel to each other.If interpolated depth values ​​are subsequently fitted to levels / surfaces to calculate corrected depth values, the position and / or orientation of these levels to which they are fitted are constrained relative to each other based on the selected generalized architectural model (e.g., a corridor model in this example).

[0040] In other cases, a scene illustrated as in a stereo image pair may include an inside corner or an outside corner, for example, where two walls and a floor intersect to form three mutually orthogonal surfaces. Again, interpolated depth values ​​are fitted to the walls and / or floor, and the position and / or orientation of these planes to which they are fitted are constrained with respect to a generalized architectural model representing this specific geometry. In some embodiments, the generalized architectural model selected for fitting to calculate corrected depth values ​​can be chosen from a structural model library of available generalized architectural models, based at least on an image space classification of the scene illustrated by the stereo image pairs.For example, a classification model applied to the stereo image pairs can infer one or more contextual classifications of the scene (e.g., corridor, room, atrium, staircase, theater, auditorium, etc.) and one or more generalized architectural models, which are selected to apply the most relevant set of constraints to fit in order to compute corrected depth values ​​for one or more identified anomalous segments of the depth image.

[0041] In some embodiments, the correction stage of the anomaly detection system can output a corrected depth image, where the anomalous depth values ​​of one or more identified anomalous segments are corrected using adjusted depth values ​​calculated based on interpolation and / or fitting algorithms as described herein. The corrected depth image can then be included with the stereo image pair as a training data sample (e.g., for training and / or evaluating a machine learning model). The corrected depth image can establish a ground-truth depth image, which can be used to assess the accuracy of a predicted depth image generated by a stereo depth perception model.For example, in a training process, a loss function can calculate the feedback loss for adjusting a stereo depth perception model based on one or more discrepancies between a predicted depth image generated by the stereo depth perception model and the ground-truth depth image. The stereo depth perception model can be adjusted iteratively to minimize the feedback loss. Alternatively, such a loss function can be used to calculate an accuracy score or rating to assess the quality of the stereo depth perception model and track improvements in the model during a development process.

[0042] With reference to Fig. 1 is Fig. 1 An exemplary data flow diagram of a process for a stereoanomaly detection system 100 in accordance with some embodiments of the present disclosure. It should be understood that this and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and other elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and position. Various functions described herein as being performed by entities may be executed by hardware, firmware, and / or software.For example, various functions can be performed by one or more processors (which may include, for example, processing circuitry) that execute instructions stored in memory. In some embodiments, the systems described herein can employ methods and processes using components, features, and / or functionalities similar to those of the exemplary autonomous vehicle 900 from [source missing]. Fig. 9A-9D, the exemplary computing device 1000 of Fig. 10 and / or the exemplary data center 1100 of Fig. 11 similar, will be executed.

[0043] As in Fig. As shown in Figure 1, a stereo anomaly detection system 100 can include a depth image anomaly processor 130 that inputs depth image 122 data (produced by a stereo depth perception model 120 from a stereo image pair 112) and can generate a sample of stereo image pair training data 160 comprising a corrected depth image 162. In some embodiments, optical image data 110 comprising a stereo image pair 112 can be produced by one or more sensors 102 (e.g., camera(s)). A stereo image pair 112 can comprise a pair of images simultaneously acquired by a set of paired image sensors from offset viewpoints and / or a pair of images acquired by a single image sensor from two offset positions.An image sensor(s) 102 can be, for example, red, green, blue (RGB) cameras, infrared (IR) cameras, RGB-IR cameras, stereo camera groups, depth cameras and / or other cameras, such as cameras related to the vehicle 900 of . Fig. 9A-9D described, include. The image sensor(s) 102 may include one or more cameras of an ego object or ego actuator, such as a stereo camera(s) 968, wide-angle camera(s) 970 (e.g., fisheye camera(s)), infrared camera(s) 972, surround-view camera(s) 974 (e.g., 360° cameras), occupant monitoring system (OMS) sensor(s) 901, and / or long- and / or medium-range camera(s) 998 of the autonomous vehicle 900. Fig. 9A-9D. The image sensor(s) 102 can therefore be used to generate the image data 110 of a three-dimensional environment (3D environment) around an ego object or an ego actuator.

[0044] In some embodiments, the depth image anomaly processor 130 may include a depth image anomaly identification stage 132. The stereo depth perception model 120 inputs a stereo image pair 112 to produce a prediction that includes the depth image 122. The depth image anomaly identification stage 132 includes depth anomaly detection logic 136 that evaluates the depth image 122 against the stereo image pair 112 to identify segments of the depth image 122 that are predicted to contain pixels with inaccurate depth values. The depth image anomaly identification stage 132 may output a labeled depth image 140 representing a version of the depth image 122, with segments identified as having inaccurate depth values ​​being tagged with labels indicating this determination.

[0045] For example, Fig. Figure 2A shows a data flow diagram illustrating a depth image anomaly identification stage 132. As shown in this figure, the depth anomaly detection logic 136 can, for example, include one or more image segmentation models 212, one or more image segment classification models 214, and / or a segment labeling function 216. In some embodiments, the functions of the image segmentation model(s) 212 and image segment classification model(s) 214 can be integrated into a combined machine learning model that generates segmentations and assigns one or more classifications to regions of the depth image 122.The outputs of the image segmentation model(s) 212 and the image segment classification model(s) 214 generate data that can characterize the physical structures and / or features of a scene captured by the stereo image pair 112, which assists the depth anomaly detection logic 136 in assessing the accuracy of a depth image produced from the stereo image pair 112. For example, the image segmentation model(s) 212 can segment the stereo image pair 112 into regions of pixels that correspond to distinct physical features (e.g., objects, surfaces, geometry, structures, etc.) present in a scene. The depth anomaly detection logic 136 can correlate segments identified from the stereo image pair 112 with corresponding pixels of a depth image 122 in order to associate depth data with specific features represented in a segment.In some embodiments, the depth anomaly detection logic 136 itself may include and / or be implemented using a machine learning model that is trained to identify and classify segments of the depth image 122 containing anomalous (e.g., inaccurate) depth values, based on evaluating feature segments extracted from the stereo image pair 112 and depth values ​​of corresponding pixels of the depth image 122. In some embodiments, the image segment classification model(s) 214 may evaluate segments of the depth image 122 generated by the image segmentation model(s) 212 to detect features exhibiting characteristics known to contribute to depth image anomalies and to apply a classification (e.g., a tag, label, etc.) to these segments based on the detected characteristics.That is, the image segment classification model(s) 214 can infer one or more classifications for an individual segment that can indicate whether a segment includes features known to contribute to depth image anomalies. For example, the image segment classification model(s) 214 can classify, based on an inference that a feature includes an optical reflection or a light source, or as a feature associated with curved or planar surfaces (e.g., a floor, wall, ceiling, stairs, corner, hallway, etc.) that extend back to a focal point in image space.The depth anomaly detection logic 136 can then locate the corresponding segment of pixels in the depth image and specifically evaluate those segments that are labelled to include features that are prone to causing anomalous depth data with respect to the accuracy of their depth values, based on the classification(s).

[0046] For example, in some embodiments, the depth anomaly detection logic 136 can evaluate segments containing potentially anomalous depth values ​​based on the classification(s) and / or depth values ​​of one or more adjacent (unsuspected) segments. If a surface in a potentially anomalous segment is classified as a feature that is physically attached to and / or touching one or more surfaces of adjacent segments, and the depth anomaly detection logic 136 detects a significant discontinuity in depth values ​​at pixels where these surfaces intersect, the potentially anomalous segment of the depth image can be relabeled as an identified anomalous segment. In some embodiments, the depth anomaly detection logic 136 can perform functions to identify regions of the depth image 122 (e.g.,to identify surfaces of segments) where the values ​​of pixels do not follow an expected pattern of depth values ​​when compared with other regions of the depth image 122.

[0047] For example, illustrated Fig. Figure 3A shows an exemplary camera image 310 of a scene (e.g., a scene represented by a stereo image pair 112) and a corresponding exemplary depth image 312 (e.g., a depth image 122 of the same scene, generated by a stereo depth perception model 120 from the stereo image pair 112). In this example, region 320 of the camera image 310 represents the ground, and the corresponding region 330 of the depth image 312 comprises a gradation of values, with pixels representing the ground surface closer to the image sensor having smaller values ​​that gradually increase as the distance of the ground surface from the sensor increases—an expected (non-anomalous) pattern for a ground. However, anomalous such surfaces can appear as discontinuities in a depth value pattern across a region of pixels.For example, optical reflections on the ground surface at 322 are evident from camera image 310, resulting in a depth value discontinuity and / or inconsistency at 332 of depth image 312. In this case, both the ground segment 320 and the reflection segment at 322 can be classified as features of the same continuous ground surface, and the significant discontinuity in the corresponding depth values ​​where these features meet (e.g., more than one threshold) can trigger the depth anomaly detection logic 136 to label depth value segment 332 as an identified anomalous segment of depth image 312.

[0048] Furthermore, it would be expected that the ceiling area 323 illustrated in camera image 310 would produce a gradation of depth values ​​in the depth image 312, with pixels for the ceiling surface closer to the image sensor having smaller values ​​that gradually increase as the distance of the ceiling surface from the sensor increases. Instead, however, the depth image 312 produced by the stereo depth perception model 120 includes corresponding regions on 333 that appear as triangular segments where the depth values ​​are relatively homogeneous and the expected pattern of depth value gradation is absent.In this case, ceiling area 323 can be classified as a ceiling surface, and the depth anomaly detection logic 136 detects that the corresponding depth values ​​for region 333 of the depth image 312 do not match an expected pattern for a ceiling surface. This can trigger the depth anomaly detection logic 136 to label the depth value segment 333 as an identified anomalous segment. That is, if a segment lacks such an expected gradient in depth values ​​and this can be associated with a segment of an optical image pair where a gradient should exist, such a segment of the depth image can be labeled as an identified anomalous segment.

[0049] Another set of anomalies can be observed in the region of surface 326 of camera image 310, which represents a series of continuous sections of a glass wall forming a surface extending along the corridor between the floor surface 320 and the ceiling surface 323. In this case, a depth value discontinuity 337 is evident in the depth image 312, caused by a door handle 327 located on the other side of the glass wall surface 326, as well as discontinuities 338 caused by optical reflections 328, which are evident in camera image 310. In each of these cases, the depth values ​​corresponding to the area of ​​the door handle 327 and / or optical reflections 328 do not match expected patterns, which can trigger the depth anomaly detection logic 136 to label these segments 337 and 338 of depth values ​​as identified anomalous segments.

[0050] As previously mentioned, anomalies other than inconsistencies and / or discontinuities in depth values ​​along transitions between connected and / or parallel surfaces may appear, such as discontinuities where a wall and floor or a wall and ceiling physically meet, or other cases where depth values ​​associated with a feature may not be internally consistent, such as significant depth discontinuities between the base, side, and top of an object, which would imply that these regions of the object are not connected and / or immediately adjacent to each other. In the example of Fig. 3A is the underside of the glass wall surface 326 physically connected to the floor surface 320, so points along the intersection of these surfaces (shown at 329) should correspond to their depth values ​​in the depth image 312. That is, the gradient pattern of depth values ​​along the glass wall surface 326 should follow the gradient pattern of depth values ​​along the floor surface 320. However, as shown in the depth image 312, the corresponding depth values ​​at segment 339 lack the expected correspondence in depth values. As such, the depth values ​​corresponding to the intersection of these surfaces do not match expected patterns, which can trigger the depth anomaly detection logic 136 to label these segments of depth values ​​as an identified anomalous segment.

[0051] In some embodiments, the depth anomaly detection logic 136 can be assisted in identifying anomalous segments in a predicted depth image 122 based on inconsistency information provided as input by a human user to a human-machine interface (HMI) 124. For example, the anomaly detection system on the HMI 124 can display one or both images of a stereo image pair 112 and a visual rendering of the depth image 122 predicted by the stereo depth perception model 120. In some embodiments, the conversion of the depth image to a point cloud can be displayed, since certain features are easier to detect in 3D than in 2D.A user viewing the images on the HMI 124 can select one or more regions of pixels that they perceive as potentially anomalous, and this specification of one or more regions can be included as input in the depth anomaly detection logic 136, which is used to define identified anomaly segments.

[0052] As illustrated by these non-restrictive examples, segments that include depth values ​​that do not exhibit value patterns consistent with the classification of that segment in an image space, and / or when evaluated in the context of neighboring features, can be identified (and labeled) by the depth image anomaly identification level 132 as exhibiting inconsistencies caused by inaccuracies in the depth predictions of the stereo depth perception model.

[0053] Based on at least one or more identified anomaly segments identified by the depth anomaly detection logic 136, the segment labeling function 216 can generate a labeled depth image 140, which includes an updated version of the depth image 122 containing labels that tag the one or more identified anomaly segments as identified anomaly segments, as shown in the exemplary labeled depth image 340 in Fig. 3B illustrates. As in Fig. As shown in Figure 3B, the segment labeling function 216 can update the initial depth image 312 to generate the labeled depth image 340, which includes one or more segments labeled to contain identified anomalous segments (shown in Figure 342). In some embodiments, the segment labeling function 216 can further label segments to include metadata based on the classifications inferred by the segment classification model(s) 214. For example, identified anomalous segments 342 can further be labeled to indicate that they include optical reflections (343), large flat surfaces (344), or glass or other transparent features (345), and / or can be labeled with other metadata that can help identify the nature of the feature and / or the cause of inaccurate depth predictions for that feature.As discussed below, in one embodiment, labeling and / or metadata can be used to perform depth image anomaly correction. A labeled depth image(s) 140 produced by the depth image anomaly identification stage 132 can be stored in a labeled image data storage 220 (e.g., a database, a data store, etc.) so that it can be retrieved for subsequent analysis, used as data samples for training machine learning models, and / or for other purposes.

[0054] In some embodiments, a labeled depth image(s) 140 can be used to compute an accuracy score or accuracy rating to assess the quality of the stereo depth perception model or to otherwise quantify the quality of the depth image output for use, for example, as ground truth for training machine learning models and / or other purposes. For example, the depth image anomaly processor 130 can include a model accuracy scoring function 150 that takes the labeled depth image(s) 140 and / or other data produced by the depth image anomaly identification stage 132 as input and outputs a model accuracy score 152 that can be presented via the HMI 124. For example, the distribution and / or severity of anomalous depth values ​​in a depth image produced by a model (e.g.,anomaly segment-based accuracy scores can be quantified using a depth image quality algorithm. Identified anomaly segment-based accuracy scores can be iteratively calculated between stereo depth perception model training sessions to determine whether continued training improves the accuracy of predictions made by the stereo depth perception model.

[0055] Returning to Fig. 1 In some embodiments, the labeled depth image(s) 140 can be applied to a depth image anomaly correction stage 134, which produces a corrected depth image 162 by applying adjustments to the depth image 122 based on the identified anomalous segments 342. The depth image anomaly correction stage 134 can include segment correction logic 138, which applies one or more correction techniques to one or more of the identified anomalous segments 342 to mitigate at least partial inaccuracies in depth values ​​in these segments. For example, the segment correction logic 138 can include algorithms that compute estimates or predictions of what the depth values ​​within these identified anomalous segments should be. For example, the segment correction logic 138 can compute corrected depth values ​​based on the context associated with an identified anomalous segment (e.g.,based on depth values ​​of adjacent non-anomalous segments, based on detected structural characteristics within the spatial volume corresponding to an identified anomalous segment, and / or on characteristics of local phenomena (e.g., illumination, reflections, glare, etc.) that have been identified as causing the inaccuracies in depth values. In some embodiments, the depth image anomaly correction stage 134 can output a stereo image pair training data sample 160 comprising the stereo image pair 112 and a corresponding corrected depth image 162 derived from the stereo image pair 112 by the depth image anomaly processor 130. Such a training data sample 160 can be used to train machine learning models, such as the stereo depth perception model 120, with the corrected depth image 162 providing a ground-truth sample (e.g.,an image representing the depth image that should be predicted by the model based on an input comprising the stereo image pair 112).

[0056] Fig. Figure 2B is a data flow diagram illustrating an exemplary implementation of a depth image anomaly correction stage 134. As shown in this figure, the segment correction logic 138 may, for example, include one or more image processing algorithms 240, one or more segment masking functions 242, one or more surface adaptation algorithms 246, a structural model library 248, and one or more geometry context classification models 250.

[0057] In some embodiments, the segment correction logic 138 may include one or more image processing algorithms 240, which include image processing tools and / or filters that apply infill and / or interpolation-based adjustments to a segment of a glabeled depth image 140 to correct inaccurate depth values ​​based on depth values ​​from one or more adjacent non-anomalous segments. An identified anomalous segment 342 may be determined (e.g., based on a label and / or metadata) to be associated with an image space segment of the stereo image pair 112 that includes artifacts producing depth value discontinuities (e.g., illumination-based artifacts such as, but not limited to, light sources, shadows, reflections, and / or glare).In some embodiments, the segment correction logic 138 executes image processing algorithms 240 to apply infill and / or interpolation functions to correct inaccurate depth values. For example, an infill algorithm can select a set of depth value key points from non-anomalous segments near an identified anomalous segment and interpolate these depth values ​​inward to fill the anomalous segment with interpolated depth values ​​that blend seamlessly with the non-anomalous segments. In some embodiments, the selected depth value key points can include three or more key points that define a polygon that at least partially overlaps the anomalous segment. The depth value interpolation can then be performed by the image processing algorithms 240, at least on the basis of a multi-point infill.

[0058] In some embodiments, the segment correction logic 138 can apply the segment masking function 242 to one or more identified anomalous segments 342 to mask the anomalous segments so that they do not appear with depth values ​​in the corrected depth image 162. Masking of identified anomalous segments 342 can be performed so that inaccurate depth value data is not provided to downstream processes (e.g., where a lack of depth data is less detrimental to the downstream process than receiving poor depth data).

[0059] In some embodiments, the segment correction logic 138 can apply other corrective functions that utilize structural geometries associated with identified anomalous segments 342 and fit interpolated depth values ​​to surfaces using one or more surface fitting algorithms 246 to generate the corrected depth values ​​for the corrected depth image 162. In some embodiments, a surface shape classification can be derived from metadata for an identified anomalous segment, and then a corresponding shape selected by the surface fitting algorithm(s) 246 can be determined for fitting based on the classification. For example, if an identified anomalous segment is located within the boundaries of a surface that is considered a flat surface (e.g.,If an identified anomalous segment falls within the boundaries of a surface designated as curved, the interpolation and / or extrapolation of depth values ​​from the selected non-analog depth key points can be calculated and adapted to the flat surface using one or more surface fitting algorithms 246, for example, using a polynomial-based fitting algorithm.

[0060] Fig. 3C provides a non-restrictive example where the segment correction logic 138 processes the labeled depth image 340 to produce a corrected depth image 360. The labeled depth image 340 may include one or more segments labeled to contain identified anomalous segments 342, and the labeled depth image 340 may further include label segments 342 to enclose metadata, for example, based on a classification applied to a segment by the depth anomaly detection logic 136. As shown in the corrected depth image 360, the depth anomaly detection logic 138 has applied at least one image processing algorithm 240 such that the depth values ​​of pixels indicating optical reflections 343 have been filled or otherwise smoothed in the corrected depth image 360 ​​based on neighboring depth values ​​(shown at 363).The large flat surface segment (344) associated with the ceiling surfaces has been corrected by the image processing algorithm(s) 240 and / or surface fitting algorithm(s) 246 to apply interpolated values ​​with a flat surface fit for rendering as the expected gradient pattern of depth values ​​(shown at 364). Similarly, for the glass wall surface 326, the discontinuity 337 caused by the support column 327 and discontinuities 338 caused by optical reflections 328 have been corrected by the image processing algorithm(s) 240 and / or surface fitting algorithm(s) 246 to apply interpolated values ​​with a flat surface fit for rendering as the expected gradient pattern of depth values ​​(shown at 366).Furthermore, with reference to the glass wall surface 326, the surface fitting algorithms 246 may be subject to one or more structural constraints based on the intersection of the glass wall surface with the surface of the floor 322 to correct discontinuities in depth values ​​where these surfaces intersect, as shown at 368.

[0061] As explained above, the surface fitting algorithm(s) can provide locally consistent depth values ​​by surface-fitting a set of depth values ​​to a specific surface of a feature in a scene (e.g., a floor, a tabletop, etc.). However, in some cases, such surface fitting can result in misalignments between distinct but contiguous surfaces in the scene. For example, a first set of depth values ​​can be corrected by surface-fitting these values ​​to the surface of a floor, and a second set of depth values ​​can be corrected by separately surface-fitting these values ​​to the surface of an adjacent wall. However, when pixels of the resulting corrected depth image are projected into 3D space, inconsistencies can become visible.An adjacent floor and wall may have inaccurate orientations and / or other inaccurate spatial relationships (such as accurately representing where the surfaces intersect). In other words, in some cases, when corrections are applied based on local adjustments of extrapolations to distinct planes, there is a deficiency in enforcing certain structural constraints—such as enforcing constraints where a first surface intersects a second surface, or constraints on surface positions where one plane intersects another along a line.

[0062] As such, in some embodiments the surface adaptation algorithms 246 can apply one or more structural constraints based on the geometry of the overall scene, as illustrated in the stereo image pair 112. In some embodiments, the interpolated depth values ​​can be adapted to planes / surfaces of an architectural model to compute corrected depth values ​​such that the position and / or orientation of these surfaces relative to each other is constrained based on the selected generalized architectural model (e.g., a corridor model in this example).

[0063] As a non-restrictive example, Fig. 4. For a corridor scene 410, surface adaptation algorithms 246 may be constrained based on a generalized architectural model, such as an architectural corridor model, as shown in 412. The architectural corridor model 412 may comprise five levels (e.g., a floor level, a ceiling level, a left-wall level, a right-wall level, and an end-of-corridor back level) that have a predefined structural relationship. For example, for an architectural corridor model 412, the surfaces of opposite left and right walls may be defined as parallel to each other and as intersecting orthogonally with the floor and / or ceiling levels, with the floor and / or ceiling levels also being defined as parallel to each other.As shown in Figure 414, the interpolated depth values ​​can be fitted to planes / surfaces of the architectural corridor model 412 to compute corrected depth values, thus constraining the position and / or orientation of these surfaces relative to each other—avoiding discontinuities that can arise from more localized surface fitting. As another example, for a scene 420 illustrating a corner of a room, the surface fitting algorithms 246 can be constrained based on another generalized architectural model, such as an architectural space, as shown in Figure 422. The architectural space model 422 may comprise three planes (e.g., a floor plane, a left-wall plane, a right-wall plane) that have a predefined structural relationship.For example, for the architectural space model 422, the surfaces of the left and right walls can be defined such that they extend from the floor and form a right-angled corner. As shown in Figure 424, the interpolated depth values ​​can be fitted to planes / surfaces of the architectural space model 422 to calculate corrected depth values, thus constraining the position and / or orientation of these surfaces relative to each other—avoiding discontinuities that can arise from more localized surface fittings.

[0064] In some embodiments, the surface fitting algorithm(s) 246 can select a generalized architectural model for fitting in order to compute corrected depth values ​​from a model library 248, which comprises a variety of generalized three-dimensional architectural models illustrating various standard geometries that can be used for surface fitting. In some embodiments, the one or more geometry context classification models 250 can input the image data (e.g., stereo image pair 112) and infer an architectural geometry that identifies at least a part of the scene (e.g., a corridor, a room, an atrium, a staircase, a theater, an auditorium, and so on).Based on the architectural geometry classification, the surface fitting algorithm(s) 246 can select a suitable generalized architectural model from the model library 248. Using the selected generalized architectural model, the surface fitting algorithm 246 can apply the most relevant set of constraints to compute corrected depth values ​​for one or more identified anomalous segments of the depth image to produce the corrected depth image 162. In some embodiments, more than one architectural model can be selected and used by the surface fitting algorithm 246 to adjust depth values ​​and produce a corrected depth image 162.

[0065] Fig. Figure 5 illustrates an exemplary training process 500 for training a stereo depth perception model 520 (e.g., stereo depth perception model 120). In the training process 500, the training data sample 160 comprises the stereo image pair 102 and a corresponding corrected depth image 162, which is derived from the stereo image pair 112 by the depth image anomaly processor 130. The trained stereo depth perception model 520 generates a prediction that includes depth image 522. The depth image 522 and the corrected depth image 162 are fed into a loss function 510, which computes loss feedback 512. This feedback is used to adjust a stereo depth perception model 520 based on one or more discrepancies between the depth image 522 and the depth image 162.The stereo depth perception model 520 can be iteratively adjusted while a series of training data samples 160 are applied to drive the loss feedback 512 to a minimum loss, the depth image 522 produced by the stereo depth perception model 520 being substantially similar to the corrected depth image 162 ground truth. In some embodiments, a loss function, such as the loss function 510, can be used by the model accuracy scoring function 150 to compute an accuracy score (accuracy rating) for assessing the quality of the stereo depth perception model (e.g., model accuracy score 152) and / or tracking improvements in the model during the course of a development process.

[0066] Fig. Figure 6A is a data flow diagram illustrating an exemplary process 600 for evaluating the accuracy of one or more stereo depth perception models using one or more depth image anomaly processors 605. In some embodiments, the depth image anomaly processor(s) 605 may include at least one depth image anomaly identification stage 132, as with reference to Fig. 1 and Fig. 2A described, and furthermore include a model accuracy scoring function 150, such as with reference to Fig. 1 and Fig. 2A described. As in Fig. As shown in Figure 6A, process 600 can be used to evaluate a set comprising any number of different stereo depth perception models (shown as 620a and 620b to 620n) so that the relative accuracy and / or quality of each model can be compared. In some embodiments, the stereo depth perception models 620a to 620n can each be a different version or variant of a stereo depth perception model, such as the one relating to Fig. The stereo depth perception model 120 described above comprises the following stereo depth perception models: 620a to 620n. That is, each stereo depth perception model 620a to 620n inputs a stereo image pair 112 to produce a respective prediction comprising a depth image (shown as 622a, 622b to 622n). The stereo image pair 112 (e.g., left and right stereo images captured by a camera pair, and / or left and right stereo images captured at offset locations by a single monocular camera) is fed as input to the depth perception models 620a to 620n, which then individually output a prediction for a depth image 622a to 622n (e.g., a disparity map). Each pixel of the depth images 622a to 622n has a value that represents information about a depth measurement from the stereo image pair 112 to a surface in the scene represented by the pixel.Although each of the depth perception models 620a to 620n can input the same stereo image pair 112, they can still produce different predictions of depth images 622a to 622n based on differences in their specific machine learning model architecture, differences in the training data and / or loss functions used in their training processes, and / or due to a different design factor(s). For example, referring to... Fig. In 6B, image 610 is an example image of a stair scene, captured by an optical image sensor(s), and is input into stereo depth perception models 620a to 620n as stereo image pair 112. In this example, stereo depth perception model 620a may include a cascaded recurrent network with an adaptive correlation (CREStereo)-based stereo matching network that predicts the depth image 622a based on the stereo image pair 112; stereo depth perception model 620b may include a recurrent all-pair field transformation (RAFT) stereo-based deep architecture that predicts the depth image 622b from the stereo image pair 112 based on an optical flow. and the stereo depth perception model 620n can include a unifying flow, stereo and depth estimation-based network (e.g. a GM stereo model) that predicts the depth image 622n from the stereo image pair 112.

[0067] As in Fig. As shown in Figure 6A, the depth image anomaly processor(s) 605 can input the stereo image pair 112 and the resulting depth image produced by a stereo depth perception model. Based on the input(s), the depth image anomaly processor(s) 605 can execute the depth image anomaly identification stage 132 to generate a labeled depth image (shown as labeled depth images 630a, 630b to 630n). The depth image anomaly identification stage 132 can output a labeled depth image 630a, 630b to 630n, which represents a version of the corresponding depth image 622a, 622b to 622n, with segments identified as having inaccurate depth values ​​tagged with labels indicating this determination.The depth image anomaly processor(s) 605 can execute the model accuracy scoring function 150, which takes the labeled depth image(s) 630a, 630b to 630n and / or other data produced by the depth image anomaly identification stage 132 as input and outputs the respective model accuracy scores 634a, 634b to 634n. For example, the model accuracy scoring function 150 can evaluate the distribution and / or severity of anomalous depth values ​​(e.g., identified anomalous segments) in the depth images and quantify the distribution and / or severity (e.g., using a depth image quality algorithm) to an objective accuracy score, which is output as the respective model accuracy scores 634a, 634b to 634n.In some embodiments, the model accuracy scores can be compared with each other and / or in the context of the labeled depth image(s) 630a, 630b to 630n, for example to classify the quality of the models, to select the suitability of a model for use in a specific application and / or to determine whether one or more of the models should be subjected to further training.

[0068] Fig. Figure 7 is a flowchart illustrating an exemplary method 700 for depth image anomaly processing in accordance with some embodiments of the present disclosure. It should be understood that the method 700 described herein is an exemplary representation of the process 700. Fig. The features and elements described in section 7 can be used in conjunction with, in combination with, or substituted for elements of any of the other embodiments discussed herein, and vice versa. Furthermore, it should be understood that the functions, structures, and other descriptions of elements for those described in section 7 may be used in conjunction with, in combination with, or substituted for elements of any of the other embodiments discussed herein. Fig. The embodiments described in section 7 can be applied to elements with the same or similar names in any of the figures and / or embodiments described herein, and vice versa.

[0069] Each block of Method 700, as described herein, comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by one or more processors (comprising processing circuitry) executing instructions stored in memory. The methods can also be embodied as computationally usable instructions stored in computer storage media. The methods can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name only a few. Furthermore, Method 700 is exemplified with reference to the Stereo Anomaly Detection System 100 by Fig. 1 described. However, these procedures can additionally or alternatively be performed by any system or any combination of systems, including, but not limited to, those described herein.

[0070] As discussed in greater detail herein, the procedure may generally include: generating a corrected depth image that includes depth information represented by a stereo image pair, at least based on the evaluation of an initial depth image generated by applying the stereo image pair as input to a machine learning model, wherein the evaluation serves to: identify at least one segment of the initial depth image that includes one or more anomalous depth values, wherein the at least one segment is identified at least based on an image space classification of the at least one segment.

[0071] Method 700, in block B702, includes the following: generating a depth image using a machine learning model based at least on an input of image data comprising at least one stereo image pair. In some embodiments, a stereo image pair (e.g., left and right stereo images acquired by a pair of cameras, and / or left and right stereo images acquired at offset locations by a single monocular camera) can be fed as input to a stereo depth perception model, which then outputs a prediction for a depth image (e.g., a disparity map). Each pixel of the depth image has a value representing information about a depth measurement from the stereo image pair to a surface in the scene represented by the pixel.The resulting depth image can have the same size in terms of resolution and / or row and column coordinates as the captured images of the stereo image pair, so that 3D depth information about a feature appearing at a given pixel in the stereo image pair can be determined based on the value of the corresponding pixel of the predicted depth image. As with reference to . Fig. As described in Figure 1, a stereo depth perception model 120 can input a stereo image pair 112 to produce a prediction that includes the depth image 122. Optical image data 110 comprising a stereo image pair(s) 112 can be produced by one or more sensors 102 (e.g., camera(s)). A stereo image pair 112 can comprise a pair of images simultaneously captured by a set of paired image sensors from offset viewpoints, and / or a pair of images captured by a single image sensor from two offset positions. An image sensor 102 can be, for example, RGB cameras, IR cameras, RGB-IR cameras, stereo camera arrays, depth cameras, and / or other cameras, such as cameras related to the vehicle 900. Fig. 9A-9D described, include. The image sensor(s) 102 may include one or more cameras of an ego object or ego actuator, such as a stereo camera(s) 968, wide-angle camera(s) 970 (e.g., fisheye camera(s)), infrared camera(s) 972, surround-view camera(s) 974 (e.g., 360° cameras), occupant monitoring system (OMS) sensor(s) 901, and / or long- and / or medium-range camera(s) 998 of the autonomous vehicle 900. Fig. 9A-9D.

[0072] Procedure 700, at block B704, includes the following: evaluating the depth image to identify at least one segment of one or more anomalous depth values. The procedure may include: identifying the at least one segment based on at least one depth value pattern associated with the image space classification of the at least one segment. The procedure may also include: identifying the at least one segment based on identifying one or more depth value discontinuities within the at least one segment.

[0073] Method 700, in block B706, includes the following: classifying the at least one segment in the image space, at least based on the identification of one or more anomalous depth values. In some embodiments, as with reference to Fig. 1 and Fig. As described in Section 2A, a depth image anomaly identification stage 132 comprises a depth anomaly detection logic 136 that evaluates the depth image 122 against the stereo image pair 112 t to identify segments of the depth image 122 that are predicted to contain pixels with inaccurate depth values. The depth anomaly detection logic 136 may, for example, comprise one or more image segmentation models 212, one or more image segment classification models 214, and / or a segment labeling function 216. The outputs of the image segmentation model(s) 212 and the image segment classification model(s) 214 generate data that can characterize the physical structures and / or features of a scene captured by the stereo image pair 112, which helps the depth anomaly detection logic 136 in the task of assessing the accuracy of a depth image produced from the stereo image pair 112.For example, the image segmentation model(s) 212 can segment the stereo image pair 112 into regions of pixels that correspond to distinct physical features (e.g., objects, surfaces, geometry, structures, etc.) present in a scene. The depth anomaly detection logic 136 can correlate segments identified from the stereo image pair 112 with corresponding pixels of a depth image 122 to associate depth data with specific features represented in a segment. The image segment classification model(s) 214 can evaluate segments of the depth image 122 generated by the image segmentation model(s) 212 to detect features that exhibit characteristics known to contribute to depth image anomalies and apply a classification (e.g. a tag, label, etc.) to these segments based on the detected characteristics.This means that the image segment classification model(s) 214 can infer one or more image space classifications (e.g., a surface geometry classification) for an individual segment, which can indicate whether a segment includes features known to contribute to depth image anomalies. For example, the image segment classification model(s) 214 can classify, based on an inference that a feature includes an optical reflection or a light source, or as a feature associated with curved or planar surfaces (e.g., a floor, wall, ceiling, stair, corner, hallway, etc.) that extend back to a focal point in an image space.The depth anomaly detection logic 136 can then locate the corresponding segment of pixels in the depth image and specifically evaluate those segments labeled as containing features prone to causing anomalous depth data with respect to the accuracy of their depth values, based on the classification(s). The depth anomaly detection logic 136 can evaluate segments containing potentially anomalous depth values ​​based on the classification(s) and / or the depth values ​​of one or more neighboring (unsuspected) segments.Based on at least one or more identified anomaly segments identified by the depth anomaly detection logic 136, the segment labeling function 216 can generate a labeled depth image 140, which includes an updated version of the depth image 122 containing labels that tag the one or more identified anomaly segments as identified anomaly segments.

[0074] In some embodiments, the at least one segment can be identified based on an input from a human-machine interface that specifies the at least one segment as containing one or more anomalous depth values. That is, the depth anomaly detection logic 136 can be assisted in identifying anomalous segments in a predicted depth image 122 based on inconsistency information provided as input by a human user to a human-machine interface (HMI) 124. For example, the anomaly detection system on the HMI 124 can display one or both images of a stereo image pair 112 and a visual rendering of the depth image 122 predicted by the stereo depth perception model 120.In some embodiments, the conversion of the depth image to a point cloud can be displayed, as certain features are easier to detect in 3D compared to 2D. A user viewing the images on the HMI 124 can select one or more regions of pixels that they perceive as potentially anomalous, and this specification of one or more regions can be included as input to the depth anomaly detection logic 136, which is used to define identified anomaly segments.

[0075] Procedure 700, at block B708, includes the following: generating, at least based on the classification of the at least one segment, an updated depth image to include one or more updated depth values ​​in the at least one segment of the updated depth image. For example, at least based on one or more identified anomaly segments identified by the depth anomaly detection logic 136, the segment labeling function 216 can generate a labeled depth image 140 that includes an updated version of the depth image 122, which includes labels that tag the one or more identified anomaly segments as identified anomaly segments, as in the exemplary labeled depth image 340 in Fig. 3B illustrates. As in Fig. As shown in Figure 3B, the segment labeling function 216 can update the initial depth image 312 to generate the labeled depth image 340, which includes one or more segments labeled to contain identified anomalous segments (shown in Figure 342). In some embodiments, the segment labeling function 216 can further label segments to include metadata based on the classifications inferred by the segment classification model(s) 214. Identified anomalous segments 342 can also be labeled to indicate that they include optical reflections (343), large flat surfaces (344), or glass or other transparent features (345), and / or can be labeled with other metadata that may help identify the nature of the feature and / or the cause of inaccurate depth predictions for that feature.

[0076] In some embodiments, the method may continue with the following: generating the updated depth image based on at least one or more updated depth values ​​calculated for the one or more anomalous depth values. In some embodiments, a corrected depth image may be generated based on at least one or more corrected depth values ​​calculated for the one or more anomalous depth values. For example, one or more interpolated depth values ​​for the one or more anomalous depth values ​​may be calculated based on at least a set of depth values ​​selected from one or more segments of the depth image that are not identified as including the one or more anomalous depth values, wherein the one or more updated (e.g.,The corrected depth values ​​are based on one or more interpolated depth values. A set of data samples can be output that includes a data sample comprising at least one stereo image pair and the corrected depth image.

[0077] As with reference to Fig. As discussed in Section 1, in some embodiments the labeled depth image(s) 140 can be applied to a depth image anomaly correction stage 134, which produces a corrected depth image 162 by applying adjustments to the depth image 122 based on the identified anomalous segments 342. The depth image anomaly correction stage 134 can include a segment correction logic 138 that applies one or more correction techniques to one or more of the identified anomalous segments 342 to at least partially mitigate inaccuracies in depth values ​​in these segments. The segment correction logic 138 can calculate corrected depth values ​​based on the context associated with an identified anomalous segment (e.g.,based on depth values ​​of adjacent non-anomalous segments, based on detected structural characteristics within the spatial volume corresponding to an identified anomalous segment, and / or on characteristics of local phenomena (e.g., illumination, reflections, glare, etc.) that have been identified as causing the inaccuracies in depth values. The depth image anomaly correction stage 134 can output a stereo image pair training data sample 160 comprising the stereo image pair 102, and a corresponding corrected depth image 162 derived from the stereo image pair 112 by the depth image anomaly processor 130.

[0078] The one or more interpolated depth values ​​can be applied to a surface fitting algorithm to compute the one or more updated (e.g., corrected) depth values. The segment correction logic 138 can apply corrective functions that utilize structural geometries associated with identified anomalous segments 342 and fit interpolated depth values ​​to surfaces using one or more surface fitting algorithms 246 to generate the corrected depth values ​​for the corrected depth image 162. In some embodiments, a surface shape classification can be determined from metadata for an identified anomalous segment, and then a corresponding shape selected by the surface fitting algorithm(s) 246 can be determined for fitting based on the classification.In some embodiments, one or more structural constraints can be applied to the surface fitting algorithm based on one or more contextual classifications determined from a geometry illustrated in the stereo image pair. The surface fitting algorithms 246 can be subject to one or more constraints based on the geometry of the overall scene, as illustrated in the stereo image pair 112. In some embodiments, the interpolated depth values ​​can be fitted to planes / surfaces of an architectural model to compute corrected depth values ​​such that the position and / or orientation of these surfaces relative to each other is constrained based on the selected generalized architectural model (e.g., a corridor model in this example).

[0079] Generating a corrected depth image can include: applying a mask to the at least one segment of the updated depth image to edit the one or more anomalous depth values ​​and output a masked depth image. One or more operations (e.g., of vehicle 900) can be performed based on the masked depth image. In some embodiments, the method can include: determining the quality of the depth image based at least on the at least one segment of one or more anomalous depth values, and outputting an accuracy score for the machine learning model based at least on the determined quality.

[0080] Fig. Figure 8 is a flowchart illustrating an exemplary method 800 for generating a depth image from a stereo image pair in accordance with some embodiments of the present disclosure. It should be understood that the method 800 described herein refers to the method 800 of Fig. The features and elements described in section 8 can be used in conjunction with, in combination with, or substituted for elements of any of the other embodiments discussed herein, and vice versa. Furthermore, it should be understood that the functions, structures, and other descriptions of elements for those described in section 8 are not identical to those described herein. Fig. The embodiments described in section 8 can be applied to elements with the same or similar names in any of the figures and / or embodiments described herein, and vice versa.

[0081] Each block of Method 800, as described herein, comprises a computational process that can be performed using any combination of hardware, firmware, and / or software. For example, various functions can be performed by one or more processors (comprising processing circuitry) executing instructions stored in memory. The methods can also be embodied as computationally usable instructions stored in computer storage media. The methods can be provided by a standalone application, a service, or a hosted service (standalone or in combination with another hosted service), or a plug-in to another product, to name just a few. Furthermore, Method 800 is exemplified with reference to the Stereo Anomaly Detection System 100 by Fig. 1 described. However, these procedures can additionally or alternatively be performed by any system or any combination of systems, including, but not limited to, those described herein.

[0082] Method 800, at block B802, includes the following: obtaining image data comprising at least one stereo image pair. As herein with reference to Fig. As described in Figure 1, in some embodiments, optical image data 110 comprising a stereo image pair 112 can be produced by one or more sensors 102 (e.g., camera(s)). A stereo image pair 112 can comprise a pair of images simultaneously captured by a set of paired image sensors from offset viewpoints, and / or a pair of images captured by a single image sensor from two offset positions. An image sensor 102 can be, for example, RGB cameras, IR cameras, RGB-IR cameras, stereo camera arrays, depth cameras, and / or other cameras, such as cameras related to the vehicle 900. Fig. 9A-9D described, include. The image sensor(s) 102 may include one or more cameras of an ego object or ego actuator, such as a stereo camera(s) 968, wide-angle camera(s) 970 (e.g., fisheye camera(s)), infrared camera(s) 972, surround-view camera(s) 974 (e.g., 360° cameras), occupant monitoring system (OMS) sensor(s) 901, and / or long- and / or medium-range camera(s) 998 of the autonomous vehicle 900. Fig. 9A-9D. The image sensor(s) 102 can therefore be used to generate the image data 110 of a three-dimensional environment (3D environment) around an ego object or an ego actuator.

[0083] Procedure 800, at block B804, includes the following: generating, using a machine learning model, a depth image based at least on one stereo image pair, wherein the machine learning model is trained to infer the depth image based at least on a feedback loss determined using at least one ground-truth depth image. The ground-truth depth image may be generated based at least on: a corrected depth image comprising one or more corrected depth values, wherein the one or more corrected depth values ​​are calculated for one or more anomalous depth values ​​identified from a training depth image determined from a training stereo image pair.In some embodiments, one or more operations of the vehicle 900 can be performed based on the depth image, such as, but not limited to, operations for depth-based object detection. In some embodiments, the machine learning model can be a stereo depth perception model, such as with reference to... Fig. As described in section 5, the trained machine learning model can be trained. For example, the trained machine learning model can be trained to make a prediction that includes a depth image based on an input that includes a stereo image pair. The depth image and a ground-truth depth image (based at least on a corrected depth image) can be fed into a loss function that computes loss feedback. This feedback is used to match a machine learning model based on one or more discrepancies between the depth image and the ground-truth depth image. The machine learning model can be trained to operate as a stereo depth perception model based on iterative model matching, while a series of training data samples are applied to drive the loss feedback to a minimum loss, with the predicted depth image produced by the model being substantially similar to the corrected depth-image-based ground-truth depth image.

[0084] In some embodiments, a loss function, such as loss function 510, can be used by a model accuracy scoring function 150 to compute an accuracy score for assessing the quality of the stereo depth perception model (e.g., model accuracy score 152) and / or tracking improvements in the model during a development process. As such, in some embodiments, the method can: determine the quality of the depth image based at least on identifying at least one segment of the depth image as having a set of anomalous depth values, and output an accuracy score for the machine learning model based at least on the determined quality.In some embodiments, the method can: output a set of data samples, including a data sample comprising the at least one stereo image pair and the depth image generated from the at least one stereo image pair. One or more operations (e.g., of vehicle 900) can be performed based on the masked depth image. In some embodiments, depth images produced by the machine learning model can be stored in an image repository (e.g., a database, a data store, etc.) so that they can be retrieved for subsequent analysis, used as data samples for training machine learning models, and / or for other purposes.

[0085] In some embodiments, the systems and methods described herein can be performed in or in conjunction with a simulation environment (e.g., NVIDIA's DriveSIM) using simulated data (e.g., simulated sensor data from simulated sensors of a virtual or simulated machine). For example, simulated sensor data and / or map data, including the application of stereo image pair-derived depth measurements from the simulation environment, can be used to perform operations (e.g., navigation) associated with the virtual machine within the environment. These simulated operations can be used to test the performance of the underlying algorithms, systems, and / or processes before their deployment in the real world. In some cases, the simulation can be used to generate synthetic training data—e.g.,Training data that includes regions of interest and / or subregions of interest within the simulation. The synthetic training data can then be processed (in addition to or as an alternative to real-world data) to determine, for example, geometry and / or other information related to road surfaces. In one example, such as when a simulation environment is used for testing, validation, training, etc., the simulation environment and / or associated training data can be rendered or otherwise generated using one or more light transport algorithms—such as ray tracing and / or path tracing algorithms. In some embodiments, the simulation environment and / or one or more objects, features, or components of it can be used within a three-dimensional content collaboration platform (3D content collaboration platform) (e.g.,NVIDIA's Omniverse) for industrial digitization, generative physical artificial intelligence (AI), and / or other use cases, applications, or services. For example, the content collaboration platform or system may include a system for using or developing Universal Scene Descriptor (USD (e.g., OpenUSD)) data to manage objects, features, scenes, etc., within a simulated environment, digital environment, etc. The platform may include real-world physics simulation, such as using NVIDIA's PhysX SDK, to simulate real-world physics and physical interactions with simulations hosted by the platform. The platform may also include OpenUSD in conjunction with ray tracing / path tracing / light transport simulation (e.g.,Integrate NVIDIA's RTX rendering technologies into software tools and simulation workflows for building, training, deploying or testing AI systems - such as systems for testing, validating, training (e.g., machine learning models, neural networks, etc.) and / or other tasks related to automotive, robotics, machinery or other applications.

[0086] In some embodiments, teleoperation or remote control of a vehicle or other machine can be performed using a remote control or teleoperation system. For example, the systems and methods described herein can be used to produce depth information relating to animated or static objects, hazards, etc., which may be used or included in a visualization or mapping of an environment to assist a remote operator in controlling—or providing waypoints or other information for control or navigation—an autonomous or semi-autonomous machine in an environment.

[0087] In some embodiments, the systems and methods described herein can be used in an in-vehicle infotainment (IVI) system or in an in-cabin experience (IX) application. For example, the infotainment system in a vehicle (e.g., cars, trucks, drones, construction equipment, robots, semi-autonomous vehicles, or autonomous vehicles) may include one or more onboard processors (e.g., CPUs, GPUs, hardware-based deep learning accelerators (DLAs), hardware-based programmable vision accelerators (PVAs)) – which may include one or more vector processing units (VPUs), direct memory access (DMA) systems, and / or pixel processing engines (PPEs), hardware-based optical flow accelerators (OFAs), SoCs, etc.The system may include a memory and / or storage (e.g., for storing control algorithms, sensor data, and one or more machine learning models) and a memory and / or storage (e.g., for storing entertainment content, navigation data, and user preferences). The system can use these processors to execute one or more machine learning processes to enable features such as occupant monitoring, gesture recognition, and real-time communication with other services via network connectivity. The in-vehicle infotainment system may also use natural language processing (NLP) models to enable voice-based interaction.The one or more machine learning models can be stored locally or accessed via one or more APIs connected to cloud services, enabling the system to process requests in real time or near real time.

[0088] In some embodiments, the systems and methods described herein can be used in robotic applications. For example, a robot or robotic system may include one or more onboard processors (e.g., CPUs, GPUs, hardware-based deep learning accelerators (DLAs), hardware-based programmable vision accelerators (PVAs)) – which may include one or more vector processing units (VPUs), direct memory access (DMA) systems and / or pixel processing engines (PPEs), hardware-based optical flow accelerators (OFAs), SoCs, etc.) and memory and / or storage (e.g., for storing control algorithms, sensor data, and one or more machine learning models).The robotic system can use these processors to execute one or more machine learning models (e.g., language models) that allow it to perform complex tasks autonomously or semi-autonomously, such as interacting with and / or manipulating static and / or dynamic objects or navigating environments using sensors such as cameras, LiDAR, radar, ultrasonic sensors, and more. The system can use sensor fusion techniques to combine data from multiple sensors (e.g., cameras, infrared, LiDAR, radar, accelerometers) to create a comprehensive model of the robot's environment. This data can be processed locally on the robot or sent to remote servers for more computationally intensive tasks, such as 3D mapping or SLAM (Simultaneous Localization and Mapping). In one or more embodiments, data from individual robots (e.g.,Sensor data, task status, or environmental conditions) are uploaded to the cloud, where centralized AI models can analyze optimized commands and distribute them to an entire fleet. In some embodiments, the machine learning model(s) described herein (e.g., language models, VLMs, LLMs, MMLMs, diffusion models, NeRF models, DNNs, etc.) can be used to allow the robot to perceive and reflect on its environment and / or communicate with one or more other robots and / or one or more people in that environment. In some embodiments, the robot can communicate (e.g., using one or more network interface cards (NICs) and / or data processing units (DPUs)) with one or more locally hosted servers / computing devices and / or with one or more remote servers / computing devices (e.g.,communicate in one or more data centers.

[0089] In some examples, the machine learning model(s) described herein (e.g., deep neural networks, language models, LLMs, VLMs, multimodal language models, perceptual models, tracking models, fusion models, transformer models, diffusion models, encoder-only models, decoder-only models, encoder-decoder models, neural rendering field models (NERF models), etc.) may be packaged as a microservice—such as an inference microservice (e.g., NVIDIA NIMs)—which may include a container (e.g., an operating system (OS)-level virtualization package) that may contain an application programming interface (API) layer, a server layer, a runtime layer, and / or a model engine. For example, the inference microservice may include the container itself and the model(s) (e.g., weights and biases). In some cases, such as when the machine learning model(s) is small enough (e.g.(Having a sufficiently small number of parameters), the model(s) can be enclosed within the container itself. In other examples—such as when the model(s) is / are large—the model(s) can be hosted / stored in the cloud (e.g., in a data center) and / or hosted / stored on-premises and / or at the edge (e.g., on a local server or local computing device, but outside the container). In such embodiments, the model(s) can be accessed via one or more APIs, such as REST APIs. As such, and in some embodiments, the machine learning model(s) described herein can be used as an inference microservice to accelerate the development of a model(s) on a cloud, data center, or edge computing system while ensuring data security.The inference microservice can include, for example: one or more APIs, a pre-configured container for simplified deployment, an optimized inference engine (e.g., built using standardized AI model deployment and execution software, such as NVIDIA's Triton Inference Server, and / or one or more APIs for high-performance deep learning inference, which may include inference runtime and model optimizations that deliver low latency and high throughput for production applications - such as NVIDIA's TensorRT) and / or enterprise management data for telemetry (e.g., including identity, metrics, health checks, and / or monitoring).The machine learning model(s) described herein may be included as part of the microservice along with an accelerated infrastructure capable of being deployed and / or orchestrated with a single command and autoscaling with a container orchestration system on accelerated infrastructure (e.g., from a single device to data center scale). As such, the inference microservice may include the machine learning model(s) (optimized, for example, for high-performance inference), inference runtime software for executing the machine learning model(s) and providing outputs / responses to inputs (e.g., user queries, prompts, etc.), and enterprise management software for providing health checks, identity verification, and / or other monitoring capabilities.In some embodiments, the inference microservice may include software for performing off-site exchange and / or updates of the machine learning model(s). During the exchange or update process, the software performing the exchange / update may maintain user configurations of the inference runtime software and enterprise management software.

[0090] The systems and procedures described herein may be used, without limitation, by non-autonomous vehicles, semi-autonomous vehicles (e.g., in one or more adaptive driver assistance systems (ADAS)), guided and unguided robots or robotic platforms, warehouse vehicles, off-road vehicles, vehicles coupled to one or more trailers, flying vehicles, boats, shuttle vehicles, emergency vehicles, motorcycles, electric or motorized bicycles, aircraft, construction vehicles, trains, underwater vehicles, remotely controlled vehicles such as drones, and / or other types of vehicles.Furthermore, the systems and methods described herein can be used for a number of purposes, for example, but not limited to, machine control, machine locomotion, machine propulsion, synthetic data generation, model training, perception, augmented reality, virtual reality, mixed reality, robotics, security and surveillance, simulation and digital twinning, autonomous or semi-autonomous machine applications, deep learning, environmental simulation, object or actuator simulation and / or digital twinning, data center processing, conversational AI, light transport simulation (e.g., ray tracing, path tracing, etc.), collaborative content creation for 3D assets, cloud computing, generative AI, and / or other suitable applications.

[0091] Disclosed embodiments may be included in a range of different systems, such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine), systems implemented using a robot, antenna systems, media systems, boat systems, intelligent area monitoring systems, systems for performing deep learning operations, systems for performing simulation operations, systems for performing digital twinning operations, systems implemented using an edge device, systems involving one or more virtual machines (VMs), systems for performing synthetic data generation operations, systems implemented at least partially in a data center, systems for performing conversational AI operations, and systems...that implement one or more language models – such as one or more Large Language Models (LLMs) and / or one or more Vision Language Models (VLMs) – systems for performing light transport simulation, systems for performing collaborative content creation for 3D assets, systems that are implemented at least partially using cloud computing resources, and / or other types of systems. EXEMPLARY AUTONOMOUS VEHICLE

[0092] Fig. Figure 9A is an illustration of an exemplary autonomous vehicle 900 in accordance with some embodiments of the present disclosure. The autonomous vehicle 900 (alternatively referred to herein as the “vehicle 900”) may, without limitation, include a passenger vehicle, such as a car, truck, bus, rescue vehicle, shuttle, electric or motorized bicycle, motorcycle, fire engine, police vehicle, ambulance, boat, construction vehicle, underwater vehicle, robotic vehicle, drone, aircraft, a vehicle coupled with a trailer (e.g., a semi-trailer truck used for carrying cargo), and / or another type of vehicle (e.g., one that is unmanned and / or that carries one or more passengers).Autonomous vehicles are generally described in terms of automation levels, as defined by the National Highway Traffic Safety Administration (NHTSA), a division of the U.S. Department of Transportation, and the Society of Automotive Engineers (SAE) in their "Taxonomy and Definitions for Terms Related to Driving Automation Systems for On-Road Motor Vehicles" (Standard No. J3016-201806, published June 15, 2018; Standard No. J3016-201609, published September 30, 2016; and earlier and future versions of this standard). The Vehicle 900 may be capable of functionality corresponding to one or more of Levels 3 through 5 of autonomous driving levels. The Vehicle 900 may be capable of functionality corresponding to one or more of Levels 1 through 5 of autonomous driving levels.The Vehicle 900, for example, may be capable of driver assistance (Level 1), partial automation (Level 2), conditional automation (Level 3), high automation (Level 4), and / or full automation (Level 5), depending on the configuration. The term "autonomous," as used herein, may include any and / or all types of autonomy for the Vehicle 900 or any other machine, such as fully autonomous, highly autonomous, conditionally autonomous, partially autonomous, providing assistive autonomy, semi-autonomous, mainly autonomous, or any other designation.

[0093] The vehicle 900 can include components such as a chassis, a vehicle body, wheels (e.g., 2, 4, 6, 8, 18, etc.), tires, axles, and other vehicle components. The vehicle 900 can include a propulsion system 950, such as an internal combustion engine, a hybrid-electric drive unit, a fully electric motor, and / or another type of propulsion system. The propulsion system 950 can be connected to a drivetrain of the vehicle 900, which may include a transmission to enable the propulsion of the vehicle 900. The propulsion system 950 can be controlled in response to receiving signals from the throttle / accelerator device 952.

[0094] A steering system 954, which may include a steering wheel, can be used to steer the vehicle 900 (e.g., along a desired path or route) when the drive system 950 is in operation (e.g., when the vehicle is in motion). The steering system 954 can receive signals from a steering actuator 956. The steering wheel may be optional for full automation functionality (Level 5).

[0095] The brake sensor system 946 can be used to operate the vehicle brakes in response to receiving signals from the brake actuators 948 and / or brake sensors.

[0096] One or more controllers 936, which include one or more CPU(s), system-on-chips (“SoCs”) 904 ( Fig. The controller(s) 936, which may include one or more GPUs (e.g., 9C), can provide signals (e.g., representing commands) for one or more components and / or systems of the vehicle 900. For example, the controller(s) can send signals to operate the vehicle brakes via one or more brake actuators 948, to operate the steering system 954 via one or more steering actuators 956, and / or to operate the propulsion system 950 via throttle / accelerator device 952. The controller(s) 936 can include one or more (e.g., integrated) onboard computing devices (e.g., supercomputers) that process sensor signals and issue operational commands (e.g., signals representing commands) to enable autonomous driving and / or to assist a human driver in operating the vehicle 900.The controller(s) 936 may include a first controller 936 for autonomous driving functions, a second controller 936 for functional safety functions, a third controller 936 for artificial intelligence functionality (e.g., computer vision), a fourth controller 936 for infotainment functionality, a fifth controller 936 for redundancy in emergency situations, and / or other controllers. In some embodiments, a single controller 936 may handle two or more of the above functionalities, two or more controllers 936 may handle a single functionality, and / or any combination thereof.

[0097] The controller(s) 936 can provide signals to control one or more components and / or one or more systems of the vehicle 900 in response to sensor data received from one or more sensors (e.g. sensor inputs). The sensor data can be received, for example, without restriction, from the following: Global Navigation Satellite System (GNSS) sensors 958 (e.g., global positioning system sensors), radar sensors 960, ultrasonic sensors 962, LiDAR sensors 964, inertial measurement unit (IMU) sensors 966 (e.g., accelerometers, gyroscopes, magnetic compasses, magnetometers, etc.), microphones 996, stereo cameras 968, wide-angle cameras 970 (e.g., fisheye cameras), infrared cameras 972, surround-view cameras 974 (e.g., 360-degree cameras), long-range and / or medium-range cameras 998, velocity sensors 944 (e.g.,for measuring the speed of the vehicle 900), vibration sensors 942, steering sensors 940, brake sensors (e.g., as part of the brake sensor system 946), one or more occupant monitoring system (OMS) sensor(s) 901 (e.g., one or more interior cameras), and / or other sensor types. In some embodiments, image data 110—including stereo image pair(s) 112—may comprise image data obtained by one or more of the following: Fig. The sensors described in 9A are detected. In some embodiments, the controller(s) 936 can provide signals for controlling one or more components and / or one or more systems of the vehicle 900, at least partially based on a corrected depth image(s) 162 produced by a depth image anomaly processor 130 as described herein.

[0098] One or more of the controllers 936 can receive inputs (e.g., represented by input data) from an instrument cluster 932 of the vehicle 900 and provide outputs (e.g., represented by output data, display data, etc.) via a human-machine interface (HMI) display 934, an acoustic alarm, a loudspeaker, and / or via other components of the vehicle 900. The outputs can include information such as vehicle speed, velocity, time, map data (e.g., the high-definition ("HD") map 922 from Fig. 9C), location data (e.g., the location data of vehicle 900, as shown on a map), direction, location of other vehicles (e.g., an occupancy grid), information about objects and the status of objects as perceived by the controller(s) 936, etc. The HMI display 934 can, for example, display information about the presence of one or more objects (e.g., a road sign, caution sign, traffic light sequence, etc.) and / or information about driving maneuvers that the vehicle has made, is making, or will make (e.g., changing lanes now, exit 34B in two miles, etc.). In some embodiments, the HMI 124 can include the HMI 934 or be implemented using it.

[0099] The vehicle 900 may further include a network interface 924, which can use one or more wireless antenna(s) 926 and / or one or more modem(s) to communicate over one or more networks. For example, the network interface 924 may be capable of communication via Long-Term Evolution (“LTE”), Wideband Code Division Multiple Access (“WCDMA”), Universal Mobile Telecommunications System (“UMTS”), Global System for Mobile (“GSM”), IMT-CDMA Multi-Carrier (“CDMA2000”), etc. The wireless antenna(s) 926 can also allow communication between objects in the environment (e.g., vehicles, mobile devices, etc.) using local area network(s), such as Bluetooth, Bluetooth Low Energy (“LE”), Z-Wave, ZigBee, etc., and / or for a low-power wide-area network (“LPWANs”), such as LoRaWAN, SigFox, etc.

[0100] Fig. 9B is an example of camera locations and fields of view for the exemplary autonomous vehicle 900 from Fig. 9A in accordance with embodiments of the present disclosure. The cameras and their respective fields of view are an exemplary embodiment and are not intended to be limiting. For example, additional and / or alternative cameras may be included and / or the cameras may be located at different locations on the vehicle 900.

[0101] The camera types may include, but are not limited to, digital cameras adapted for use with the components and / or systems of the Vehicle 900. The camera(s) may operate at Automotive Safety Integrity Level (ASIL) B and / or another ASIL. Depending on the configuration, the camera types may be capable of any frame rate, such as 60 frames per second (fps), 120 fps, 240 fps, etc. The cameras may be capable of using rolling shutters, global shutters, another type of shutter, or a combination thereof.In some embodiments, the color filter array may include a red-clear-clear-clear (RCCC) color filter array, a red-clear-clear-blue (RCCB) color filter array, a red-blue-green clear (RBGC) color filter array, a Foveon X3 color filter array, a Bayer sensor (RGGB) color filter array, a monochrome sensor color filter array, and / or another type of color filter array. In some embodiments, clear-pixel cameras, such as cameras with an RCCC, RCCB, and / or RBGC color filter array, may be used in an effort to increase light sensitivity.

[0102] In some examples, one or more of the camera(s) can be used to perform advanced driver assistance system (ADAS) functions (e.g., as part of a redundant or fail-safe system). For example, a multi-function monocular camera can be installed to provide functions such as lane departure warning, traffic sign recognition, and intelligent headlight control. One or more of the camera(s) (e.g., all of the cameras) can simultaneously record and provide image data (e.g., video).

[0103] One or more of the cameras can be mounted in a custom-designed (three-dimensionally printed) assembly to eliminate stray light and reflections from inside the car (e.g., reflections off the dashboard in the windshield mirrors) that could impair the camera's image capture capabilities. Regarding side mirror mounting assemblies, the assemblies can be custom 3D printed so that the camera mounting plate matches the shape of the side mirror. In some examples, the camera(s) can be integrated into the side mirror. For side-view cameras, the camera(s) can also be integrated within four pillars at each corner of the cabin.

[0104] Cameras with a field of view that includes parts of the environment in front of the vehicle (e.g., forward-facing cameras) can be used for surround view to identify ahead routes and obstacles and, with the aid of one or more controllers and / or control SoCs, to provide information critical for generating an occupancy grid and / or determining preferred vehicle routes. Forward-facing cameras can be used to perform many of the same ADAS functions as LiDAR, including emergency braking, pedestrian detection, and collision avoidance. Forward-facing cameras can also be used for ADAS functions and systems, including lane departure warnings (LDW), autonomous cruise control (ACC), and / or other functions such as traffic sign recognition.

[0105] Various cameras can be used in a forward-facing configuration, for example, including a monocular camera platform that incorporates a CMOS (Complementary Metal Oxide Semiconductor) color imager. Another example could be one or more wide-angle cameras 970, which can be used to detect objects entering the field of view from the periphery (e.g., pedestrians, crossing traffic, or bicycles). Although in Fig. Since only one wide-angle camera is illustrated in Figure 9B, there can be any number (including zero) of wide-angle cameras 970 on the vehicle 900. Additionally, any number of remote cameras 998 (e.g., a pair of long-range stereo cameras) can be used for depth-based object detection, especially for objects for which a neural network has not yet been trained. The remote camera(s) 998 can also be used for object detection and classification, as well as simple object tracking.

[0106] Any number of stereo cameras 968 can also be included in a forward-facing configuration. In at least one embodiment, one or more stereo camera(s) 968 can include an integrated control unit comprising a scalable processing unit that can provide programmable logic (“FPGA”) and a multi-core microprocessor with an integrated controller area network (“CAN”) or Ethernet interface on a single chip. Such a unit can be used to generate a 3D map of the vehicle's surroundings, including a distance estimate for all points in the image. An alternative stereo camera(s) 968 can include a compact stereo vision sensor, which can include two camera lenses (one each on the left and right) and an image processing chip that measures the distance from the vehicle to the target object and processes the generated information (e.g.,Metadata) can be used to activate the autonomous emergency braking and lane departure warning functions. Other types of stereo camera(s) 968 may be used in addition to or as an alternative to those described herein. In some embodiments, the optical image sensor(s) 102 is / are implemented using one or more stereo cameras 968.

[0107] Cameras with a field of view that includes parts of the surroundings at the side of the vehicle 900 (e.g., side-view cameras) can be used for surround view, which provides information used to generate and update the occupancy grid and to generate side-impact collision warnings. For example, one or more surround camera(s) 974 (e.g., four surround cameras 974 as shown in Fig. (Figure 9B illustrates) is positioned on the vehicle 900. The surround view camera(s) 974 can include one or more wide-angle cameras 970, fisheye cameras, 360-degree cameras, and / or the like. For example, four fisheye cameras can be positioned at the front, rear, and sides of the vehicle. In an alternative arrangement, the vehicle can use three surround view cameras 974 (e.g., left, right, and rear) and can use one or more other cameras (e.g., a forward-facing camera) as a fourth surround view camera.

[0108] Cameras with a field of view that includes parts of the area behind the vehicle 900 (e.g., rear-view cameras) can be used for parking assistance, surround view, rear collision warnings, and generating and updating the occupancy grid. A wide range of cameras can be used, including, but not limited to, cameras that are also suitable as forward-facing cameras (e.g., long-range and / or medium-range cameras 998, stereo cameras 968, infrared cameras 972, etc.), as described herein.

[0109] Cameras with a field of view that includes parts of the interior environment within the cabin of the vehicle 900 (e.g., one or more OMS sensor(s) 901) can be used as part of an occupant monitoring system (OMS), such as, but not limited to, a driver monitoring system (DMS). For example, OMS sensors (e.g., the OMS sensor(s) 901) can be used (e.g., by the controller(s) 936) to track the gaze direction, head pose, and / or blinking of an occupant and / or driver. This gaze information can be used to determine a level of attention of the occupant or driver (e.g., to detect drowsiness, fatigue, and / or distraction) and / or to take a reactive action to prevent harm to the occupant or operator.In some embodiments, data from OMS sensors can be used to enable gaze-controlled operations triggered by driver and / or non-driver occupants, such as, but not limited to, adjusting cabin temperature and / or airflow, opening and closing windows, controlling cabin lighting, controlling entertainment systems, adjusting mirrors, adjusting seat positions, and / or other operations. In some embodiments, an OMS can be used for applications such as determining when objects and / or occupants have been left in a vehicle cabin (e.g., by detecting occupant presence after the driver leaves the vehicle).

[0110] Fig. 9C is a block diagram of an exemplary system architecture for the exemplary autonomous vehicle 900 from Fig. 9A in accordance with some embodiments of the present disclosure. It should be understood that these and other arrangements described herein are presented only as examples. Other arrangements and elements (e.g., machines, interfaces, functions, orders, groupings of functions, etc.) may be used in addition to or instead of those shown, and other elements may be omitted entirely. Furthermore, many of the elements described herein are functional entities that may be implemented as discrete or distributed components, or in conjunction with other components, and in any suitable combination and position. Various functions described herein as being performed by entities may be executed by hardware, firmware, and / or software. For example, various functions may be performed by a processor executing instructions stored in memory.

[0111] The components, features and systems of the 900 vehicle in Fig. 9C are each illustrated as connected via bus 902. Bus 902 may include a Controller Area Network (CAN) data interface (alternatively referred to herein as the CAN bus). A CAN can be a network within the vehicle 900 that is used to support the control of various features and functionality of the vehicle 900, such as actuation of the brakes, acceleration, braking, steering, windshield wipers, etc. A CAN bus may be configured to have dozens or even hundreds of nodes, each with its own unique identifier (e.g., a CAN ID). The CAN bus can be read to obtain steering wheel angle, ground speed, engine revolutions per minute (RPM), button positions, and / or other vehicle status indicators. The CAN bus may be ASIL-B compliant.

[0112] Although described herein as a CAN bus, this is not intended to be restrictive. For example, FlexRay and / or Ethernet may be used in addition to or as an alternative to the CAN bus. Furthermore, although a single line is used to represent the 902 bus, this is not intended to be restrictive. For example, there may be any number of 902 buses, which may include one or more CAN buses, one or more FlexRay buses, one or more Ethernet buses, and / or one or more other types of buses using a different protocol. In some examples, two or more 902 buses may be used to perform different functions and / or for redundancy. For example, a first 902 bus may be used for collision avoidance functionality, and a second 902 bus may be used for actuation control.In any given example, the bus 902 can communicate with any of the components of the vehicle 900, and two or more buses 902 can communicate with the same components. In some examples, each SoC 904, each controller 936, and / or each computer within the vehicle can have access to the same input data (e.g., inputs from sensors of the vehicle 900) and be connected to a common bus, such as the CAN bus.

[0113] The vehicle 900 can include one or more controllers 936, such as those mentioned herein in relation to Fig. 9A are described. The 936 controller(s) can be used for a number of functions. The 936 controller(s) can be coupled with any of the various other components and systems of the 900 vehicle and used for controlling the 900 vehicle, the 900 vehicle's artificial intelligence, the 900 vehicle's infotainment system, and / or the like.

[0114] The vehicle 900 can include one or more system-on-a-chip (SoC) 904. The SoC 904 can include one or more CPUs 906, GPUs 908, one or more processors 910, caches 912, accelerators 914, data storage 916, and / or other components and features not illustrated. The SoC(s) 904 can be used to control the vehicle 900 in a variety of platforms and systems. For example, the SoC(s) 904 in a system (e.g., the system of the vehicle 900) can be combined with an HD card 922, enabling map refreshes and / or updates via a network interface 924 from one or more servers (e.g., the server(s) 978). Fig. 9D). In some embodiments, one or more functions of the depth image anomaly processor 130, depth image anomaly identification data 132 and / or depth image anomaly correction status 134 can be implemented, at least in part, using code executed by the SoC(s) 904, CPU(s) 906 and / or GPU(s) 908.

[0115] The CPU(s) 906 may include a CPU cluster or CPU complex (alternatively referred to herein as "CCPLEX"). The CPU(s) 906 may include multiple cores and / or L2 caches. For example, in some embodiments, the CPU(s) 906 may include eight cores in a coherent multiprocessor configuration. In some embodiments, the CPU(s) 906 may include four dual-core clusters, each cluster having a dedicated L2 cache (e.g., a 2 MB L2 cache). The CPU(s) 906 (e.g., the CCPLEX) may be configured to support concurrent cluster operation, allowing a combination of CPU(s) 906 clusters to be active at any given time.

[0116] The CPU(s) 906 can implement power management capabilities that include one or more of the following features: individual hardware blocks can be automatically clock-gated (automatically disconnected from the clock signal in a gate) when idle to save dynamic power; each core clock can be disconnected when the core is not actively executing instructions due to the execution of WFI / WFE instructions; each core can be independently power-gated; each core cluster can be independently clock-gated when all cores are clock-gated or power-gated; and / or each core cluster can be independently power-gated when all cores are power-gated.The CPU(s) 906 can further implement an improved power state management algorithm, specifying the permissible power states and expected wake-up times, and the hardware / microcode for the core, cluster, and CCPLEX determining the best power state to enter. The processing cores can support simplified software sequences for entering power states, offloading the work to the microcode.

[0117] The GPU(s) 908 may include an integrated GPU (alternatively referred to herein as the "iGPU"). The GPU(s) 908 may be programmable and efficient for parallel workloads. The GPU(s) 908 may, in some examples, use an enhanced tensor instruction set. The GPU(s) 908 may include one or more streaming microprocessors, each of which may include an L1 cache (e.g., an L1 cache with a minimum storage capacity of 96 KB), and two or more of the streaming microprocessors may share an L2 cache (e.g., an L2 cache with a storage capacity of 512 KB). In some embodiments, the GPU(s) 908 may include at least eight streaming microprocessors. The GPU(s) 908 can use Compute Application Programming Interface(s) (API(s)). Furthermore, the GPU(s) 908 can utilize one or more parallel computing platforms and / or programming models (e.g.,NVIDIA's CUDA).

[0118] The GPU(s) 908 can be performance-optimized to achieve the best performance in automotive and embedded applications. For example, the GPU(s) 908 can be manufactured using a FinFET field-effect transistor. However, this is not intended to be a limitation, and the GPU(s) 908 can be manufactured using other semiconductor manufacturing processes. Each streaming microprocessor can contain a number of mixed-precision processing cores divided into multiple blocks. For example, and without limitation, 64 PF32 cores and 32 PF64 cores can be divided into four processing blocks. In such an example, each processing block could be allocated 16 FP32 cores, 8 FP64 cores, 16 INT32 cores, two NVIDIA TENSOR CORES for mixed-precision deep learning matrix arithmetic, an L0 instruction cache, a warp scheduler, a dispatch unit and / or a 64 KB register file.Furthermore, streaming microprocessors can include independent parallel integer and floating-point data paths to enable efficient execution of workloads with a mix of computational and addressing operations. Streaming microprocessors can include independent thread scheduling functions to allow for finer-grained synchronization and collaboration between parallel threads. Streaming microprocessors can also include a combined L1 data cache and a shared memory unit to improve performance while simplifying programming.

[0119] The GPU(s) 908 can include high-bandwidth memory (HBM) and / or a 16 GB HBM2 memory subsystem to provide a peak memory bandwidth of approximately 900 GB / second in some examples. In some examples, synchronous graphics random access memory (SGRAM), such as double-data-rate type five synchronous graphics random access memory (GDDR5), can be used in addition to or as an alternative to the HBM memory.

[0120] The GPU(s) 908 can include unified memory technology, including access counters, to allow more accurate migration of memory pages to the processor that accesses them most frequently, thereby improving efficiency for memory areas shared by processors. In some examples, support for address translation services (ATS) can be used to allow the GPU(s) 908 to directly access the page tables of the CPU(s) 906. In such examples, an address translation request can be sent to the CPU(s) 906 if the memory management unit (MMU) of the GPU(s) 908 experiences an erroneous access. In response, the CPU(s) 906 can search its page tables for the virtual-to-physical mapping for the address and send the translation back to the GPU(s) 908.As such, Unified Memory technology can allow a single, unified virtual address space for the memory of both the CPU(s) 906 and the GPU(s) 908, thereby simplifying the programming of the GPU(s) 908 and the porting of applications to the GPU(s) 908.

[0121] Furthermore, GPU(s) 908 can include an access counter that tracks the frequency of GPU(s) 908 access to the memory of other processors. The access counter can help ensure that memory pages are moved to the physical memory of the processor that accesses them most frequently.

[0122] The SoC(s) 904 can include any number of Cache(s) 912, including those described herein. For example, the Cache(s) 912 can include an L3 cache available to both the CPU(s) 906 and the GPU(s) 908 (e.g., connected to both the CPU(s) 906 and the GPU(s) 908). The Cache(s) 912 can include a write-back cache capable of tracking row states, e.g., using a cache coherence protocol (e.g., MEI, MESI, MSI, etc.). The L3 cache can include 4 MB or more, although smaller cache sizes can be used.

[0123] The SoC(s) 904 may include one or more arithmetic logic units (ALUs) that can be used to perform processing with respect to any of the tasks or operations of the Vehicle 900, such as processing DNNs. Furthermore, the SoC(s) 904 may include one or more floating-point units (FPUs) – or other mathematical or numerical coprocessor types – for performing mathematical operations within the system. For example, the SoC(s) 904 may include one or more FPUs that are integrated as execution units within one or more CPU(s) 906 and / or GPU(s) 908.

[0124] The SoC(s) 904 can include one or more Accelerators 914 (e.g., hardware accelerators, software accelerators, or a combination thereof). For example, the SoC(s) 904 can include a hardware acceleration cluster, which may include optimized hardware accelerators and / or a large amount of on-chip memory. The large on-chip memory (e.g., 4 MB SRAM) can allow the hardware acceleration cluster to accelerate neural networks and other computations. The hardware acceleration cluster can be used to complement the GPU(s) 908 and offload some tasks from the GPU(s) 908 (e.g., to free up more GPU cycles for other tasks). As an example, the Accelerator 914 can be used for targeted workloads (e.g., perception, convolutional neural networks (CNNs), etc.).), which are stable enough to be amenable to acceleration. The term “CNN”, as used herein, can include all types of CNNs, including region-based or regional convolutional neural networks (RCNNs) and fast RCNNs (e.g., such as those used for object detection).

[0125] The Accelerator 914 (e.g., the hardware acceleration cluster) can include one or more deep learning accelerators (DLAs). The DLA(s) can include one or more tensor processing units (TPUs), which can be configured to provide an additional ten trillion operations per second for deep learning applications and inference. The TPUs can be accelerators configured and optimized to perform image processing functions (e.g., for CNNs, RCNNs, etc.). The DLA(s) can further be optimized for a specific set of neural network types and floating-point operations, as well as for inference. The design of the DLA(s) can deliver more performance per millimeter than a general-purpose GPU and far surpasses the performance of a CPU.The TPU(s) can perform several functions, including a single-instance folding function that supports, for example, INT8, INT16 and FP16 data types for both features and weights, as well as post-processor functions.

[0126] The DLA(s) can quickly and efficiently execute neural networks, especially CNNs, on processed or unprocessed data for any variety of functions, including, but not limited to: a CNN for object identification and detection using camera sensor data; a CNN for distance estimation using camera sensor data; a CNN for emergency vehicle detection and identification using microphone data; a CNN for facial recognition and vehicle owner identification using camera sensor data; and / or a CNN for security-related events.

[0127] The DLA(s) can perform any function of the GPU(s) 908, and using an inference accelerator, a designer can, for example, use either the DLA(s) or the GPU(s) 908 for any given function. A designer can, for instance, focus the processing of CNNs and floating-point operations on the DLA(s) and leave other functions to the GPU(s) 908 and / or another accelerator 914.

[0128] The Accelerator 914 (e.g., the hardware acceleration cluster) can include one or more programmable vision accelerators (PVAs), which may alternatively be referred to herein as computer vision accelerators. The PVA(s) can be designed and configured to accelerate computer vision algorithms for advanced driver assistance systems (ADAS), autonomous driving applications, augmented reality (AR), and / or virtual reality (VR). The PVA(s) can provide a balance between performance and flexibility. For example, and without limitation, each PVA can include any number of reduced instruction set compute cores (RISC), direct memory access (DMA), and / or any number of vector processors.

[0129] The RISC cores can interact with image sensors (e.g., the image sensors of any of the cameras described herein), image signal processor(s), and / or the like. Each RISC core can include any memory location. The RISC cores can use any of a number of protocols, depending on the implementation. In some examples, the RISC cores can run a real-time operating system (RTOS). The RISC cores can be implemented using one or more integrated circuit devices, application-specific integrated circuits (ASICs), and / or memory devices. For example, the RISC cores can include an instruction cache and / or tightly coupled RAM.

[0130] The DMA can allow components of the PVA(s) to access system memory independently of the CPU(s) 906. The DMA can support any number of features provided to optimize the PVA, including, but not limited to, support for multidimensional addressing and / or circular addressing. In some examples, the DMA can support up to six or more addressing dimensions, which may include block width, block height, block depth, horizontal block gradation, vertical block gradation, and / or depth gradation.

[0131] Vector processors can be programmable processors designed to efficiently and flexibly execute computer vision algorithms and provide signal processing capabilities. In some examples, the PVA may include a PVA core and two vector processing subsystem partitions. The PVA core may include a processor subsystem, DMA engine(s) (e.g., two DMA engines), and / or other peripherals. The vector processing subsystem may act as the primary processing engine of the PVA and may include a vector processing unit (VPU), an instruction cache, and / or vector memory (e.g., VMEM). A VPU core may include a digital signal processor, such as a single-instruction multiple data (SIMD) or very-long instruction word (VLIW) signal processor. The combination of SIMD and VLIW can improve throughput and speed.

[0132] Each vector processor can include an instruction cache and be coupled to dedicated memory. Consequently, in some examples, each vector processor can be configured to operate independently of the others. In other examples, vector processors enclosed in a given PVA can be configured to utilize data parallelism. For example, in some embodiments, the multitude of vector processors enclosed in a single PVA can execute the same computer vision algorithm, but in different regions of an image. In other examples, the vector processors enclosed in a given PVA can simultaneously execute different computer vision algorithms on the same image, or even different algorithms on sequential images or portions of an image.Among other things, any number of PVAs can be included in a hardware acceleration cluster, and any number of vector processors can be included in each of the PVAs. Furthermore, the PVA(s) can include additional ECC (Error Correction Code) memory to improve overall system security.

[0133] The Accelerator 914 (e.g., the hardware acceleration cluster) can include a computer vision network on a single chip and SRAM to provide high-bandwidth, low-latency SRAM for the Accelerator 914. In some examples, the on-chip memory can include at least 4 MB of SRAM, consisting, for example, and without limitation, of eight field-configurable memory blocks accessible by both the PVA and the DLA. Each pair of memory blocks can include an Advanced Peripheral Bus (APB) interface, a configuration circuit, a controller, and a multiplexer. Any type of memory can be used. The PVA and DLA can access the memory via a backbone, providing high-speed memory access for both the PVA and DLA.The backbone can include a computer vision network on a chip that connects the PVA and DLA to the memory (e.g., using the APB).

[0134] The computer vision network can include an interface on a single chip that, prior to the transmission of control signals / addresses / data, ensures that both the PVA and the DLA provide ready-to-use and valid signals. Such an interface can provide separate phases and channels for the transmission of control signals / addresses / data, as well as burst communication for continuous data transfer. This type of interface can conform to ISO 26262 or IEC 61508 standards, although other standards and protocols can also be used.

[0135] In some examples, the SoC(s) 904 may include a real-time ray-tracing hardware accelerator as described in U.S. Patent Application No. 16 / 101,232, filed on August 10, 2018. The real-time ray-tracing hardware accelerator can be used for fast and efficient determination of the positions and extents of objects (e.g., within a world model), for generating real-time visualization simulations, for interpreting radar signals, for synthesizing and / or analyzing sound propagation, for simulating SONAR systems, for general wave propagation simulation, for comparison with LiDAR data for localization purposes, and / or for other functions and / or other purposes. In some embodiments, one or more tree traversal units (TTUs) may be used to perform one or more ray-tracing-related operations.

[0136] The Accelerator 914 (e.g., the hardware accelerator cluster) has a wide range of applications in autonomous driving. The PVA can be a programmable vision accelerator used for key processing stages in ADAS and autonomous vehicles. The PVA's capabilities are well-suited for algorithmic domains requiring predictable processing with low power consumption and low latency. As such, the PVA performs well with semi-dense or dense regular computations, even with small datasets, that require predictable runtimes with low latency and low power consumption. In the context of autonomous vehicle platforms, PVAs are therefore designed to execute classical computer vision algorithms, as they are efficient at object detection and operate with integer mathematics.

[0137] For example, according to one embodiment of the technology, the PVA is used to perform computer stereo vision. A semi-global matching-based algorithm can be used in some examples, although this is not intended to be a limiting factor. Many Level 3-5 autonomous driving applications require motion estimation / stereo matching during operation (on-the-fly) (e.g., structure from motion, pedestrian detection, lane detection, etc.). The PVA can perform computer stereo vision functions with input from two monocular cameras.

[0138] In some examples, the PVA can be used to perform dense optical flow processing, such as processing raw radar data (e.g., using 4D Fast Fourier Transform) to produce processed radar data. In other examples, the PVA is used for time-of-flight depth processing, by processing the raw time-of-flight data to produce, for example, processed time-of-flight data.

[0139] The DLA can be used to operate any type of network to improve steering and driving safety, for example, a neural network that outputs a confidence score for each object detection. Such a confidence score can be interpreted as a probability or as providing a relative "weight" to each detection compared to other detections. This confidence score allows the system to make further decisions about which detections should be considered true positives and not false positives. For example, the system can set a confidence threshold and only consider detections that exceed this threshold as true positives.In an automatic emergency braking (AEB) system, false positive detections would cause the vehicle to automatically initiate emergency braking, which is obviously undesirable. Therefore, only the most confident detections should be considered as triggers for AEB. The DLA can operate a neural network to regress the confidence level. The neural network can use as input at least a subset of parameters, such as the bounding frame dimensions, the ground plane estimate obtained (e.g. from another subsystem), the output of an inertial measurement unit (IMU) sensor 966 correlated with the vehicle's orientation 900, the distance, 3D position estimates of the object obtained from the neural network and / or other sensors (e.g. LiDAR sensor(s) 964 or RADAR sensor(s) 960).

[0140] The SoC(s) 904 may include one or more data stores 916 (e.g., memory). The data store 916 may be on-chip memory of the SoC(s) 904 capable of storing neural networks to be executed on the GPU and / or the DLA. In some examples, the data store 916 may be large enough to store multiple instances of neural networks for redundancy and safety. The data store 916 may include L2 or L3 cache(s) 912. Reference to the data store 916 may include reference to the memory associated with the PVA, DLA, and / or other accelerator(s) 914, as described herein.

[0141] The SoC(s) 904 can include one or more Processor(s) 910 (e.g., embedded processors). The Processor(s) 910 can include a Boot and Power Management Processor, which may be a dedicated processor and subsystem that handles boot power and management functions and associated security enforcement. The Boot and Power Management Processor can be part of the SoC(s) 904 boot sequence and provide runtime power management services. The Boot and Power Management Processor can provide clock and voltage programming, support for low-power system state transitions, management of the SoC(s) 904's thermals and temperature sensors, and / or management of the SoC(s) 904's power supply states.Each temperature sensor can be implemented as a ring oscillator whose output frequency is proportional to the temperature, and the SoC(s) 904 can use ring oscillators to detect temperatures of the CPU(s) 906, GPU(s) 908, and / or accelerator(s) 914. If it is determined that the temperatures exceed a threshold, the boot and power management processor can enter a temperature fault routine and put the SoC(s) 904 into a lower power consumption state and / or put the vehicle 900 into a chauffeur-to-safe-stop mode (e.g., bring the vehicle 900 to a safe stop).

[0142] The 910 processor(s) may further include a set of embedded processors that can serve as an audio processing engine. The audio processing engine can be an audio subsystem that provides full hardware support for multi-channel audio across multiple interfaces and a wide and flexible range of audio I / O interfaces. In some examples, the audio processing engine is a dedicated processor core with a digital signal processor and dedicated RAM.

[0143] The 910 processor(s) may further include an always-on processor engine that can provide the necessary hardware features to support the management of low-power sensors and wake-up use cases. The always-on processor engine may include a processor core, tightly coupled RAM, supporting peripherals (such as timers and interrupt controllers), various I / O controller peripherals, and routing logic.

[0144] The 910 processor(s) can further include a security cluster engine, which comprises a dedicated processor subsystem for handling security management for automotive applications. The security cluster engine can include two or more processor cores, tightly coupled RAM, supporting peripherals (such as timers, an interrupt controller, etc.), and / or routing logic. In a security mode, the two or more cores can operate in lockstep mode, functioning as a single core with comparison logic to detect differences between their operations.

[0145] The 910 processor(s) may also include a real-time camera engine, which may include a dedicated processor subsystem for handling real-time camera management.

[0146] The 910 processor(s) may also include a high dynamic range signal processor, which may include an image signal processor, which is a hardware engine that is part of the camera processing pipeline.

[0147] The 910 processor(s) can include a video image compositor, which may be a processing block (e.g., implemented on a microprocessor) that implements video post-processing functions required by a video playback application to produce the final image for the playback window. The video image compositor can perform lens distortion correction on the 970 wide-angle camera(s), the 974 surround-view camera(s), and / or the in-cabin surveillance camera sensors. An in-cabin surveillance camera sensor is preferably monitored by a neural network running on a separate instance of the Advanced SoC, configured to identify and respond to in-cabin events.An in-cabin system can perform lip reading to activate cellular service and make phone calls, dictate emails, change the vehicle's destination, activate or change the infotainment system and vehicle settings, or provide voice-activated web browsing. Certain functions are only available to the driver when the vehicle is operating in autonomous mode and are otherwise disabled.

[0148] The video image compositor can include enhanced temporal noise reduction for both spatial and temporal noise reduction. For example, if motion occurs in a video, the noise reduction can weight spatial information accordingly, thereby reducing the impact of information provided by adjacent frames. If an image or part of an image does not contain motion, the temporal noise reduction performed by the video image compositor can use information from the previous image to reduce noise in the current image.

[0149] The video image compositor can also be configured to perform stereo equalization on the input stereo lens frames. Furthermore, the video image compositor can be used for user interface composition when the operating system desktop is in use and the GPU(s) 908 are not required to continuously render new surfaces. The video image compositor can also be used to offload the GPU(s) 908, even when the GPU(s) 908 are powered on and actively performing 3D rendering, to improve performance and responsiveness.

[0150] The SoC(s) 904 may further include a Mobile Industry Processor Interface (MIPI) camera serial interface for receiving video and camera input, a high-speed interface, and / or a video input block that can be used for camera and associated pixel input functions. The SoC(s) 904 may also include one or more input / output controllers that can be software-controlled and used to receive I / O signals not designated for a specific role.

[0151] The SoC(s) 904 can also include a wide selection of peripheral interfaces to allow communication with peripheral devices, audio codecs, power management, and / or other devices. The SoC(s) 904 can be used to process data from cameras (e.g., connected via Gigabit Multimedia Serial Link and Ethernet), sensors (e.g., LiDAR sensor(s) 964, RADAR sensor(s) 960, etc., which may be connected via Ethernet), data from bus 902 (e.g., vehicle speed 900, steering wheel position, etc.), data from GNSS sensor(s) 958 (e.g., connected via an Ethernet bus or a CAN bus), etc. The SoC(s) 904 can also include dedicated high-performance mass storage controllers, which can include their own DMA engines and can be used to offload routine data management tasks from the CPU(s) 906.

[0152] The 904 SoC(s) can be an end-to-end platform with a flexible architecture encompassing automation levels 3-5, providing a comprehensive functional safety architecture that leverages and efficiently utilizes computer vision and ADAS techniques for diversity and redundancy, and offers a platform for a flexible, reliable powertrain software stack, along with deep learning tools. The 904 SoC(s) can be faster, more reliable, and even more energy- and space-efficient than conventional systems. For example, the 914 accelerator(s), in combination with the 906 CPU(s), 908 GPU(s), and 916 data storage(s), can provide a fast, efficient platform for Level 3-5 autonomous vehicles.

[0153] This technology thus provides capabilities and functionality that cannot be achieved by conventional systems. For example, computer vision algorithms can be run on CPUs that can be configured using a higher-level programming language, such as C, to execute a wide range of processing algorithms on a wide range of visual data. However, CPUs are often unable to meet the performance requirements of many computer vision applications, such as those related to execution time and power consumption. In particular, many CPUs are unable to execute complex object detection algorithms in real time, which is a requirement for in-vehicle ADAS applications and for practical Level 3-5 autonomous vehicles.

[0154] Unlike conventional systems, the technology described herein, by providing a CPU complex, a GPU complex, and a hardware acceleration cluster, allows multiple neural networks to run simultaneously and / or sequentially and to combine the results to enable autonomous driving functionality at levels 3-5. For example, a CNN running on the DLA or the dGPU (e.g., the GPU(s) 920) can include text and word recognition, allowing the supercomputer to read and understand traffic signs, including those for which the neural network has not been specifically trained. The DLA can further include a neural network capable of identifying and interpreting a sign, providing a semantic understanding of the sign, and passing this semantic understanding to the route planning modules running on the CPU complex.

[0155] As another example, multiple networks can operate simultaneously, as required for Level 3, 4, or 5 driving. For instance, a warning sign consisting of "Caution: Flashing lights indicate black ice" along with an electric light can be interpreted independently or jointly by several neural networks. The sign itself can be identified as a traffic sign by a first neural network (e.g., a trained neural network), while the text "Flashing lights indicate black ice" can be interpreted by a second neural network, which informs the vehicle's route planning software (preferably running on the CPU complex) that black ice is present when flashing lights are detected.The flashing light can be identified by a third, deployed neural network over several frames, thus informing the vehicle's route planning software of the presence (or absence) of flashing lights. All three neural networks can operate simultaneously, for example, within the DLA and / or on the GPU(s) 908.

[0156] In some examples, a CNN for facial recognition and vehicle owner identification can use data from camera sensors to identify the presence of an authorized driver and / or vehicle owner of the Vehicle 900. The always-on sensor processing engine can be used to unlock the vehicle and turn on the lights when the owner approaches the driver's door, and to disable the vehicle in security mode when the owner leaves. In this way, the SoC(s) 904 provides security against theft and / or forced vehicle removal.

[0157] In another example, a CNN for emergency vehicle detection and identification can use data from microphones 996 to detect and identify emergency vehicle sirens. Unlike conventional systems that use general classifiers to detect sirens and manually extract features, the SoC(s) 904 can use the CNN to classify ambient and urban noise as well as visual data. In a preferred embodiment, the CNN, operating on the DLA, is trained to identify the relative approach speed of the emergency vehicle (e.g., using the Doppler effect). The CNN can also be trained to identify emergency vehicles specific to the local area in which the vehicle operates, as identified by the GNSS sensor(s) 958.For example, when operating in Europe, the CNN will attempt to detect European sirens, and when operating in the United States, the CNN will attempt to identify only North American sirens. A control program, with the assistance of ultrasonic sensors 962, can be used, once an emergency vehicle is detected, to execute a safety routine for the emergency vehicle, such as slowing down, pulling over to the side of the road, parking the vehicle, and / or holding the vehicle in neutral until the emergency vehicle(s) has / have passed.

[0158] The vehicle can include one or more CPU(s) 918 (e.g., discrete CPU(s) or dCPU(s)) that can be coupled to the SoC(s) 904 via a high-speed connection (e.g., PCIe). The CPU(s) 918 can include, for example, an x86 processor. The CPU(s) 918 can be used to perform any of a number of functions, including, for example, arbitrating potentially inconsistent results between ADAS sensors and the SoC(s) 904 and / or monitoring the status and state of the Controller(s) 936 and / or Infotainment SoC 930.

[0159] The Vehicle 900 can include one or more GPU(s) 920 (e.g., discrete GPU(s) or dGPU(s)) that can be coupled to the SoC(s) 904 via a high-speed connection (e.g., NVIDIA's NVLINK). The GPU(s) 920 can provide additional artificial intelligence functionality, such as running redundant and / or different neural networks, and can be used to train and / or update neural networks based on input (e.g., sensor data) from sensors in the Vehicle 900.

[0160] The vehicle 900 can also include the network interface 924, which can include one or more wireless antennas 926 (e.g., one or more wireless antennas for different communication protocols, such as a cellular antenna, a Bluetooth antenna, etc.). The network interface 924 can be used to enable wireless connectivity via the internet to the cloud (e.g., to the server(s) 978 and / or other network devices), to other vehicles, and / or to computing devices (e.g., passenger client devices). Communication with other vehicles can be established via a direct connection between the two vehicles and / or an indirect connection (e.g., via networks and the internet). Direct connections can be provided using a vehicle-to-vehicle communication link.The vehicle-to-vehicle communication link can provide the Vehicle 900 with information about vehicles in its vicinity (e.g., vehicles in front of, to the side of, and / or behind the Vehicle 900). This functionality can be part of a cooperative adaptive cruise control feature of the Vehicle 900.

[0161] The network interface 924 can include a SoC that provides modulation and demodulation functionality, enabling the controller(s) 936 to communicate over wireless networks. The network interface 924 can include a radio frequency front end for upconverting baseband to radio frequency and downconverting radio frequency to baseband. The frequency conversions can be performed using known processes and / or superheterodyne processes. In some examples, the radio frequency front-end functionality can be provided by a separate chip. The network interface can include wireless functionality for communication over LTE, WCDMA, UMTS, GSM, CDMA2000, Bluetooth, Bluetooth LE, Wi-Fi, Z-Wave, ZigBee, LoRaWAN, and / or other wireless protocols.

[0162] The vehicle 900 may further include one or more data storage devices 928, which may include off-chip (e.g., off-SoC(s) 904) storage. The data storage device(s) 928 may include one or more storage elements, including RAM, SRAM, DRAM, VRAM, flash, hard disks, and / or other components and / or devices capable of storing at least one data bit.

[0163] The vehicle 900 can also include one or more GNSS sensors 958. The GNSS sensor(s) 958 (e.g., GPS, supported GPS sensors, differential GPS sensors (DGPS), etc.) support mapping, perception, occupancy grid generation, and / or route planning functions. Any number of GNSS sensors 958 can be used, including, for example, and without limitation, one GPS using a USB port with an Ethernet-to-serial (e.g., RS-232) bridge.

[0164] The vehicle 900 can also include one or more RADAR sensor(s) 960. The RADAR sensor(s) 960 can be used by the vehicle 900 for long-range vehicle detection, even in darkness and / or adverse weather conditions. The functional RADAR safety levels can be ASIL B. In some examples, the RADAR sensor(s) 960 can use CAN and / or bus 902 (e.g., to transmit data generated by the RADAR sensor(s) 960) for control and access to object tracking data, with access to Ethernet for raw data. A wide range of RADAR sensor types can be used. For example, and without limitation, the RADAR sensor(s) 960 can be suitable for front, rear, and side RADAR applications. In some examples, a pulse Doppler radar sensor(s) is / are used.

[0165] The 960 RADAR sensor(s) can include various configurations, such as long-range with a narrow field of view, short-range with a wide field of view, side coverage with short-range, etc. In some examples, long-range RADAR can be used for adaptive cruise control functionality. The long-range RADAR systems can provide a wide field of view, achieved through two or more independent scans, for example, within a range of 250 m. The 960 RADAR sensor(s) can assist in distinguishing between stationary and moving objects and can be used by ADAS systems for emergency braking assistance and forward collision warning. Long-range radar sensors can include monostatic multimodal radar with multiple (e.g., six or more) fixed radar antennas and a high-speed CAN and FlexRay interface.In an example with six antennas, the central four antennas can generate a focused beam pattern designed to record the area around vehicle 900 at higher speeds with minimal interference from traffic in adjacent lanes. The other two antennas can expand the field of view, allowing vehicles entering or leaving vehicle 900's lane to be detected quickly.

[0166] Mid-range radar systems, for example, can include a range of up to 960 m (front) or 80 m (rear) and a field of view of up to 42 degrees (front) or 950 degrees (rear). Short-range radar systems can, without limitation, include radar sensors designed for installation at both ends of the rear bumper. When such radar sensor systems are installed at both ends of the rear bumper, they can generate two beams that continuously monitor the blind spot in one direction, both rearward and close to the vehicle.

[0167] Short-range radar systems can be used in the ADAS system to detect a blind spot and / or to assist with a lane change.

[0168] The vehicle 900 can also include one or more ultrasonic sensor(s) 962. The ultrasonic sensor(s) 962 can be positioned on the front, rear, and / or sides of the vehicle 900 and can be used for parking assistance and / or for generating and updating an occupancy grid. A wide range of ultrasonic sensor(s) 962 can be used, and different ultrasonic sensor(s) 962 can be used for different detection ranges (e.g., 2.5 m, 4 m). The ultrasonic sensor(s) 962 can operate according to the functional safety level ASIL B.

[0169] The vehicle 900 can include one or more LiDAR sensors 964. The LiDAR sensor(s) 964 can be used for object and pedestrian detection, emergency braking, collision avoidance, and / or other functions. The LiDAR sensor(s) 964 can operate at functional safety level ASIL B. In some examples, the vehicle 900 can include multiple LiDAR sensors 964 (e.g., two, four, six, etc.) that can use Ethernet (e.g., to provide data to a Gigabit Ethernet switch).

[0170] In some examples, the LiDAR sensor(s) 964 can provide a list of objects and their distances for a 360-degree field of view. A commercially available LiDAR sensor(s) 964, for example, may have a specified range of approximately 900 m with an accuracy of 2 cm to 3 cm and support for a 900 Mbps Ethernet connection. In some examples, one or more non-protruding LiDAR sensors 964 may be used. In such examples, the LiDAR sensor(s) 964 may be implemented as a small device that can be embedded in the front, rear, sides, and / or corners of the vehicle 900. The LiDAR sensor(s) 964 can provide a horizontal field of view of up to 120 degrees and a vertical field of view of up to 35 degrees in such examples, with a range of 200 m, even for objects with low reflectivity.One or more front-mounted LiDAR sensor(s) 964 can be configured for a horizontal field of view between 45 degrees and 135 degrees.

[0171] In some examples, LiDAR technologies, such as 3D flash LiDAR, can also be used. 3D flash LiDAR uses a laser flash as a transmission source to illuminate the vehicle's surroundings up to approximately 200 m. A flash LiDAR unit includes a sensor that records the laser pulse time-of-flight and the reflected light at each pixel, which corresponds to the vehicle's range to the objects. With flash LiDAR, highly accurate and distortion-free images of the surroundings can be generated with each laser flash. In some examples, four flash LiDAR sensors can be used, one on each side of the vehicle. Available 3D flash LiDAR systems include a solid-state 3D LiDAR camera with a rigid array and no moving parts except for a fan (e.g., a non-scanning LiDAR device).The Blitz LiDAR device can use a 5-nanosecond Class I (eye-safe) laser pulse per frame and capture reflected laser light in the form of 3D distance point clouds and co-recorded intensity data. By using Blitz LiDAR, and because Blitz LiDAR is a solid-state device with no moving parts, the LiDAR sensor(s) 964 may be less susceptible to motion blur, vibration, and / or shock.

[0172] The vehicle may further include one or more IMU sensor(s) 966. The IMU sensor(s) 966 may be located in the center of the rear axle of the vehicle 900 in some examples. The IMU sensor(s) 966 may include, for example, without limitation, one or more accelerometers, one or more magnetometers, one or more gyroscopes, one or more magnetic compasses, and / or other sensor types. In some examples, such as in six-axis applications, the IMU sensor(s) 966 may include accelerometers and gyroscopes, while in nine-axis applications, the IMU sensor(s) 966 may include accelerometers, gyroscopes, and magnetometers.

[0173] In some embodiments, the IMU sensor(s) 966 can be implemented as a miniature, high-performance GPS-aided inertial navigation system (GPS / INS) that combines microelectromechanical system (MEMS) inertial sensors, a high-sensitivity GPS receiver, and advanced Kalman filter algorithms to provide estimates of position, velocity, and attitude. As such, in some examples, the IMU sensor(s) 966 can allow the vehicle 900 to estimate its course without requiring input from a magnetic sensor by directly observing and correlating velocity changes from a GPS to the IMU sensor(s) 966. In some examples, the IMU sensor(s) 966 and the GNSS sensor(s) 958 can be combined in a single integrated unit.

[0174] The vehicle can include one or more microphone(s) 996, which are placed in and / or around the vehicle 900. The microphone(s) 996 can be used, among other things, for emergency vehicle detection and identification.

[0175] The vehicle may also include any number of camera types, including one or more stereo cameras 968, wide-angle cameras 970, infrared cameras 972, surround-view cameras 974, long-range cameras and / or medium-range cameras 998, and / or other camera types. The cameras may be used to capture image data around the entire periphery of the vehicle 900. The types of cameras used depend on the embodiments and requirements for the vehicle 900, and any combination of camera types may be used to provide the necessary coverage around the vehicle 900. Furthermore, the number of cameras may vary depending on the embodiment. For example, the vehicle may include six cameras, seven cameras, ten cameras, twelve cameras, and / or any other number of cameras.The cameras can, for example, support Gigabit Multimedia Serial Link (GMSL) and / or Gigabit Ethernet communication without restriction. Each camera is described in more detail below. Fig. 9A and Fig. 9B described.

[0176] The vehicle 900 may also include one or more vibration sensors 942. The vibration sensor(s) 942 may measure vibrations of vehicle components, such as the axle(s). For example, changes in vibration may indicate a change in the road surface. In another example, if two or more vibration sensors 942 are used, the differences between the vibrations may be used to determine the friction or slipperiness of the road surface (e.g., if the difference in vibration is between a driven axle and a freely rotating axle).

[0177] The vehicle 900 may include an ADAS system 938. The ADAS system 938 may include a SoC in some examples. The ADAS system 938 may include systems for autonomous / adaptive / automatic cruise control (ACC), cooperative adaptive cruise control (CACC), forward crash warning (FCW), automatic emergency braking (AEB), lane departure warnings (LDW), lane keep assist (LKA), blind spot warning (BSW), rear cross-traffic warning (RCTW), collision warning systems (CWS), lane centering (LC), and / or other features and functions.

[0178] The ACC systems can use one or more radar sensors (960), LiDAR sensors (964), and / or one or more cameras. The ACC systems can include longitudinal ACC and / or lateral ACC. Longitudinal ACC monitors and controls the distance to the vehicle immediately in front of the vehicle (900) and automatically adjusts the vehicle's speed to maintain a safe distance from vehicles ahead. Lateral ACC maintains the distance and advises the vehicle (900) to change lanes if necessary. Lateral ACC is related to other ADAS applications, such as LCA and CWS.

[0179] CACC uses information from other vehicles, which can be received via the network interface 924 and / or the wireless antenna(s) 926 from other vehicles either wirelessly or indirectly via a network connection (e.g., the internet). Direct connections can be provided through a vehicle-to-vehicle (V2V) communication link, while indirect connections can be an infrastructure-to-vehicle (I2V) communication link. Generally, the V2V communication concept provides information about vehicles immediately ahead (e.g., vehicles directly in front of and in the same lane as vehicle 900), while the I2V communication concept provides information about traffic at a greater distance. CACC systems can incorporate one or both of the I2V and V2V information sources.Thanks to the information about the vehicles in front of the vehicle 900, the CACC can be more reliable and has the potential to improve traffic flow and reduce congestion on the road.

[0180] FCW systems are designed to alert the driver to a hazard, allowing the driver to take corrective action. FCW systems utilize a forward-facing camera and / or RADAR 960 sensors, coupled with a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component. FCW systems can provide a warning, for example, in the form of an audible signal, a visual warning, a vibration, and / or a rapid braking pulse.

[0181] AEB systems detect an impending forward collision with another vehicle or object and can automatically apply the brakes if the driver does not take corrective action within a specified time or distance parameter. AEB systems use one or more forward-facing cameras and / or one or more radar sensors coupled to a dedicated processor, DSP, FPGA, and / or ASIC. When the AEB system detects a hazard, it will typically first warn the driver to take corrective action to avoid the collision. If the driver does not take corrective action, the AEB system will automatically apply the brakes in an effort to avoid or at least mitigate the impact of the predicted collision. AEB systems may include techniques such as dynamic brake assist and / or anticipatory braking.

[0182] Lane Departure Warning (LDW) systems provide visual, audible, and / or tactile warnings, such as steering wheel or seat vibrations, to alert the driver if the vehicle crosses lane markings. An LDW system will not activate if the driver indicates an intention to leave the lane, for example, by activating a turn signal. LDW systems may use forward-facing cameras coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0183] LKA systems are a variant of LDW systems. LKA systems provide steering or braking input to correct vehicle 900 if it begins to leave its lane.

[0184] Blind Spot Warning (BSW) systems detect vehicles in a car's blind spot and warn the driver. BSW systems can provide a visual, audible, and / or tactile alert to indicate that merging or changing lanes is unsafe. The system can provide an additional warning if the driver uses a turn signal. BSW systems can use rear-facing camera(s) and / or radar sensor(s) coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibration component.

[0185] RCTW systems can provide visual, audible, and / or tactile alerts when an object outside the reversing camera's field of view is detected while the vehicle is reversing. Some RCTW systems include AEB to ensure the vehicle's brakes are applied to avoid a collision. RCTW systems can use one or more rear-facing radar sensors coupled to a dedicated processor, DSP, FPGA, and / or ASIC, which is electrically coupled to driver feedback, such as a display, speaker, and / or vibrating component.

[0186] Conventional ADAS systems can be prone to false positives, which can be annoying and distracting for a driver, but are generally not catastrophic because the ADAS systems warn the driver and allow them to decide whether a safety condition actually exists and act accordingly. In an autonomous vehicle 900, however, the vehicle 900 itself must decide, in the case of conflicting results, whether the result should be considered by a primary computer or a secondary computer (e.g., a first controller 936 or a second controller 936). For example, the ADAS system 938 in some embodiments may be a backup and / or secondary computer that provides perceptual information to a rationality module of the backup computer. The rationality monitor of the backup computer may run redundant software on hardware components to detect perceptual errors and dynamic driving tasks.Output from the ADAS system 938 can be provided to a higher-level MCU. If output from the primary and secondary computers conflict, the higher-level MCU must decide how to resolve the conflict to ensure safe operation.

[0187] In some examples, the primary computer can be configured to provide the higher-level MCU with a confidence score indicating its level of trust in the chosen result. If the confidence score exceeds a certain threshold, the higher-level MCU can follow the primary computer's guidance, regardless of whether the secondary computer provides a conflicting or inconsistent result. If the confidence score does not reach a certain threshold, and if the primary and secondary computers display different results (e.g., a conflict), the higher-level MCU can arbitrate between the computers to determine the appropriate result.

[0188] The higher-level MCU can be configured to run a neural network(s) trained and configured to determine, based on output from the primary and secondary computers, the conditions under which the secondary computer will generate false alarms. Thus, the neural network(s) in the higher-level MCU can learn when the secondary computer's output can be trusted and when it cannot. For example, if the secondary computer is a radar-based FCW system, the neural network(s) in the higher-level MCU can learn when the FCW system identifies metallic objects that are not actually hazards, such as a drainage grate or a manhole cover, triggering an alarm.Similarly, if the secondary computer is a camera-based lane departure warning (LDW) system, a neural network in the higher-level MCU can learn to override the LDW when cyclists or pedestrians are present and leaving the lane is indeed the safest maneuver. In embodiments that include one or more neural networks running on the higher-level MCU, the higher-level MCU can include at least one DLA or GPU suitable for running the neural network(s) with associated memory. In preferred embodiments, the higher-level MCU can include a component of the SoC(s) 904 and / or be included as such.

[0189] In other examples, the ADAS System 938 can include a secondary computer that performs ADAS functionality using traditional computer vision rules. As such, the secondary computer can use classic computer vision rules (if-then), and the presence of one or more neural networks in the higher-level MCU can improve reliability, safety, and performance. For example, the overall system becomes more fault-tolerant due to the different implementation and intentional non-identity, particularly with regard to errors caused by software functionality (or the software-hardware interface).For example, if there is a software bug or error in the software running on the primary computer, and the non-identical software code running on the secondary computer provides the same overall result, the higher-level MCU can have greater confidence that the overall result is correct and that the bug in the software or hardware on the primary computer does not cause a significant error.

[0190] In some examples, the output of the ADAS system 938 can be fed into the perception block of the primary computer and / or into the dynamic driving task block of the primary computer. For example, if the ADAS system 938 displays a forward collision warning due to an object immediately in front of it, the perception block can use this information in object identification. In other examples, the secondary computer may have its own neural network that is trained, thus reducing the risk of false positives, as described herein.

[0191] The Vehicle 900 may further include the Infotainment SoC 930 (e.g., an In-Vehicle Infotainment System (IVI)). Although illustrated and described as an SoC, the Infotainment System may not be an SoC and may include two or more discrete components. The Infotainment SoC 930 may include a combination of hardware and software that can be used to provide the Vehicle 900 with audio (e.g., music, a personal digital assistant, navigation instructions, news, radio, etc.), video (e.g., television, movies, streaming, etc.), telephone (e.g., hands-free calling), network connectivity (e.g., LTE, Wi-Fi, etc.), and / or information services (e.g., navigation systems, reversing parking assistance, a radio data system, vehicle-related information such as fuel level, total distance traveled, brake fluid level, oil level, door open / closed status, air filter information, etc.). The Infotainment SoC 930 may, for example,This includes radios, record players, navigation systems, video players, USB and Bluetooth connectivity, car computers, in-car entertainment, WiFi, steering wheel audio controls, hands-free voice control, a heads-up display (HUD), an HMI display 934, a telematics device, a control panel (e.g., for controlling and / or interacting with various components, functions, and / or systems), and / or other components. The Infotainment SoC 930 can also be used to provide information (e.g., visual and / or audible) to the vehicle's user(s), such as information from the ADAS system 938, autonomous driving information like planned vehicle maneuvers, trajectories, environmental information (e.g., intersection information, vehicle information, road information, etc.), and / or other information.

[0192] The Infotainment SoC 930 can include GPU functionality. The Infotainment SoC 930 can communicate with other devices, systems, and / or components of the vehicle 900 via bus 902 (e.g., CAN bus, Ethernet, etc.). In some examples, the Infotainment SoC 930 can be coupled with a higher-level MCU so that the infotainment system's GPU can perform some self-regulating functions if the primary controller(s) 936 (e.g., the primary and / or backup computer of the vehicle 900) fails. In such an example, the Infotainment SoC 930 can put the vehicle 900 into a chauffeur-to-safe-stop mode, as described herein.

[0193] The vehicle 900 may further include an instrument cluster 932 (e.g., a digital instrument panel, an electronic instrument cluster, a digital instrument panel, etc.). The instrument cluster 932 may include a controller and / or a supercomputer (e.g., a discrete controller or a discrete supercomputer). The instrument cluster 932 may include a set of instruments such as a speedometer, fuel gauge, oil pressure gauge, tachometer, odometer, turn signals, gear shift indicator, seatbelt warning light(s), parking brake warning light(s), engine malfunction light(s), airbag (SRS) system information, lighting controls, safety system controls, navigation information, etc. In some examples, information may be displayed and / or shared between the infotainment SoC 930 and the instrument cluster 932.As such, the Instrument Cluster 932 can be included as part of the Infotainment SoC 930, or vice versa.

[0194] Fig. 9D is a system representation in accordance with some embodiments of the present disclosure for communication between one or more cloud-based server(s) and the exemplary autonomous vehicle 900 of Fig. 9A. The System 976 can include one or more Servers 978, one or more Network(s) 990, and vehicles, including the Vehicle 900. The Server(s) 978 can include a variety of GPUs 984(A)-984(H) (collectively referred to herein as GPUs 984), PCIe Switches 982(A)-982(D) (collectively referred to herein as PCIe Switches 982), and / or CPUs 980(A)-980(B) (collectively referred to herein as CPUs 980). The GPUs 984, CPUs 980, and PCIe Switches can be interconnected using high-speed links, such as, but not limited to, NVIDIA's NVLink 988 interfaces and / or PCIe 986 links. In some examples, the GPUs 984 are connected via an NVLink and / or NVSwitch SoC, and the GPUs 984 and PCIe switches 982 are connected via PCIe links. Although eight GPUs 984, two CPUs 980, and two PCIe switches are illustrated, this is not intended to be limiting.Depending on the configuration, each Server 978 can include any number of GPUs 984, CPUs 980, and / or PCIe switches. For example, the Server 978 can include eight, sixteen, thirty-two, and / or more GPUs 984.

[0195] The server(s) 978 can receive image data from the network(s) 990 and the vehicles, representing images showing unexpected or changed road conditions, such as recently started roadworks. The server(s) 978 can transmit neural networks 992, updated neural networks 992, and / or map information 994, including information regarding traffic and road conditions, to the vehicles via the network(s) 990 and the vehicles. Updates to the map information 994 may include updates to the HD map 922, such as information about construction sites, potholes, detours, floods, and / or other obstacles.In some examples, the neural networks 992, the updated neural networks 992 and / or the map information 994 may result from new training and / or new experiences represented in data received from any number of vehicles in the vicinity, and / or based on training performed at a data center (e.g. using server(s) 978 and / or other servers).

[0196] Server 978 can be used to train machine learning models (e.g., neural networks) based on training data. The training data can be generated using the vehicles and / or in a simulation (e.g., using a game engine). In some examples, the training data is tagged (e.g., if the neural network benefits from supervised learning) and / or preprocessed, while in other examples, the training data is untagged and / or preprocessed (e.g., if the neural network does not require supervised learning).Training can be performed according to one or more classes of machine learning techniques, including, but not limited to, classes such as: supervised training, semi-supervised training, unsupervised training, self-learning, reinforcement learning, federated learning, transfer learning, feature learning (including principal component and cluster analysis), multilinear subspace learning, manifold learning, representational learning (including sparse dictionary learning), rule-based machine learning, anomaly detection, and any other variants or combinations thereof. After the machine learning models have been trained, they can be used by the vehicles (e.g., transmitted to the vehicles via network(s) 990) and / or the machine learning models can be used by server(s) 978 for remote monitoring of the vehicles.In some embodiments, a stereo depth perception model 120, as described herein, can be trained using a server / servers 978.

[0197] In some examples, the Server 978 can receive data from the vehicles and apply the data to current real-time neural networks for intelligent real-time inference. The Server 978 can include supercomputers for deep learning and / or dedicated AI computers powered by GPU(s) 984, such as NVIDIA's DGX and DGX Station machines. However, in some examples, the Server 978 can also include deep learning infrastructure that uses only CPU-powered data centers.

[0198] The deep learning infrastructure of the server(s) 978 can be capable of fast real-time inference and use this capability to evaluate and verify the state of the processors, software, and / or associated hardware in the vehicle 900. For example, the deep learning infrastructure can receive periodic updates from the vehicle 900, such as a sequence of images and / or objects that the vehicle 900 has located within that sequence of images (e.g., through computer vision and / or other machine learning object classification techniques).The deep learning infrastructure can operate its own neural network to identify the objects and compare them with the objects identified by the vehicle 900, and if the results do not match and the infrastructure concludes that the AI ​​in the vehicle 900 is faulty, the server 978 can send a signal to the vehicle 900 and instruct a fail-safe computer of the vehicle 900 to take over control, notify the passengers and perform a safe parking maneuver.

[0199] The Server 978 can include GPU(s) 984 and one or more programmable inference accelerators (e.g., NVIDIA's TensorRT). The combination of GPU-powered servers and inference acceleration can enable real-time responsiveness. In other examples, such as when performance is less critical, servers powered by CPUs, FPGAs, and other processors can be used for inference purposes. EXAMPLE CALCULATION DEVICE

[0200] Fig. 10 a block representation of an exemplary computing device(s) 1000 suitable for use in implementing some embodiments of the present disclosure. The computing device 1000 can include a connection system 1002 that directly or indirectly couples the following devices: memory 1004, one or more central processing units (CPUs) 1006, one or more graphics processing units (GPUs) 1008, a communication interface 1010, input / output (I / O) ports 1012, input / output components 1014, a power supply 1016, one or more presentation components 1018 (e.g., display(s)), and one or more logic units 1020. In at least one embodiment, the computing device(s) 1000 can include one or more virtual machines (VMs) and / or any of its components can include virtual components (e.g., virtual hardware components).For non-limiting examples, one or more of the GPUs 1008 may comprise one or more vGPUs, one or more of the CPUs 1006 may comprise one or more vCPUs, and / or one or more of the logic units 1020 may comprise one or more virtual logic units. As such, a computing device(s) 1000 may include discrete components (e.g., a complete GPU dedicated to the computing device 1000), virtual components (e.g., a portion of a GPU dedicated to the computing device 1000), or a combination thereof. In some embodiments, one or more functions of the depth image anomaly processor 130, depth image anomaly identification data 132, and / or depth image anomaly correction status 134 may be implemented, at least in part, using a computing device 1000.

[0201] Although the various blocks of Fig. Where components 10 are shown to be connected via lines through the connection system 1002, this is not intended to be restrictive and serves only for clarity. For example, in some embodiments, a presentation component 1018, such as a display device, can be considered an I / O component 1014 (e.g., if the display is a touchscreen). As another example, the CPUs 1006 and / or GPUs 1008 can include memory (e.g., the memory 1004 can be representative of a storage device, in addition to the memory of the GPUs 1008, the CPUs 1006, and / or other components). As such, the computing device of Fig. 10 is for illustrative purposes only. No distinction is made between categories such as "workstation", "server", "laptop", "desktop", "tablet", "client device", "mobile device", "handheld device", "game console", "electronic control unit (ECU)", "virtual reality system" and / or other device or system types, as all fall under the category of computing device. Fig. 10 will be considered.

[0202] In some embodiments, one or more functions of the depth image anomaly processor 130, depth image anomaly identification data 132 and / or depth image anomaly correction status 134 can be implemented at least partially by code that runs on CPUs 1006 and / or GPUs 1008.

[0203] The connection system 1002 can represent one or more links or buses, such as an address bus, a data bus, a control bus, or a combination thereof. The connection system 1002 can include one or more bus or link types, such as an Industry Standard Architecture (ISA) bus, an Extended Industry Standard Architecture (EISA) bus, a Video Electronics Standard Association (VESA) bus, a Peripheral Component Connection (PCI) bus, a Peripheral Component Connection Express (PCIe) bus, and / or other bus or link types. In some embodiments, there are direct connections between components. For example, the CPU 1006 can be directly connected to the memory 1004. Furthermore, the CPU 1006 can be directly connected to the GPU 1008. In the case of a direct or point-to-point connection between components, the connection system 1002 can include a PCIe link to implement the connection.In these examples, the computing device 1000 does not need to include a PCI bus.

[0204] Memory 1004 can include any of a range of computer-readable media. The computer-readable media can be any available media accessible to the computing device 1000. The computer-readable media can include both volatile and non-volatile media, and both removable and non-removable media. As an example, and not as a limitation, the computer-readable media can include computer storage media and communication media.

[0205] Computer storage media can include both volatile and non-volatile media and / or removable and non-removable media, implemented in any method or technology for storing information, such as computer-readable instructions, data structures, program modules, and / or other data types. For example, memory can store 1004 computer-readable instructions (e.g., representing a program and / or program element, such as an operating system).Computer storage media may include, but are not limited to, RAM, ROM, EEPROM, flash memory or other storage technology, CD-ROM, Digital Versatile Discs (DVDs) or other optical disk storage, magnetic cartridges, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store the desired information and that can be accessed by the computing device 1000. As used herein, computer storage media do not, per se, include signals.

[0206] Computer storage media can embody computer-readable instructions, data structures, program modules, and / or other data types in a modulated data signal, such as a carrier wave or other transport mechanism, and include information delivery media. The term "modulated data signal" can refer to a signal for which one or more of its characteristics are set or modified in a way that encodes information in the signal. By way of example, and not limited to this, computer storage media can include wired media, such as a wired network or a directly wired connection, and wireless media, such as acoustic, RF, infrared, and other wireless media. Combinations of any of the foregoing should also be included in the scope of computer-readable media.

[0207] The CPU(s) 1006 can be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 in order to perform one or more of the procedures and / or processes described herein. The CPU(s) 1006 can include one or more cores (e.g., one, two, four, eight, twenty-eight, seventy-two, etc.) capable of handling a plurality of software threads concurrently. The CPU(s) 1006 can include any type of processor and can include different types of processors depending on the type of computing device 1000 implemented (e.g., processors with fewer cores for mobile devices and processors with more cores for servers).For example, depending on the type of computing device 1000, the processor can be an Advanced RISC Machines (ARM) processor implemented using Reduced Instruction Set Computing (RISC), or an x86 processor implemented using Complex Instruction Set Computing (CISC). The computing device 1000 can include one or more CPUs 1006 in addition to one or more microprocessors or supplementary coprocessors, such as math coprocessors.

[0208] In addition to or as an alternative to the CPU(s) 1006, the GPU(s) 1008 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the procedures and / or processes described herein. One or more of the GPU(s) 1008 may be an integrated GPU (e.g., in one or more of the CPU(s) 1006) and / or one or more of the GPU(s) 1008 may be a discrete GPU. In embodiments, one or more of the GPU(s) 1008 may be a coprocessor of one or more of the CPU(s) 1006. The GPU(s) 1008 may be used by the computing device 1000 to render graphics (e.g., 3D graphics) or to perform general-purpose computing. For example, the GPU(s) 1008 can be used for general-purpose computing on GPUs (GPGPU).The GPU(s) 1008 can include hundreds or thousands of cores capable of handling hundreds or thousands of software threads simultaneously. The GPU(s) 1008 can generate pixel data for output images in response to rendering commands (e.g., rendering commands from the CPU(s) 1006 received via a host interface). The GPU(s) 1008 can include graphics memory, such as display memory, for storing pixel data or any other suitable data, such as GPGPU data. The display memory can be included as part of the memory 1004. The GPU(s) 1008 can include two or more GPUs operating in parallel (e.g., via a link). The link can connect the GPUs directly (e.g., using an NVLINK) or it can connect the GPUs via a switch (e.g., using an NVSwitch).When combined, each GPU can generate 1008 pixel data or GPGPU data for different dividers, or output for different outputs (e.g., a first GPU for a first image and a second GPU for a second image). Each GPU can include its own memory or share memory with other GPUs.

[0209] In addition to or alternatively to the CPU(s) 1006 and / or the GPU(s) 1008, the logic unit(s) 1020 may be configured to execute at least some of the computer-readable instructions to control one or more components of the computing device 1000 to perform one or more of the methods and / or processes described herein. In embodiments, the CPU(s) 1006, the GPU(s) 1008, and / or the logic unit(s) 1020 may discretely or jointly perform any combination of the methods, processes, and / or parts thereof. One or more of the logic units 1020 may be part of and / or integrated into one or more of the CPU(s) 1006 and / or the GPU(s) 1008, and / or one or more of the logic units 1020 may be discrete components or otherwise external to the CPU(s) 1006 and / or GPU(s). the GPU(s) 1008.In embodiments, one or more of the logic units 1020 can be a coprocessor of one or more of the CPU(s) 1006 and / or one of the GPU(s) 1008.

[0210] Examples of logic unit(s) 1020 include: one or more processing cores and / or components thereof, such as data processing units (DPUs), tensor cores (TCs), tensor processing units (TPUs), pixel visual cores (PVCs), vision processing units (VPUs), graphics processing clusters (GPCs), texture processing clusters (TPCs), streaming multiprocessors (SMs), tree traversal units (TTUs), artificial intelligence accelerators (AIAs), deep learning accelerators (DLAs), arithmetic logic units (ALUs), application-specific integrated circuits (ASICs), floating-point units (FPUs), and input / output (I / O) elements. Peripheral Component Connector (PCI) or Peripheral Component Connector Express (PCIe) components and / or the like.

[0211] The communication interface 1010 can include one or more receivers, transmitters, and / or transceivers that allow the computing device 1000 to communicate with other computing devices via an electronic communication network, including wired and / or wireless communication. The communication interface 1010 can include components and functionality to allow communication over any number of different networks, such as wireless networks (e.g., Wi-Fi, Z-Wave, Bluetooth, Bluetooth LE, ZigBee, etc.), wired networks (e.g., communicating via Ethernet or InfiniBand), low-power wide-area networks (e.g., LoRaWAN, SigFox, etc.), and / or the Internet.In one or more embodiments, the logic unit(s) 1020 and / or communication interface 1010 may include one or more data processing units (DPUs) to transfer data received via a network and / or the connection system 1002 directly to (e.g., a memory of) one or more GPU(s) 1008.

[0212] The I / O ports 1012 allow the computing device 1000 to be logically coupled with other devices, including the I / O components 1014, the presentation component(s) 1018, and / or other components, some of which may be built into (e.g., integrated with) the computing device 1000. Illustrative I / O components 1014 include a microphone, mouse, keyboard, joystick, gamepad, game controller, satellite table, scanner, printer, wireless device, etc. The I / O components 1014 can provide a natural user interface (NUI) that processes air gestures, speech, or other physical input generated by a user. In some cases, input can be transmitted to an appropriate network element for further processing.A NUI can implement any combination of speech recognition, pen recognition, facial recognition, biometric recognition, gesture recognition both on-screen and adjacent to the screen, air gestures, head and eye tracking, and touch recognition (as described in more detail below), associated with a display of the Computing Device 1000. The Computing Device 1000 can include depth cameras, such as stereoscopic camera systems, infrared camera systems, RGB camera systems, touchscreen technology, and combinations thereof for gesture detection and recognition. Furthermore, the Computing Device 1000 can include accelerometers or gyroscopes (e.g., as part of an inertial measurement unit, IMU) that allow motion detection.In some examples, the output of the accelerometers or gyroscopes can be used by the computing device 1000 to render immersive augmented reality or virtual reality.

[0213] The power supply 1016 can include a hardwired power supply, a battery power supply, or a combination thereof. The power supply 1016 can provide power to the computing device 1000 to allow the components of the computing device 1000 to operate.

[0214] The presentation component(s) 1018 may include a display (e.g., a monitor, a touchscreen, a television screen, a heads-up display (HUD), other display types, or a combination thereof), speakers, and / or other presentation components. The presentation component(s) 1018 may receive data from other components (e.g., the GPU(s) 1008, the CPU(s) 1006, DPUs, etc.) and output the data (e.g., as an image, video, sound, etc.). In some embodiments, the HMI 124 may be implemented by and / or comprise one or more of the presentation component(s) 1018. EXEMPLARY DATA CENTER

[0215] Fig. Figure 11 illustrates an exemplary data center 1100 that can be used in at least one embodiment of the present disclosure. The data center 1100 can include a data center infrastructure layer 1110, a framework layer 1120, a software layer 1130, and / or an application layer 1140.

[0216] As in Fig. As shown in Figure 11, the data center infrastructure layer 1110 can include a resource orchestrator 1112, clustered compute resources 1114, and node compute resources (“node RR”) 1116(1)-1116(N), where “N” represents an integer, positive number. In at least one embodiment, Node-RR 1116(1)-1116(N) can include, but are not limited to, any number of central processing units (CPUs) or other processors (including DPUs, accelerators, field-programmable gate arrays (FPGAs), graphics processing units or graphics processing units (GPUs), etc.), storage devices (e.g., dynamic read-only memories), storage devices (e.g., solid-state or disk drives), network input / output (NW I / O) devices, network switches, virtual machines (VMs), power modules and / or cooling modules, etc. In some embodiments, one or more Node-RRs can be controlled by the Node-RR1116(1)-1116(N) correspond to a server that has one or more of the computing resources mentioned above. Furthermore, in some embodiments, the node RR 1116(1)-1116(N) may include one or more virtual components, such as vGPUs, vCPUs, and / or the like, and / or may correspond to one or more of the node RR 1116(1)-1116(N) of a virtual machine (VM).

[0217] In some embodiments, one or more functions of the depth image anomaly processor 130, depth image anomaly identification data 132 and / or depth image anomaly correction status 134 can be implemented at least partially by code executed on one or more of nodes RR 1116(1)-1116(N).

[0218] In at least one embodiment, grouped compute resources 1114 can include separate groupings of node RR 1116 located in one or more racks (not shown), or many racks located in data centers at different geographic locations (also not shown). Separate groupings of node RR 1116 within grouped compute resources 1114 can include grouped compute, network, storage, or memory resources that can be configured or allocated to support one or more workloads. In at least one embodiment, multiple node RR 1116, including CPUs, GPUs, DPUs, and / or other processors, can be grouped in one or more racks to provide compute resources to support one or more workloads.The one or more racks can also include any number of power modules, cooling modules and / or network switches in any combination.

[0219] The resource orchestrator 1112 can configure or otherwise control one or more node RR 1116(1)-1116(N) and / or grouped compute resources 1114. In at least one embodiment, the resource orchestrator 1112 can include a software design infrastructure (SDI) management entity for the data center 1100. The resource orchestrator 1112 can include hardware, software, or a combination thereof.

[0220] In at least one embodiment, as in Fig. As shown in Figure 11, a framework layer 1120 can include a job scheduler 1133, a configuration manager 1134, a resource manager 1136, and / or a distributed file system 1138. The framework layer 1120 can include a framework to support software 1132 of software layer 1130 and / or one or more applications 1142 of application layer 1140. The software 1132 or application 1142 can each include web-based service software or applications, such as those provided by Amazon Web Services, Google Cloud, and Microsoft Azure. The framework layer 1120 can, but is not limited to, a type of free and open-source software web application framework, such as Apache Spark. TM(hereinafter "Spark"), which can use the distributed file system 1138 for large-scale data processing (e.g., "big data"). In at least one embodiment, the job scheduler 1133 can include a Spark driver to facilitate the scheduling of workloads supported by various layers of the data center 1100. The configuration manager 1134 can be capable of configuring various layers, such as the software layer 1130 and the framework layer 1120, including Spark and the distributed file system 1138, to support large-scale data processing. The resource manager 1136 can be capable of managing clustered or grouped compute resources that are allocated or assigned to support the distributed file system 1138 and the job scheduler 1133.In at least one embodiment, the clustered or grouped computing resources can include a grouped computing resource 1114 at the data center infrastructure layer 1110. The resource manager 1136 can coordinate with the resource orchestrator 1112 to manage these allocated or assigned computing resources.

[0221] In some embodiments, one or more functions of the depth image anomaly processor 130, depth image anomaly identification data 132 and / or depth image anomaly correction status 134 can be implemented at least partially using one or more applications 1142 and / or software 1132.

[0222] In at least one embodiment, software 1132, which is enclosed in software layer 1130, can include software used by at least parts of the node RR 1116(1)-1116(N), grouped computing resources 1114, and / or distributed file system 1138 of framework layer 1120. One or more types of software can include, but are not limited to, internet web page search software, email virus scanning software, database software, and streaming video content software.

[0223] In at least one embodiment, one or more application(s) 1142 enclosed in the application layer 1140 may include one or more types of applications used by at least parts of the node RR 1116(1)-1116(N), grouped compute resources 1114, and / or distributed file system 1138 of the framework layer 1120. One or more types of applications may include, but are not limited to, any number of a genomics application, a cognitive computation application, and a machine learning application, including training or inference software, machine learning framework software (e.g., PyTorch, TensorFlow, Caffe, etc.), and / or other machine learning applications used in conjunction with one or more embodiments.

[0224] In at least one embodiment, any configuration manager 1134, resource manager 1136, and resource orchestrator 1112 can implement any number and any type of self-modifying operations based on any set and any type of data captured in any technically feasible manner. Self-modifying operations can relieve a data center operator of data center 1100 of the burden of potentially making poor configuration decisions and potentially avoiding underutilized and / or poorly performing parts of a data center.

[0225] Data Center 1100 may include tools, services, software, or other resources to train one or more machine learning models or to predict or infer information using one or more machine learning models according to one or more embodiments described herein. For example, one or more machine learning models may be trained by calculating weight parameters according to a neural network architecture using software and / or computing resources as described above in relation to Data Center 1100.In at least one embodiment, trained or deployed machine learning models corresponding to one or more neural networks can be used to infer and predict information regarding Data Center 1100 using the resources described above, by using weight parameters calculated via one or more training techniques, such as, but not limited to, those described herein.

[0226] In at least one embodiment, the data center can use 1100 CPUs, application-specific integrated circuits (ASICs), GPUs, FPGAs, and / or other hardware (or equivalent virtual computing resources) to perform training and / or inference using the resources described above. Additionally, one or more of the software and / or hardware resources described above can be configured as a service to allow users to train or perform information inference, such as image recognition, speech recognition, or other artificial intelligence services. EXEMPLARY NETWORK ENVIRONMENTS

[0227] Network environments suitable for use in implementing embodiments of the disclosure may include one or more client devices, servers, network-attached storage (NAS), other backend devices, and / or other device types. The client devices, servers, and / or other device types (e.g., each device) may be connected to one or more instances of the computing device(s). Fig. 10. Each device can include similar components, features, and / or functionality to the computing device(s) 1000. Furthermore, if backend devices (e.g., servers, NAS, etc.) are implemented, the backend devices can be included as part of a data center 1100; an example of this is described in more detail below with reference to… Fig. 11 described.

[0228] Components of a network environment can communicate with each other over a network, which can be wired, wireless, or both. The network can include multiple networks or a network of networks. For example, the network can include one or more wide area networks (WANs), one or more local area networks (LANs), one or more public networks, such as the internet, and / or a public switched telephone network (PSTN), and / or one or more private networks. If the network includes a wireless telecommunications network, components such as a base station, a communications tower, or even access points (as well as other components) can provide wireless connectivity.

[0229] Compatible network environments can include one or more peer-to-peer network environments—in which case no server may be included in a network environment—and one or more client-server network environments—in which case one or more servers may be included in a network environment. In peer-to-peer network environments, functionality described herein with respect to one or more servers can be implemented on any number of client devices.

[0230] In at least one embodiment, a network environment can include one or more cloud-based network environments, a distributed computing environment, a combination thereof, etc. A cloud-based network environment can include a framework layer, a job scheduler, a resource manager, and a distributed file system implemented on one or more servers, which may include one or more core network servers and / or edge servers. A framework layer can include a framework to support software of a software layer and / or one or more application(s) of an application layer. The software or application(s) can each include web-based service software or applications. In embodiments, one or more of the client devices can use the web-based service software or applications (e.g.,by accessing the service software and / or applications via one or more application programming interfaces (APIs). The framework layer can be a type of free and open-source software web application framework, but is not limited to those that can use a distributed file system for large-volume data processing (e.g., "big data").

[0231] A cloud-based network environment can provide cloud computing and / or cloud storage, performing any combination of computing and / or data storage functions as described herein (or one or more parts thereof). Any of these various functions can be distributed across multiple locations of central or core servers (e.g., one or more data centers that may be distributed across a state, region, country, the Earth, etc.). If a connection to a user (e.g., a client device) is relatively close to an edge server(s), a core server may designate at least some of the functionality for that edge server(s). A cloud-based network environment can be private (e.g., restricted to a single organization), public (e.g., available to many organizations), and / or a combination thereof (e.g., a hybrid cloud environment).

[0232] The client device(s) may include at least some of the components, features, and functionality described herein with reference to Fig.The 10 exemplary computing device(s) described include 1000. By way of example, and not limited to, a client device may be a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, personal digital assistant (PDA), MP3 player, virtual reality headset, global positioning system (GPS) or global positioning device, video player, video camera, surveillance device or system, vehicle, boat, flying vehicle, virtual machine, drone, robot, handheld communication device, hospital device, gaming device or system, entertainment system, vehicle computer system, embedded system controller, remote control, device, consumer electronics device, workstation, edge device, any combination of these described devices, or any other suitable device.

[0233] The disclosure can generally be described in the context of computer code or machine-usable instructions, including computer-executable instructions such as program modules that are executed by a computer or other machine, such as a personal data assistant or a handheld device. Generally, program modules, including routines, programs, objects, components, data structures, etc., refer to code that performs specific tasks or implements specific abstract data types. The disclosure can be exercised in various system configurations, including handheld devices, consumer electronics, general-purpose computers, other specialized computing devices, etc. The disclosure can also be exercised in distributed computing environments where tasks are performed by remote processing devices linked via a communication network.

[0234] As used herein, any mention of "and / or" in relation to two or more elements shall be interpreted as referring to only one element or a combination of elements. For example, "element A, element B, and / or element C" may include only element A, only element B, only element C, element A and element B, element A and element C, element B and element C, or elements A, B, and C. Furthermore, "at least one of element A or element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B. Additionally, "at least one of element A and element B" may include at least one of element A, at least one of element B, or at least one of element A and at least one of element B.

[0235] The subject matter of this disclosure is described herein with specificity to satisfy legal requirements. However, the description itself is not intended to limit the scope of this disclosure. Rather, the inventors have provided that the claimed subject matter may also be embodied in other ways to include different steps or combinations of steps similar to those described in this document, in conjunction with other current or future technologies. Furthermore, although the terms "step" and / or "block" may be used herein to denote different elements of methods employed, they should not be interpreted as implying a particular sequence of or between different steps disclosed herein, except where the sequence of individual steps is explicitly described.

[0236] Different embodiments can be described by the following clauses: Clause 1. One or more processors comprising a processing circuit arrangement for: Generating a depth image using a machine learning model based at least on an input of image data comprising at least one stereo image pair; Evaluate the depth image to identify at least one segment with one or more anomalous depth values; Classifying at least one segment in the image space, at least based on the identification of one or more anomalous depth values; and Generate, at least based on the classification of the at least one segment of an updated depth image, to include one or more updated depth values ​​in the at least one segment of the updated depth image. Clause 2. The one or more processors according to Clause 1, wherein the processing circuitry further serves the following purpose: Applying a mask to at least one segment of the updated depth image to edit the one or more anomalous depth values ​​in order to output a masked depth image; and Performing one or more operations based on at least the masked depth image. Clause 3. The one or more processors according to Clause 1 or 2, wherein the processing circuitry further serves the following purpose: Identifying the at least one segment based on a depth value pattern associated with the image space classification of the at least one segment. Clause 4. The one or more processors according to a preceding clause, wherein the processing circuitry further serves the following purpose: Identifying at least one segment based on a lighting-based artifact image space classification of at least one segment. Clause 5. The one or more processors according to a preceding clause, wherein the processing circuitry further serves the following purpose: Identifying the at least one segment based on identifying one or more depth value discontinuities within the at least one segment. Clause 6. The one or more processors according to a preceding clause, wherein the processing circuitry further serves the following purpose: Determining the quality of the depth image based at least on at least one segment of one or more anomalous depth values; and Outputting an accuracy score for the machine learning model, at least based on the determined quality. Clause 7. The one or more processors according to a preceding clause, wherein image space classification includes at least one surface geometry classification. Clause 8. The one or more processors according to a preceding clause, wherein the processing circuitry further serves the following purpose: Identifying the at least one segment based on at least one input from a human-machine interface that includes a specification of the at least one segment such that it contains one or more anomalous depth values. Clause 9. The one or more processors according to a preceding clause, wherein the updated depth image is generated based on at least one or more updated depth values ​​calculated for the one or more anomalous depth values. Clause 10. The one or more processors according to Clause 9, wherein the processing circuitry further serves the following purpose: Calculating one or more interpolated depth values ​​for the one or more anomalous depth values, at least based on a set of depth values ​​selected from one or more segments of the depth image that are not identified as including the one or more anomalous depth values; and where one or more updated depth values ​​are based on one or more interpolated depth values. Clause 11. The one or more processors according to Clause 10, wherein the processing circuitry further serves the following purpose: Applying one or more interpolated depth values ​​to a surface fitting algorithm to calculate the one or more updated depth values. Clause 12. The one or more processors according to Clause 11, wherein the processing circuitry further serves the following purpose: Applying one or more structural constraints to the surface fitting algorithm based on at least one or more contextual classifications determined from a geometry illustrated in the at least one stereo image pair. Clause 13. The one or more processors according to Clause 12, wherein the one or more structural constraints are defined at least on the basis of a generalized architectural model selected at least on the basis of the geometry illustrated in the at least one stereo image pair. Clause 14. The one or more processors according to any one of Clauses 10 to 13, wherein the processing circuitry further serves the following purpose: Outputting a set of data samples that include a data sample comprising at least one stereo image pair and the corrected depth image. Clause 15. The one or more processors according to a preceding clause, wherein the processing circuitry is comprised of at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for conducting digital twinning operations; a system for performing a light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for generating or presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational artificial intelligence operations (AI operations); a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more Visual Language Models (VLMs); a system for performing generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. Clause 16. System comprising one or more processors for: Receiving image data that includes at least one stereo image pair; and Generate, using a machine learning model, a depth image based at least on a stereo image pair, wherein the machine learning model is trained to infer the depth image based at least on a feedback loss determined using at least one ground-truth depth image, wherein the ground-truth depth image is generated based at least on the following: a corrected depth image comprising one or more corrected depth values, wherein the one or more corrected depth values ​​are calculated for one or more anomalous depth values ​​identified from a training depth image determined from a training stereo image pair. Clause 17. System according to Clause 16, wherein the one or more processors further serve the following purpose: Determining the quality of the depth image based at least on identifying at least one segment of the depth image to contain a set of anomalous depth values; and Outputting an accuracy score for the machine learning model, at least based on the determined quality. Clause 18. System according to Clause 16 or 17, wherein the one or more processors further serve the following purpose: Outputting a set of data samples that include a data sample comprising the at least one stereo image pair and the depth image generated from the at least one stereo image pair. Clause 19. System according to one of Clauses 16 to 18, wherein the system includes at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for conducting digital twinning operations; a system for performing a light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for generating or presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational artificial intelligence operations (AI operations); a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more Visual Language Models (VLMs); a system for performing generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. Clause 20. Procedure, which includes the following: Generating a corrected depth image that includes depth information represented by a stereo image pair, at least based on the evaluation of an initial depth image generated by applying the stereo image pair as input to a machine learning model, wherein the evaluation serves the following purpose: Identifying at least one segment of the first depth image that includes one or more anomalous depth values, wherein the at least one segment is identified at least based on an image space classification of the at least one segment.

[0237] It is understood that the aspects and embodiments described above are merely examples and that modifications in detail may be made within the scope of the claims.

[0238] Each device, method and feature disclosed in the description and (if applicable) in the claims and drawings can be provided independently or in any suitable combination.

[0239] Reference numerals appearing in the claims serve only for illustration and are not intended to have any limiting effect on the scope of the claims. QUOTES INCLUDED IN THE DESCRIPTION

[0000] This list of documents cited by the applicant was automatically generated and is included solely for the reader's convenience. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions. Cited patent literature

[0000] US 16 / 101,232

[0135] Cited non-patent literature

[0000] ISO-26262

[0134]

Claims

[1] One or more processors comprising a processing circuit arrangement for: Generating a depth image using a machine learning model based at least on an input of image data comprising at least one stereo image pair; Evaluate the depth image to identify at least one segment with one or more anomalous depth values; Classifying at least one segment in the image space, at least based on the identification of one or more anomalous depth values; and Generate, at least based on the classification of the at least one segment of an updated depth image, to include one or more updated depth values ​​in the at least one segment of the updated depth image. [2] The one or more processors according to claim 1, wherein the processing switching technology further serves the following purpose: Applying a mask to at least one segment of the updated depth image to edit the one or more anomalous depth values ​​in order to output a masked depth image; and Performing one or more operations based on at least the masked depth image. [3] The one or more processors according to claim 1 or 2, wherein the processing switching technology further serves the following purpose: Identifying the at least one segment based on a depth value pattern associated with the image space classification of the at least one segment. [4] The one or more processors according to a preceding claim, wherein the processing switching technology further serves the following purpose: Identifying at least one segment based on a lighting-based artifact image space classification of at least one segment. [5] The one or more processors according to a preceding claim, wherein the processing switching technology further serves the following purpose: Identifying the at least one segment based on identifying one or more depth value discontinuities within the at least one segment. [6] The one or more processors according to a preceding claim, wherein the processing circuitry further serves the following purpose: Determining the quality of the depth image based at least on at least one segment of one or more anomalous depth values; and Outputting an accuracy score for the machine learning model, at least based on the determined quality. [7] The one or more processors according to a preceding claim, wherein image space classification includes at least one surface geometry classification. [8] The one or more processors according to a preceding claim, wherein the processing switching technology further serves the following purpose: Identifying the at least one segment based on at least one input from a human-machine interface that includes a specification of the at least one segment such that it contains one or more anomalous depth values. [9] The one or more processors according to a preceding claim, wherein the updated depth image is generated at least on the basis of one or more updated depth values ​​calculated for the one or more anomalous depth values. [10] The one or more processors according to claim 9, wherein the processing switching technology further serves the following purpose: Calculating one or more interpolated depth values ​​for the one or more anomalous depth values, at least based on a set of depth values ​​selected from one or more segments of the depth image that are not identified as including the one or more anomalous depth values; and where one or more updated depth values ​​are based on one or more interpolated depth values. [11] The one or more processors according to claim 10, wherein the processing switching technology further serves the following purpose: Applying one or more interpolated depth values ​​to a surface fitting algorithm to calculate the one or more updated depth values. [12] The one or more processors according to claim 11, wherein the processing switching technology further serves the following purpose: Applying one or more structural constraints to the surface fitting algorithm, based at least on one or more contextual classifications determined from a geometry illustrated in the at least one stereo image pair. [13] The one or more processors according to claim 12, wherein the one or more structural constraints are defined at least on the basis of a generalized architectural model selected at least on the basis of the geometry illustrated in the at least one stereo image pair. [14] The one or more processors according to any one of claims 10 to 13, wherein the processing switching technology further serves the following purpose: Outputting a set of data samples that include a data sample comprising at least one stereo image pair and the corrected depth image. [15] The one or more processors according to a preceding claim, wherein the processing switching technology comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for conducting digital twinning operations; a system for performing a light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for generating or presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational artificial intelligence operations (AI operations); a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more Visual Language Models (VLMs); a system for performing generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [16] System comprising one or more processors for: Receiving image data that includes at least one stereo image pair; and Generate, using a machine learning model, a depth image based at least on a stereo image pair, wherein the machine learning model is trained to infer the depth image based at least on a feedback loss determined using at least one ground-truth depth image, wherein the ground-truth depth image is generated based at least on the following: a corrected depth image comprising one or more corrected depth values, wherein the one or more corrected depth values ​​are calculated for one or more anomalous depth values ​​identified from a training depth image determined from a training stereo image pair. [17] System according to claim 16, wherein the one or more processors further serve the following: Determining the quality of the depth image based at least on identifying at least one segment of the depth image to contain a set of anomalous depth values; and Outputting an accuracy score for the machine learning model, at least based on the determined quality. [18] System according to claim 16 or 17, wherein one or more processors further serve the following: Outputting a set of data samples that include a data sample comprising the at least one stereo image pair and the depth image generated from the at least one stereo image pair. [19] System according to any one of claims 16 to 18, wherein the system comprises at least one of the following: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for conducting digital twinning operations; a system for performing a light transport simulation; a system for conducting collaborative content creation for 3D assets; a system for generating or presenting at least one type of virtual reality content, augmented reality content, or mixed reality content; a system for performing deep learning operations; a system implemented using an edge device; a system that is implemented using a robot; a system for performing conversational artificial intelligence operations (AI operations); a system that implements one or more language models; a system that implements one or more large language models (LLMs); a system that implements one or more Visual Language Models (VLMs); a system for performing generative AI operations; a system for generating synthetic data; a system that includes one or more virtual machines (VMs); a system that is at least partially implemented in a data center; or a system that is implemented at least partially using cloud computing resources. [20] Method comprising the following: Generating a corrected depth image that includes depth information represented by a stereo image pair, at least based on the evaluation of an initial depth image generated by applying the stereo image pair as input to a machine learning model, wherein the evaluation serves the following purpose: Identifying at least one segment of the first depth image that includes one or more anomalous depth values, wherein the at least one segment is identified at least based on an image space classification of the at least one segment.

Images