Probabilistic three-dimensional (3D) scene parsing

Abstract

An example system includes memory for storing sensor data. The sensor data includes data from camera sensors. Each camera sensor has a respective field of view. The system includes one or more processors configured to obtain the sensor data. The one or more processors are configured to determine an object in the sensor data using three-dimensional (3D) object detection. The one or more processors are configured to determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty including an aleatoric uncertainty. The one or more processors are configured to adjust model parameters of one or more object detection models to reduce the uncertainty over time, generating one or more adjusted object detection models. The one or more processors are configured to navigate a vehicle based on output from the one or more adjusted object detection models.

Description

Qualcomm Ref. No.2401738WO 1 PROBABILISTIC THREE-DIMENSIONAL (3D) SCENE PARSING

[0001] This application claims priority to India Patent Application No. 202441042550, filed May 31, 2024, the entire contents of which is incorporated herein by reference. TECHNICAL FIELD

[0002] This disclosure relates to systems, including systems autonomous driving vehicles and / or advanced driver-assistance systems (ADAS). BACKGROUND

[0003] An autonomous driving vehicle is a vehicle that is configured to sense the environment around the vehicle, such as the existence and location of other objects, and operate without, or with limited, human control. An autonomous driving vehicle may include a Light Detection and Ranging (LiDAR) system, a camera system, and / or other sensor system for sensing data indicative of the existence and location of other objects around the autonomous driving vehicle. In some examples, such an autonomous driving vehicle may be referred to as an ego vehicle. A vehicle having an advanced driver- assistance systems (ADAS) is a vehicle that includes systems which may assist a driver in operating the vehicle, such as parking or driving the vehicle.

[0004] Some autonomous driving vehicles may include a bird’s-eye-view (BEV) representation of sensor data which may be displayed or otherwise used in navigating the vehicle. A BEV representation of objects in the vicinity of an autonomous driving vehicle may include an overhead representation of such objects. SUMMARY

[0005] The present disclosure generally relates to techniques and devices for processing multiple sensor system data of a device (e.g., vehicle, robot, virtual reality (VR) device, etc.) to compress and fuse bird’s-eye-view (BEV) features, while modeling uncertainty to improve the accuracy of predictions (e.g., object detection) and decision-making based on such predictions. While the techniques of this disclosure are primarily discussed with respect to a vehicle, it should be understood that these techniques are applicable for use with other devices, such as robots, VR devices, or other devices where more accurate perception of space and things within the space may be desirable. 1616-438WO01Qualcomm Ref. No.2401738WO 2

[0006] Existing three-dimensional (3D) object detection techniques include using BEV (e.g., overhead) representations from multiple camera sensor and / or LiDAR sensors. Sensor detections from individual camera sensor views can be noisy and ambiguous for various reasons such as occlusion, weather conditions (e.g., fog, heavy precipitation, etc.), varying lighting conditions, object poses, and / or the like, introducing uncertainty to the detections. Current techniques may not properly account for such uncertainty.

[0007] As such, the techniques of this disclosure include a probabilistic framework for 3D object detection that explicitly models different sources of uncertainty. In some examples, this uncertainty includes aleatoric uncertainty due to sensor noise. In some examples, this uncertainty includes epistemic uncertainty in the model parameters themselves. By estimating uncertainty from multiple detections of the same object, a system may better determine a reliability of each detection.

[0008] The techniques of this disclosure may provide numerous benefits over existing techniques. For example, the techniques of this disclosure may provide for more reliable detections by intelligently weighting camera sensor views based on estimated uncertainty. Such a weighting may lead to improved accuracy over single-view or naive fusion approaches. The techniques of this disclosure may provide an ability to quantify confidence and identify ambiguous or occluded regions in an environment requiring further sensing. This may provide potentially valuable information for decision making, such as whether to immediately take an action or to delay taking an action while additional sensor data is gathered to improve confidence and / or reduce ambiguity associated with such regions.

[0009] The techniques of this disclosure may propagate uncertainty through the entire pipeline from raw sensor data to 3D object beliefs over time, thereby capturing effects of noise at every stage of the pipeline. The techniques of this disclosure may learn informative sensing actions through reinforcement learning to actively reduce uncertainty and optimize long-term perception, instead of optimizing single independent snapshots of data.

[0010] The techniques of this disclosure may handle missing and ambiguous detections via probabilistic state representation and may be more robust to real-world sensing challenges than existing techniques. The techniques of this disclosure may provide a flexible framework that can incorporate different modalities and fuse on various representations like BEV, 3D points, and / or the like. 1616-438WO01Qualcomm Ref. No.2401738WO 3

[0011] The techniques of this disclosure may provide calibrated estimates of aleatoric and / or epistemic uncertainty to diagnose model and data limitations. The techniques of this disclosure may enable uncertainty-aware applications, such as autonomous driving applications, that require or desire reliability guarantees in uncertain environments

[0012] In one example, a system includes: one or more memories for storing sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; and one or more processors in communication with the one or more memories, the one or more processors configured to: obtain the sensor data; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models.

[0013] In another example, a method includes: obtaining sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determining an object in the sensor data using one or more three-dimensional (3D) object detection models; determining an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjusting model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigating a vehicle based on output from the one or more adjusted 3D object detection models.

[0014] In another example, computer-readable media stores instructions that, when executed by one or more processors, causes the one or more processors to: obtain sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjust model parameters of the one or more 3D object detection models to reduce the 1616-438WO01Qualcomm Ref. No.2401738WO 4 uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models.

[0015] The details of one or more examples are set forth in the accompanying drawings and the description below. Other features, objects, and advantages will be apparent from the description, drawings, and claims. BRIEF DESCRIPTION OF DRAWINGS

[0016] FIG.1 is a block diagram illustrating an example processing system according to one or more aspects of this disclosure.

[0017] FIG.2 is a block diagram of an example BEV fusion architecture.

[0018] FIG.3 is a block diagram of another example BEV fusion architecture according to one or more aspects of this disclosure.

[0019] FIG. 4 is a flow diagram illustrating example probabilistic 3D scene parsing according to one or more aspects of this disclosure. DETAILED DESCRIPTION

[0020] Multiple sensor systems, such as camera and Light Detection and Ranging (LiDAR) systems, may be used together in various different robotic, vehicular, and virtual reality (VR) applications. One such vehicular application is an advanced driver assistance system (ADAS). ADAS is a system that utilizes multiple sensor systems, such as camera and LiDAR sensor systems, to improve driving safety, comfort, and overall vehicle performance. Such a system combines the strengths of both sensors to provide a more comprehensive view of a vehicle’s surroundings, enabling the ADAS to better assist the driver in various driving scenarios.

[0021] In some examples, a camera-based system is responsible for capturing high- resolution images and processing them in real time. The output images of such a camera- based system may be used in applications such as depth estimation, object detection, and / or pose detection, including the detection and recognition of objects, such as other vehicles, pedestrians, traffic signs, and lane markings. Cameras may be particularly good at capturing color and texture information, which is useful for accurate object recognition and classification. 1616-438WO01Qualcomm Ref. No.2401738WO 5

[0022] LiDAR sensors emit laser pulses to measure the distance, shape, and relative speed of objects around the vehicle. LiDAR sensors provide three-dimensional (3D) data, enabling the ADAS to create a detailed map of the surrounding environment. LiDAR may be particularly effective in low-light or adverse weather conditions, where camera performance may be hindered. In some examples, the output of a LiDAR sensor may be used as partial ground truth data for determining neural network-based depth information on corresponding camera images.

[0023] By fusing the data gathered from both camera and LiDAR sensors, an ADAS or another kind of system, can deliver enhanced situational awareness and improved decision-making capabilities. This enables various driver assistance features such as adaptive cruise control, lane keeping assist, pedestrian detection, automatic emergency braking, and parking assistance. The combined system can also contribute to the development of semi-autonomous and fully autonomous driving technologies, which may lead to a safer and more efficient driving experience.

[0024] However, sensor detections from individual camera sensor views can be noisy and ambiguous for various reasons such as object occlusion, weather conditions (e.g., fog, heavy precipitation, etc.), varying lighting conditions, object poses, sensor cleanliness, and / or the like, introducing uncertainty to the detections. Current techniques may not properly account for such uncertainty. Without modeling this uncertainty, it is difficult to reliably fuse information across different camera sensor views and make robust decisions based on the fused information.

[0025] Existing 3D object detection techniques include using BEV representations from multiple cameras and / or LiDAR sensors. Such techniques may not properly account for the noise and ambiguities inherent in real-world multi-sensor perception. The existing techniques may assume idealized conditions and treat observations from different camera sensor views as independent. However, there may be underlying dependencies and uncertainties across different perspectives and sensing modalities, which the existing techniques may fail to model. As a result, existing techniques lack robustness to real- world vision and LiDAR noise, occlusion, sensor failure, and / or other issues. Therefore, existing techniques may be unable to provide reliable measures of uncertainty to reduce ambiguity. As such, existing techniques may be relatively unsafe for critical applications, like autonomous driving.

[0026] The techniques of this disclosure may include efficient techniques to propagate uncertainty through a 3D object detection pipeline, including camera projection, multi- 1616-438WO01Qualcomm Ref. No.2401738WO 6 view fusion, and data association over time. The techniques may allow active querying of camera sensor views to reduce the most significant uncertainties, such as in occluded or ambiguous regions. By formally modeling uncertainty, the techniques of this disclosure may improve the reliability of 3D scene understanding. This improved reliability may enable more informed decision making for applications, such as autonomous navigation, that require robust perception even in the presence of noise and ambiguity from real-world sensing.

[0027] In the context of machine learning and statistical modeling, there are generally two types of uncertainty: aleatoric uncertainty and epistemic uncertainty. Aleatoric uncertainty is a type of uncertainty that arises from the inherent randomness and variability in the data itself. Aleatoric uncertainty is often referred to as "noise" and is generally not reducible, even with more data or a better model. Examples of sources of aleatoric uncertainty in BEV perception models include sensor noise, object occlusion, and imperfect annotations.

[0028] Epistemic uncertainty is a type of uncertainty that arises from a lack of knowledge or information. Epistemic uncertainty can be reduced with more data or by improving the model. Examples of sources of epistemic uncertainty in BEV perception models include model architecture, hyperparameters, and limited training data.

[0029] In summary, aleatoric uncertainty represents the irreducible noise and variability in the data, while epistemic uncertainty represents the uncertainty that can be reduced by improving the model or by acquiring more data. Understanding and modeling these different forms of uncertainty can help improve the robustness, reliability, and performance of BEV perception models.

[0030] According to the techniques of this disclosure, a BEV system may model aleatoric uncertainty of sensor data from multiple sensors across a plurality of passes. In some examples, the BEV system may model epistemic uncertainty. The BEV system may use determined uncertainties over time to adjust model parameters to reduce uncertainty and improve a probabilistic belief state associated the 3D environment surrounding the BEV system, which may improve the safety of navigation within the 3D environment. The techniques of this disclosure allow for different combinations of sensors for use in various sensor setups.

[0031] FIG.1 is a block diagram illustrating an example processing system according to one or more aspects of this disclosure. Processing system 100 may be used in a vehicle, such as an autonomous driving vehicle or an assisted driving vehicle (e.g., a vehicle 1616-438WO01Qualcomm Ref. No.2401738WO 7 having an ADAS or an “ego vehicle”). In such an example, processing system 100 may represent an ADAS. In other examples, processing system 100 may be used in robotic applications, VR applications, or other kinds of applications that may include a plurality of sensor systems, such as one or more camera sensors and / or a LiDAR system. The techniques of this disclosure are not limited to any specific sensor setup or to vehicular applications. The techniques of this disclosure may be applied by any system that processes data from a plurality of sensors.

[0032] Processing system 100 may include LiDAR system 102, camera(s) 104, controller 106, one or more sensor(s) 108, input / output device(s) 120, wireless connectivity component 130, and / or memory 160. LiDAR system 102 may include one or more light emitters (e.g., lasers) and one or more light sensors. LiDAR system 102 may, in some cases, be deployed in or about a vehicle. For example, LiDAR system 102 may be mounted on a roof of a vehicle, in bumpers of a vehicle, and / or in other locations of a vehicle. LiDAR system 102 may be configured to emit light pulses and sense the light pulses reflected off of objects in the environment. LiDAR system 102 is not limited to being deployed in or about a vehicle. LiDAR system 102 may be deployed in or about another kind of object.

[0033] In some examples, the one or more light emitters of LiDAR system 102 may emit such pulses in a 360-degree field around the vehicle so as to detect objects within the 360- degree field by detecting reflected pulses using the one or more light sensors. For example, LiDAR system 102 may detect objects in front of, behind, or beside LiDAR system 102. While described herein as including LiDAR system 102, it should be understood that another distance or depth sensing system may be used in place of LiDAR system 102. The output of LiDAR system 102 are called point clouds or point cloud frames.

[0034] A point cloud frame output by LiDAR system 102 is a collection of 3D data points that represent the surface of objects in the environment. LiDAR processing circuitry of LiDAR system 102 may generate one or more point cloud frames based on the one or more optical signals emitted by the one or more light emitters of LiDAR system 102 and the one or more reflected optical signals sensed by the one or more light sensors of LiDAR system 102. These points are generated by measuring the time it takes for a laser pulse to travel from a light emitter to an object and back to a light detector. Each point in the cloud has at least three attributes: x, y, and z coordinates, which represent its position in a 1616-438WO01Qualcomm Ref. No.2401738WO 8 Cartesian coordinate system. Some LiDAR systems also provide additional information for each point, such as intensity, color, and classification.

[0035] Intensity (also called reflectance) is a measure of the strength of the returned laser pulse signal for each point. The value of the intensity attribute depends on various factors, such as the reflectivity of the object’s surface, distance from the sensor, and the angle of incidence. Intensity values can be used for several purposes, including distinguishing different materials, and enhancing visualization: Intensity values can be used to generate a grayscale image of the point cloud, helping to highlight the structure and features in the data.

[0036] Color information in a point cloud is usually obtained from other sources, such as digital cameras mounted on the same platform as the LiDAR sensor, and then combined with the LiDAR data. Cameras used to capture color information for point cloud data may, in some examples, be separate from camera(s) 104. The color attribute includes color values (e.g., red, green, and blue (RGB)) values for each point. The color values may be used to improve visualization and aid in enhanced classification (e.g., the color information can aid in the classification of objects and features in the scene, such as vegetation, buildings, and roads.)

[0037] Classification is the process of assigning each point in the point cloud to a category or class based on its characteristics or its relation to other points. The classification attribute may be an integer value that represents the class of each point, such as ground, vegetation, building, water, etc. Classification can be performed using various algorithms, often relying on machine learning techniques or rule-based approaches.

[0038] Camera(s) 104 may include any type of camera configured to capture video or image data in the environment around processing system 100 (e.g., around a vehicle). In some examples, processing system 100 may include multiple camera(s) 104. For example, camera(s) 104 may include a front facing camera (e.g., a front bumper camera, a front windshield camera, and / or a dashcam), a rear facing camera (e.g., a backup camera), side facing cameras (e.g., cameras mounted in sideview mirrors). Camera(s) 104 may include a color camera or a grayscale camera. In some examples, camera(s) 104 may be a camera system including more than one camera sensor. Sensor(s) 108 may include radar sensor, a location sensor, a sonar sensor, an infrared camera, and / or a time-of-flight (ToF) camera.

[0039] LiDAR system 102 may, in some examples, be configured to collect 3D point cloud frames 166. Camera(s) 104 may, in some examples, be configured to collect 2D 1616-438WO01Qualcomm Ref. No.2401738WO 9 camera images 168. An importance of data input modalities such as 3D point cloud frames 166 and 2D camera images 168 may vary for indicating one or more characteristics of objects in a 3D environment. For example, when color and texture are important characteristics of a first object and when color and texture are not important characteristics of a second object, 2D camera images 168 may be more important for identifying characteristics of the first object as compared with the importance of 3D point cloud frames 166 for identifying characteristics of the second object. It may be beneficial to consider the importance of 3D point cloud frames 166 and 2D camera images 168 for indicating characteristics of a 3D environment when generating BEV features corresponding to 3D point cloud frames 166 and / or generating BEV features corresponding to 2D camera images 168.

[0040] Wireless connectivity component 130 may include subcomponents, for example, for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth generation (5G) connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards. Wireless connectivity component 130 is further connected to one or more antennas 135. Processing system 100 may communicate with external processing system and / or processing systems of other devices (e.g., other vehicles) via wireless connectivity component 130.

[0041] Processing system 100 may also include one or more input / output devices 120, such as screens, touch-sensitive surfaces (including touch-sensitive displays), physical buttons, speakers, microphones, and the like. Input / output device(s) 120 (e.g., which may include an I / O controller) may manage input and output signals for processing system 100. In some cases, input / output device(s) 120 may represent a physical connection or port to an external peripheral. In some cases, input / output device(s) 120 may utilize an operating system. In other cases, input / output device(s) 120 may represent or interact with a modem, a keyboard, a mouse, a touchscreen, or a similar device. In some cases, input / output device(s) 120 may be implemented as part of a processor (e.g., a processor of processing circuitry 110). In some cases, a user may interact with a device via input / output device(s) 120 or via hardware components controlled by input / output device(s) 120.

[0042] Controller 106 may be an autonomous or assisted driving controller (e.g., an ADAS) configured to control operation of processing system 100 (e.g., including the operation of a vehicle). For example, controller 106 may control acceleration, braking, 1616-438WO01Qualcomm Ref. No.2401738WO 10 and / or navigation of vehicle through the environment surrounding vehicle. Controller 106 may include one or more processors, e.g., processing circuitry 110. Controller 106 is not limited to controlling vehicles. Controller 106 may additionally or alternatively control any kind of controllable device, such as a robotic component. Processing circuitry 110 may include one or more central processing units (CPUs), such as single-core or multi- core CPUs, graphics processing units (GPUs), digital signal processor (DSPs), application specific integrated circuits (ASICs), field programmable gate arrays (FPGAs), neural processing unit (NPUs), multimedia processing units, and / or the like. Instructions applied by processing circuitry 110 may be loaded, for example, from memory 160 and may cause processing circuitry 110 to perform the operations attributed to processor(s) in this disclosure.

[0043] An NPU is generally a specialized circuit configured for implementing control and arithmetic logic for executing machine learning algorithms, such as algorithms for processing artificial neural networks (ANNs), deep neural networks (DNNs), random forests (RFs), kernel methods, and the like. An NPU may sometimes alternatively be referred to as a neural signal processor (NSP), a tensor processing unit (TPU), a neural network processor (NNP), an intelligence processing unit (IPU), or a vision processing unit (VPU).

[0044] Processing circuitry 110 may also include one or more sensor processing units associated with LiDAR system 102, camera(s) 104, and / or sensor(s) 108. For example, processing circuitry 110 may include one or more image signal processors associated with camera(s) 104 and / or sensor(s) 108, and / or a navigation processor associated with sensor(s) 108, which may include satellite-based positioning system components (e.g., Global Positioning System (GPS) or Global Navigation Satellite System (GLONASS)) as well as inertial positioning system components. Sensor(s) 108 may include direct depth sensing sensors, which may function to determine a depth of or distance to objects within the environment surrounding processing system 100 (e.g., surrounding a vehicle).

[0045] Processing system 100 also includes memory 160, which is representative of one or more static and / or dynamic memories, such as a dynamic random-access memory, a flash-based static memory, and the like. In this example, memory 160 includes computer- executable components, which may be applied by one or more of the aforementioned components of processing system 100.

[0046] Examples of memory 160 include random access memory (RAM), read-only memory (ROM), electrically erasable programmable ROM (EEPROM), compact disk 1616-438WO01Qualcomm Ref. No.2401738WO 11 ROM (CD-ROM), and / or another kind of hard disk. Examples of memory 160 include solid state memory and / or a hard disk drive. In some examples, memory 160 is used to store computer-readable, computer-executable software including instructions that, when applied, cause one or more processors to perform various functions described herein. In some cases, memory 160 contains, among other things, a basic input / output system (BIOS) which controls basic hardware or software operation such as the interaction with peripheral components or devices. In some cases, a memory controller operates memory cells of memory 160. For example, the memory controller can include a row decoder, column decoder, or both. In some cases, memory cells within memory 160 store information in the form of a logical state.

[0047] Processing system 100 may be configured to perform techniques for extracting features from image data and position data, processing the features, fusing the features, or any combination thereof. In some examples, processing system 100 may perform a portion of such techniques while external processing system 180 may perform another portion of such techniques.

[0048] Processing circuitry 110 may include BEV unit 140. BEV unit 140 may be implemented in software, firmware, and / or any combination of hardware described herein. As will be described in more detail below, BEV unit 140 may be configured to receive a plurality of 2D camera images 168 captured by camera(s) 104 and receive a plurality of 3D point cloud frames 166 captured by LiDAR system 102. BEV unit 140 may be configured to receive 2D camera images 168 and 3D point cloud frames 166 directly from camera(s) 104 and LiDAR system 102, respectively, or from memory 160. In some examples, the plurality of 3D point cloud frames 166 may be referred to herein as “position data.” In some examples, the plurality of 2D camera images 168 may be referred to herein as “image data.”

[0049] BEV unit 140 may fuse features corresponding to the plurality of 3D point cloud frames 166 and features corresponding to the plurality of 2D camera images 168 in order to combine image data corresponding to one or more objects within a 3D space with position data corresponding to the one or more objects. For example, each camera image of the plurality of 2D camera images 168 may comprise a 2D array of pixels that includes image data corresponding to one or more objects. Each point cloud frame of the plurality of 3D point cloud frames 166 may include a 3D multi-dimensional array of points corresponding to the one or more objects. Since the one or more objects are located in the same 3D space where processing system 100 is located, it may be beneficial to fuse 1616-438WO01Qualcomm Ref. No.2401738WO 12 features of the image data present in 2D camera images 168 that indicate information corresponding to the identity one or more objects with features of the position data present in the 3D point cloud frames 166 that indicate a location of the one or more objects within the 3D space. This is because image data may include at least some information that position data does not include, and position data may include at least some information that image data does not include.

[0050] Fusing features of image data and features of position data may provide a more comprehensive view of a 3D environment corresponding to processing system 100 as compared with analyzing features of image data and features of position data separately. For example, the plurality of 3D point cloud frames 166 may indicate an object in front of a processing system 100, and BEV unit 140 may be able to process the plurality of 3D point cloud frames 166 to determine that the object is a stoplight. This is because the plurality of 3D point cloud frames 166 may indicate that the object includes three round components oriented vertically and / or horizontally relative to a surface of a road intersection, and the plurality of 3D point cloud frames 166 may indicate that the size of the object is within a range of sizes that stoplights normally occupy. But the plurality of 3D point cloud frames 166 might not include information that indicates which of the three lights of the stoplight is turned on and which of the three lights of the stoplight is turned off. 2D camera images 168 may include image data indicating that a green light of the stoplight is turned on, for example. This means that it may be beneficial to fuse features of image data with features of position data so that BEV unit 140 can analyze image data and position data to determine characteristics of one or more objects within the 3D environment.

[0051] Fusing image data BEV features and position data BEV features may involve associating image data BEV features with position data BEV features corresponding to the image data BEV features. For example, processing system 100 may fuse image data BEV features indicating a color and an identity of a stoplight with position data BEV features indicating a position of the stoplight. This means that the fused set of BEV features may include information from both image data and position data corresponding to the stoplight that is important for generating an output. Some systems may fuse image data BEV features with position data BEV features by generating “grids” of image data BEV features with position data BEV features and fusing the grids. A BEV feature grid may correspond to a 2D BEV of a 3D environment. Each “cell” of the BEV feature grid may include features corresponding to a portion of the 3D environment corresponding to 1616-438WO01Qualcomm Ref. No.2401738WO 13 the cell. This allows the system to fuse image data BEV features with position data BEV features corresponding to the same portion of the 3D environment.

[0052] For example, BEV unit 140 may obtain sensor data. The sensor data may include sensor data from a plurality of camera sensors (camera(s) 104). Each camera sensor may have a respective field of view. BEV unit 140 may determine an object in the sensor data using 3D object detection, such as through fusing camera sensor data and LiDAR sensor data. BEV unit 140 may determine an aleatoric uncertainty associated with a detected object in the fused sensor data based on a plurality of passes of the sensor data at a particular instant in time. BEV unit 140 may adjust model parameters of one or more models to reduce uncertainty over time. Controller 106 may navigate a vehicle based on the adjusted model parameters. In some examples, BEV unit 140 may also determine an epistemic uncertainty associated with the object. For examples, BEV unit 140 may use the aleatoric uncertainty and the epistemic uncertainty to improve the determination of free space around one or more objects in the sensor data and thereby improve the safety of navigation in an area surrounding the vehicle.

[0053] Control unit 142 may control the device based on information included in the fused BEV representations relating to one or more objects within a 3D space including processing system 100. For example, the fused BEV representations and / or the output of BEV unit 140 may include an identity of one or more objects, a position of one or more objects relative to the processing system 100, characteristics of movement (e.g., speed, acceleration) of one or more objects, or any combination thereof. Based on this information, control unit 142 may control the device corresponding to processing system 100. The fused BEV representations may be stored in memory 160 as model output 172.

[0054] External processing system 180 may represent one or more servers in a cloud computing environment and / or a roadside unit. External processing system 180 may obtain (e.g., receive) data from processing system 100 and / or similar processing systems and process the received data, for example, to determine uncertainty associated with a detected object and / or to train one or more encoders and / or decoders of BEV unit 140.

[0055] External processing system 180 may include processing circuitry 190, which may be any of the types of processors described above for processing circuitry 110. Processing circuitry 190 may include a BEV unit 194 configured to determine uncertainty associated with a detected object. Processing circuitry 190 may obtain data from controller 106 or from memory 160. External processing system 180 may also include memory 198 that may be configured to store data obtained from processing system 100 and other similar 1616-438WO01Qualcomm Ref. No.2401738WO 14 processing systems. Memory 198 may also be configured to store training data (similar to training data 170) and model output (similar to model output 172) for encoders, decoders, or other models that are part of BEV unit 194. Memory 198 may include any of the types of memory described above for memory 160.

[0056] Wireless connectivity component 182 may facilitate communication between external processing system 180 and processing system 100. Wireless connectivity component 182 may include subcomponents, for example, for third generation (3G) connectivity, fourth generation (4G) connectivity (e.g., 4G Long Term Evolution (LTE)), fifth generation (5G) connectivity (e.g., 5G or New Radio (NR)), Wi-Fi connectivity, Bluetooth connectivity, and other wireless data transmission standards.

[0057] In some examples, processing circuitry 110 may be configured to train one or more encoders, decoders, or any combination thereof applied by BEV unit 140 using training data 170. For example, training data 170 may include sensor data such as one or more training point cloud frames and / or one or more camera images. Training data 170 may additionally or alternatively include features known to accurately represent one or more point cloud frames and / or features known to accurately represent one or more camera images. This may allow processing circuitry 110 to train one or more encoders to generate features that accurately represent point cloud frames and train one or more encoders to generate features that accurately represent camera images. Processing circuitry 110 may also use training data 170 to train one or more decoders. In some examples, training data 170 may be stored separately from processing system 100. In some examples, processing circuitry other than processing circuitry 110 and / or processing circuitry 190 and separate from processing system 100 may train one or more encoders, decoders, or any combination thereof applied by BEV unit 140 using training data 170.

[0058] In some examples, processing circuitry 110 may train one or more encoders and / or decoders of BEV unit 140 based on determined aleatoric uncertainty and / or determined epistemic uncertainty. For example, processing circuitry 110 may adjust model parameters of one or more encoder(s) and / or decoder(s) to reduce uncertainty over time.

[0059] In some examples, processing circuitry 190 may be configured to train one or more encoders, decoders, or any combination thereof applied by BEV unit 194 using training data, such as training data 170. For example, training data 170 may include one or more training point cloud frames and / or one or more camera images. Training data 170 may additionally or alternatively include features known to accurately represent one or more point cloud frames and / or features known to accurately represent one or more 1616-438WO01Qualcomm Ref. No.2401738WO 15 camera images. This may allow processing circuitry 190 to train one or more encoders to generate features that accurately represent point cloud frames and train one or more encoders to generate features that accurately represent camera images. In some examples, processing circuitry 190 may train one or more encoders and / or decoders of BEV unit 140 based on determined aleatoric uncertainty and / or determined epistemic uncertainty. For example, processing circuitry 190 may adjust model parameters of one or more encoder(s) and / or decoder(s) to reduce uncertainty over time.

[0060] In some examples, training data 170 may be stored in memory 198. In some examples, external processing system 180 may update BEV unit 140 via wireless connectivity component 182 and wireless connectivity component 130 based on the trained encoder(s) and / or decoder(s) of BEV unit 194.

[0061] FIG.2 is a block diagram of an example BEV fusion architecture. For example, BEV units 140 (and / or BEV unit 194) may include architecture 200 which may be used to generate a BEV grid. BEV unit 140 may obtain point cloud data 202 from, e.g., LiDAR system 102 (FIG.1). Encoder 206 may encode point cloud data 202 to extract 3D sparse features 210. BEV unit 140 may flatten projection unit 214 for example, from 3D to 2D. BEV unit 140 may obtain therefrom LiDAR BEV features 218.

[0062] BEV unit 140 may also obtain image data 204 from, e.g., camera(s) 104 (FIG.1). Encoder 208 may encode image data 204 to obtain perspective view features 212. BEV unit 140 may perform a perspective view-to BEV projection unit 216 to obtain camera BEV features 220.

[0063] Feature fusion unit 219 may fuse LiDAR BEV features 218 and camera BEV features 220. The output of feature fusion unit 219 may be decoded by decoder 222 to generate fused features 226.

[0064] Encoders 206 and 208 may be configured to extract information from input data and process the extracted information to generate an output. In general, encoders are configured to receive data as an input and extract one or more features from the input data. The features are the output from the encoder. The features may include one or more vectors of numerical data that can be processed by a machine learning model. These vectors of numerical data may represent the input data in a way that provides information concerning characteristics of the input data. In other words, encoders are configured to process input data to identify characteristics of the input data.

[0065] Encoders 206 and 208 may represent encoders of neural networks such as a convolutional neural network (CNN), another kind of artificial neural network (ANN), or 1616-438WO01Qualcomm Ref. No.2401738WO 16 another kind of model that includes one or more layers and / or nodes that is configured to extract information from input data and process the extracted information to generate an output. Examples of layers include input layers, output layers, and hidden layers between the input layers and the output layers. A CNN, for example, may include one or more convolutional layers comprising convolutional filters. Each convolutional filter may perform one or more mathematical operations on the input data to detect one or more features such as edges, shapes, textures, or objects. CNNs may additionally or alternatively include activation functions that identify complex relationships between elements of a point cloud data 202 and / or image data 204 and pooling layers that recognize patterns regardless of location within point cloud data 202 and / or image data 304.

[0066] Architecture 200 may include one or more encoders (e.g., encoder 206, encoder 208, flatten projection unit 214, perspective view (PV)-to-BEV projection unit 216) and one or more decoders (e.g., decoder 222). Architecture 200 may be configured to process image data 204 and position data (e.g., point cloud data 202). An encoder-decoder architecture for image feature extraction can be used in computer vision tasks, such as image captioning, image-to-image translation, and image generation. The encoder- decoder architecture may transform input data into a compact and meaningful representation known as a feature vector that captures salient visual information from the input data. The encoder may extract features from the input data, while the decoder reconstructs the input data from the learned features.

[0067] In some cases, an encoder (e.g., encoder 206, encoder 208, flatten projection unit 214, PV-to-BEV projection unit 216) is built using CNN layers to analyze input data in a hierarchical manner. The CNN layers may apply filters to capture local patterns and gradually combine them to form higher-level features. Each convolutional layer extracts increasingly complex visual representations from the input data. These representations may be compressed and down sampled through operations such as pooling or strided convolutions, reducing spatial dimensions while preserving desired information. The final output of the encoder may represent a flattened feature vector that encodes the input data’s high-level visual features.

[0068] A decoder (e.g., decoder 222) may be built using transposed convolutional layers or fully connected layers, may reconstruct the input data from the learned feature representation. A decoder may take the feature vector obtained from the encoder as input and processes the feature vector to generate an output that is similar to the input data. The 1616-438WO01Qualcomm Ref. No.2401738WO 17 decoder may up-sample and expand the feature vector, gradually recovering spatial dimensions lost during encoding. A decoder may apply transformations, such as transposed convolutions or deconvolutions, to reconstruct the input data. The decoder layers progressively refine the output, incorporating details and structure until a visually plausible image is generated.

[0069] During training, an encoder-decoder architecture for feature extraction is trained using a loss function that measures the discrepancy between the reconstructed image and the ground truth image. This loss guides the learning process, encouraging the encoder to capture meaningful features and the decoder to produce accurate reconstructions. The training process may involve minimizing the difference between the generated image and the ground truth image, typically using backpropagation and gradient descent techniques. Encoders and decoders of architecture 200 may be trained using training data 170 stored by the memory 160 of FIG. 1. Additionally, or alternatively, encoders and decoders of architecture 200 may be trained using training data stored separately from memory 160.

[0070] As illustrated, encoders of architecture 200 may extract high-level features from the input data and decoders of architecture 200 may reconstruct the input data from the learned features. This architecture may allow for the transformation of input data into compact and meaningful representations. The encoder-decoder architecture may enable the model to learn and utilize important visual and positional features, facilitating tasks like image generation, captioning, and translation.

[0071] Encoder 206 may encode point cloud data 202 to extract 3D sparse features 210. Flatten projection unit 214 may represent an encoder that is configured to perform one or more tasks involving one or more spatial transformations, such as to flatten a projection of 3D sparse features 210. For example, flatten projection unit 214 may include a spatial transformer network (STN) including localization and sampling generation modules that spatially transform input data. An STN, such as flatten projection unit 214, may perform geometric transformations on 3D sparse features 210 to improve an alignment of 3D sparse features 210 and / or or to adapt 3D sparse features 210 for specific tasks. Flatten projection unit 214 may output LiDAR BEV features 218.

[0072] Encoder 208 may encode image data 204 to extract perspective view features 212. PV-to-BEV projection unit 216 may represent an encoder that is configured to perform one or more tasks involving one or more spatial transformations, such as to project the perspective view features into a BEV format and may output camera BEV features 220. For example, PV-to-BEV projection unit 216 may include an STN including localization 1616-438WO01Qualcomm Ref. No.2401738WO 18 and sampling generation modules that spatially transform input data. An STN, such as PV-to-BEV projection unit 216, may perform geometric transformations on perspective view features 212 to improve an alignment of perspective view features 212 and / or or to adapt perspective view features for specific tasks.

[0073] Feature fusion unit 219 may be configured to fuse LiDAR BEV features 218 and camera BEV features 220 to generate a fused set of 3D features. In some examples, feature fusion unit 219 may use a concatenation operation to fuse LiDAR BEV features 218 and camera BEV features 220 to generate the set of 3D features so that the fused set of 3D features includes useful information present in each of fuse LiDAR BEV features 218 and camera BEV features 220.

[0074] FIG. 3 is a block diagram of another example of a BEV fusion architecture according to one or more aspects of this disclosure. For example, BEV unit 140 (and / or BEV unit 194) may include system 300 which may be used to generate a BEV grid. BEV unit 140 may obtain point cloud data 302 (which may be an example of point cloud data 202) from, e.g., LiDAR system 102 (FIG.1). Encoder 306 may encode point cloud data 302 to extract 3D sparse features 310. BEV unit 140 may flatten projection unit 314 for example, from 3D to 2D. BEV unit 140 may obtain therefrom LiDAR BEV features 318.

[0075] BEV unit 140 may also obtain image data 304 (which may be an example of image data 204) from, e.g., camera(s) 104 (FIG.1). Encoder 308 may encode image data 304 to obtain perspective view features 312. BEV unit 140 may apply PV-to-BEV projection unit 316 to obtain camera BEV features 320.

[0076] Feature fusion unit 319 may fuse LiDAR BEV features 318 and camera BEV features 320. The output of feature fusion unit 319 may be decoded by decoder 322 to generate fused features.

[0077] Encoders 306 and 308 may be respective examples of, and function similar to, encoders 206 and 208 of FIG. 2. Flatten projection unit 314 and PV-to-BEV projection unit 316 may be respective examples of, and function similar to, flatten projection unit 214 and PV-BEV projection unit 216 of FIG. 2. Feature fusion unit 319 may be an example of, and function similar to, feature fusion unit 219 of FIG.2. Decoder 322 may be an example of, and function similar to, decoder 322 of FIG.2.

[0078] For example, encoder 308 may lift 2D image features from each camera sensor view (e.g., for each camera of camera(s) 104 of FIG. 1) of image data 304 to generate perspective view features 312. Such lifted features for the ithcamera sensor view may be denoted as ^^^^. 1616-438WO01Qualcomm Ref. No.2401738WO 19

[0079] PV-to-BEV projection unit 316 may project perspective view features 312 onto the BEV plane to generate camera BEV features 320 which may include the BEV features for each camera sensor view. The BEV features for the ithcamera sensor view may be denoted as ^^^^. For point cloud data 302, encoder 306 may extract 3D sparse features 310 and flatten projection unit 314 may transform 3D sparse features 310 into a 2D grid representation to generate LiDAR BEV features 318.

[0080] For example, encoder 308 may extract 2D image features ^^^^ from each camera sensor view i using a convolutional neural network (CNN) backbone, like a residual network (ResNet): ^^^^= ^^^^^^ (^^^^^ Where ^^^^is the RGB image from camera i. This CNN may generate a 2D feature map (e.g., of perspective view features 312) for each camera sensor view. ResNet is a deep learning model generally used for computer vision applications.

[0081] PV-to-BEV projection unit 316 may lift the 2D feature maps of perspective view features 312 to a 3D Point Cloud to generate camera BEV features 320: For each 2D point ^^ = (^^, ^^^ in ^^^^, PV-to-BEV projection unit 316 may project the 2D point to a corresponding 3D point P = (X,Y,Z) using camera intrinsics K. The projection may be learned, for example, via a latent semantic scaling (LSS) approach.

[0082] PV-to-BEV projection unit 316 may project lifted 3D points onto a BEV plane: ^^^^(^^, ^^^ = Σ^^^^^^(^^^ PV-to-BEV projection unit 316 may aggregate features ^^^^(P) that fall into each BEV grid cell (x, y). As such, PV-to-BEV projection unit may generate a BEV feature map Bifor each camera sensor view i in camera BEV features 320.

[0083] In some examples, for point cloud data 302, system 300 may project 3D points onto a BEV grid directly to generate LiDAR BEV features 318: ^^^^^^^^^^^^(^^, ^^^ = Σ^^^^^^^^ Where ^^ (^^^ are LiDAR features like reflectivity.

[0084] Feature fusion unit 319 may fuse LiDAR BEV features 318 and camera BEV features 320. The fused BEV features ^^^^are processed by decoder 322, which may include a CenterPoint detector. Such processing may include: ^^^^ = ^^ (^^^^^ Where ^^ contains a series of convolutional layers, such as deconvolution layers to expand the spatial dimensions of the fused BEV features ^^^^. 1616-438WO01Qualcomm Ref. No.2401738WO 20

[0085] The final prediction layer of decoder 322 generates detections 326: ^^^^ = ^^ (Ĝ^^^ P is typically a convolutional layer with number of filters equal to number of output attributes (e.g., class, box coordinates, etc.)

[0086] Modelling uncertainty for 3D object detection (3DOD) may not be as straight forward as segmentation approaches. The use of 3D boxes versus per-pixel classes makes representing aleatoric uncertainty more complex and may require critical thinking of multiple object interaction uncertainty modelling due to the fact that multiple objects may induce dependencies between their uncertainties. For example, one object may occlude another object from the view of one or more sensors, thereby increasing uncertainty with respect to the occluded object. Also, 3D boxes are generally more sparse than dense pixel labels, providing less signal for modeling uncertainty.

[0087] The fused features of FIG. 3 output by decoder 322 may be collected over a plurality of passes of sensor data of a particular time which may be input to uncertainty unit 340. Uncertainty unit 340 may model uncertainties associated with system 300. Such uncertainties may include aleatoric uncertainties from sensor data (e.g., point cloud data 302 and / or image data 304) and / or epistemic uncertainties associated with model parameters (e.g., of models used for encoders 306, 308, flatten projection unit 314, PV- to-BEV projection unit 316, and / or decoder 322). It should be noted that uncertainty unit 340 is not limited to use in system 300, but may be used with any system attempting to fuse data from multiple sensor systems.

[0088] For example, the determination of camera BEV features 320 may be repeated N times. These steps may be repeated by performing noise augmentation on input camera images, randomizing the networks weights (e.g., of encoder 308 and / or PV-to-BEV projection unit 316), and varying the other hyperparameters. The variance of ^^^^across these different passes yields uncertainty. An alternate way to quantify uncertainty is to estimate cross entropy loss for class probabilities and regression loss for 3D location, box coordinates, etc., relative to a ground truth, and model class probabilities as Bernoulli and object locations as zero mean Gaussian distribution to compute the variance of the error.

[0089] Thus, decoder 322 extracts object features, and the prediction layer of decoder 322 finally generates the detections ^^^^ for each camera sensor view i. Uncertainty unit 340 may determine the mean or median of the object detections across the different sets to obtain a more robust and accurate prediction. For example, uncertainty unit 340 may compute the mean or median and compute a variance of the object detections across the 1616-438WO01Qualcomm Ref. No.2401738WO 21 different data sets (e.g., the plurality of passes for each camera sensor view i) as a measure of the aleatoric uncertainty in the model predictions.

[0090] For example, ensemble unit 330 may generate an ensemble of fused detections based on the plurality of passes as discussed above. Detections and uncertainties unit 332 may determine uncertainties associated with each detection.

[0091] Aleatoric uncertainty unit 334 may quantify the aleatoric uncertainty, for example, through determining the variance across detections at a given time. For example, the object detection for the ithcamera sensor view on the jthforward pass may be denoted as ^^^^,^^, where ^^ = 1, 2, ⋯ ^^. N may be the index of the forward pass. The mean of the object detections across the different forward passes for the ithcamera sensor view may be denoted as ^^^^, and the variance may be denoted as σଶ^.

[0092] Aleatoric uncertainty unit 334 may compute the mean and variance as follows:where ^^ is the number of forward passes.

[0093] The variance σଶ^ is a measure of the variability in the object detections across the different data sets, and hence provides an estimate of the aleatoric uncertainty in the model predictions for the ithcamera sensor view of the associated sensor. A higher value of σଶ^ indicates higher uncertainty in the model predictions for that particular camera sensor view.

[0094] By computing the variance of the object detections across a plurality of data sets, aleatoric uncertainty unit 334 may provide a more reliable estimate of the uncertainty in the model predictions. This allows system 300 to better understand the reliability of the object detections and make more informed decisions (e.g., navigational decisions) based on the level of uncertainty.

[0095] It may be desirable to actively minimize ambiguity and uncertainty in the 3D scene understanding, such as when using a multi-view detection framework. The detections and uncertainty estimates from a plurality of camera sensor views can be noisy and ambiguous. Intelligently fusing the most relevant camera sensor views can improve accuracy.

[0096] As such, uncertainty unit 340 may include a Markov Decision Process (MDP) unit 336. MDP unit 336 may provide a principled mathematical framework for sequential decision making under uncertainty. For example, at each time step t, MDP unit 336 may 1616-438WO01Qualcomm Ref. No.2401738WO 22 provide an observation, a belief state, an action model, a transition model, an observation model, and / or a reward.

[0097] For example, MDP unit 336 may determine an observation ^^^^ = ^^^^^^,for each camera sensor view i (detections and uncertainties). At each time step t, the observation ^^^^includes the object detections ^^^^^^and uncertainty estimates σଶ^௧from each camera sensor view i.

[0098] MDP unit 336 may determine a belief state ^^^^. The belief state ^^^^ may represent the agent's probabilistic belief over the true state of the environment, based on past observations. Specifically, ^^^^may encode: 3D locations of all objects in the scene; velocities of dynamic objects like cars; shape, size, orientation, and class of objects; and / or any other aspects of the environment relevant to the detections. The belief state ^^^^may be represented as a probability distribution over these state variables at each time step t.

[0099] For example, an object's location may be represented as a Gaussian N(μ, Σ), where the covariance Σ encodes the uncertainty in the location. The belief state ^^^^, thus may compactly capture an agent's uncertain knowledge of the full 3D environment state based on sensor observations over time.

[0100] MDP unit 336 may determine an action model ^^^^ = ^^^^. For example, at each time step t, the agent receives object detections ^^^^^^and uncertainty estimatesfrom each camera sensor view i. It may be desirable to intelligently fuse the information from the multiple camera sensor views to minimize overall uncertainty about the environment state ^^^^. A fusion policy π takes in the current state ^^^^and outputs a fusion strategy, such as a weighted average where camera sensor views with lower uncertainty are weighted higher. The observation ^^^^is generated by executing this fusion strategy on the current view-level detections. This observation ^^^^may then be used to update the environment state belief ^^^^via, for example, Bayesian filtering.

[0101] MDP unit 336 may determine a transition model: ^^(^^^^+1| ^^^^, ^^^^). The transition model defines the probability distribution over the next state ^^^^+1given the current state ^^^^ and action taken ^^^^. This models how the true environment state evolves over time due to factors like object motion, how object locations change over time due to motion, the evolution of object velocities and trajectories and / or changes in object orientations, shapes, etc.

[0102] Mathematically, the transition model defines the conditional probability: 1616-438WO01Qualcomm Ref. No.2401738WO 23 ^^(^^^^+1|^^^^, ^^^^^= ^^(^^^^+1|^^^^, ^^^^^ T is the transition model, ^^^^is the state at time t, ^^^^is the action taken at time t, ^^^^+1is the next state at time t+1.

[0103] ^^^^^ gives the probability distribution over next states given current state and action. So, this defines the transition model T as the conditional probability distribution over the next state ^^^^+1, given the previous state ^^^^ and action taken ^^^^.

[0104] This probability distribution can be learned if uncertainty unit 340 has access to the true states ^^^^ at each time step. If uncertainty unit 340 does not have access to the true states ^^^^ at each time step, the probability distribution can be approximated by domain knowledge about object motion. For example, MDP unit 336 may model the motion of vehicles on the road as following predictable trajectories with some noise. The transition model may capture the natural evolution of the true environment state over time in response to the agent's actions. Such a model may predict how the belief state changes.

[0105] MDP unit 336 may determine an observation model ^^(^^^^|^^^^, ^^^^^ = ^^(^^^^|^^^^, ^^^^^. O is the observation model, ^^^^ is the observation at time t, ^^^^ is the true state at time t ^^^^ is the action taken at time t. ^^(^^^^|^^^^, ^^^^^ gives the probability of observation ^^^^ given state ^^^^ and action ^^^^.

[0106] So, the observation model O is a conditional probability distribution over observations ^^^^, given the true state ^^^^ and action ^^^^. This models the process of generating observations from the true environment state. At each time step t, the observation ^^^^ may include object detections ^^^^^^and uncertainty estimates σଶ^௧from each camera sensor view i.

[0107] The true state ^^^^represents the actual locations, velocities etc. of objects in the environment. The observation model may account for: the camera projection process to generate 2D observations from the 3D state; sensor noise and errors that affect the detections ^^^^^^; and limitations of the camera sensor view that influence uncertainties σଶ^௧.

[0108] By modeling how observations are generated from the hidden true state, MDP unit 336 may infer ^^^^ from the observations ^^^^ received over time. This allows estimating the belief state, which represents uncertain knowledge of ^^^^ based on the observed data. The observation model essentially models the uncertainties and noise inherent in real-world sensing from multiple camera sensor views. 1616-438WO01Qualcomm Ref. No.2401738WO 24

[0109] MDP unit 336 may determine a reward. As it may be desirable to minimize uncertainty in the object detections from multiple camera sensor views, the reward quantifies the amount of uncertainty reduction at each time step. ^^^^ = ∑ ^ σଶ ^௧ − ∑ ^ σଶ ^^௧ା^^(reduction in uncertainty) Where ^^^^is the reward at time t, σଶ^௧is the uncertainty (variance) for camera sensor view i at time t, and is the uncertainty for camera sensor view i at next time t+.

[0110] The reward sums the uncertainty across all camera sensor views at each time step.The difference ∑ଶ ^σ^௧ − ∑ଶ ^σ^^௧ା^^measures how much the total uncertainty decreased from t to t+1.

[0111] Maximizing this reward results in actively reducing ambiguity by selecting camera sensor views that minimize uncertainty in the object detections. So, the reward may be defined as the reduction in total uncertainty across all camera sensor views from time t to t+1. Maximizing this reward results in actively minimizing the detection uncertainty at each time step.

[0112] MDP unit 336 may determine a fusion policy. For example, MDP unit 336 may generate a policy π(^^^^|^^^^) to intelligently fuse information from the multiple camera sensor views to maximize discounted reward (negative of uncertainty) summed over all camera sensor views over the duration of a given episode. ^^^^^^^|^^^^,^^^^^ = ∑௧ ∑^^ ^^ ^^^^^|^^^^, ^^^^^ ∗ ^^^^^^^^^, ^^^^, ^^^^^Where ^^^^^^^|^^^^, ^^^^^ is the observation likelihood given fusion strategy ^^^^, and ^^^^^^^^^, ^^^^, ^^^^^ is the reward after fusing camera sensor views with strategy ^^^^ and observing ^^^^.

[0113] The policy evaluates each possible fusion strategy ^^^^by 1) considering all possible observations ^^^^that could result from fusing with ^^^^; and 2) calculating the expected reward across all these ^^^^, weighted by their likelihood. By looking ahead at future rewards, the policy aims to minimize ambiguity from detections in the long run.

[0114] MDP unit 336 may update a belief state. The belief state ^^^^represents the agent's probabilistic belief over the true state of the environment at time t. When a new observation ^^^^ is received after taking action at, system 300 may use filtering, such as Bayesian filtering, to update the belief state to ^^^^+1 :1616-438WO01Qualcomm Ref. No.2401738WO 25 Where ^^(^^^^+1|^^^^) is the posterior distribution over states after observing ^^^^, ^^^^^^^|^^^^+1^ is the observation likelihood,is the transition model, ^^^^^^^^ is the prior belief state, and η is a normalization constant.

[0115] MDP unit 336 may perform Bayesian updating to incorporate the new observation ^^^^ and obtain the posterior belief ^^(^^^^+1|^^^^) over the next state.

[0116] So, Bayesian filtering may incorporate the new observation ^^^^ to update the probability distribution over environment states via Bayes' rule. This allows the belief state to accumulate evidence over time and maintain an estimate of the true world state as new observations come in.

[0117] MDP unit 336 may perform policy learning. It may be desirable to learn an optimal policy π* that maximizes expected long-term reward. This reward may be the cumulative reduction in detection uncertainty over the episode. System 300 may use reinforcement learning algorithms, like policy gradient or Q-learning, to optimize the policy.

[0118] For example, with a policy gradient, the agent interacts with the environment, selecting camera sensor views and observing states / rewards. The agent calculates the expected return Gt for each timestep based on the rewards received. The policy parameters θ are updated to maximize the expected returns: ^^^^ ≈ ^^^^^^^^ [^^^^^

[0119] By repeating this process over many episodes (e.g., over time), the policy maximizes long-term reward through gradient ascent on E[Gt].

[0120] For example, with Q-learning, a Q-function is learned to estimate expected reward ^^^^^, ^^^. Actions are selected to maximize ^^^^^, ^^^. Q is updated based on observed rewards: ^^(^^^^, ^^^^) ← ^^(^^^^, ^^^^) + α (^^^^+ γ maxa^^(^^^^^1, ^^) − ^^(^^^^, ^^^^)) Where ^^(^^^^, ^^^^) is the Q-value for state ^^^^and action ^^^^, α is the learning rate, ^^^^is the reward received at time t, γ is the discount factor, and maxa^^(^^^^^1, ^^) is the estimated future value.

[0121] This updates the Q-value ^^(^^^^, ^^^^) towards the observed reward ^^^^plus the maximum discounted future Q-value, in order to optimize the policy. This also optimizes the policy to maximize cumulative future reward.

[0122] In summary, uncertainty estimation is formulated as an MDP that reasons about beliefs, uncertainties, and rewards to reduce ambiguity in the scene understanding. 1616-438WO01Qualcomm Ref. No.2401738WO 26

[0123] FIG. 4 is a flow diagram illustrating example multi-vehicle BEV feature fusion techniques according to one or more aspects of this disclosure. While primarily described with respect to processing system 100 of FIG.1 and system 300 of FIG.3, the techniques of this disclosure may be practiced by any system capable of doing so.

[0124] Processing system 100 may obtain the sensor data (400). For example, processing system 100 may capture or receive point cloud data 202 from LiDAR system 102, image data 304 from camera(s) 104, and / or other sensor data from sensor(s) 108.

[0125] Processing system 100 may determine an object in the sensor data using one or more 3D object detection models (402). For example, processing system 100 may fuse LiDAR BEV features 318 and camera BEV features 320 and determine an object in the sensor data.

[0126] Processing system 100 may determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty (404). For example, aleatoric uncertainty unit 334 may determine an aleatoric uncertainty associated with the object.

[0127] Processing system 100 may adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models (406). For example, processing system 100 may iteratively adjust model parameters, such as weights of encoders, decoders, and / or weights of system 300 and / or may be applied by feature fusion unit 319, to data from particular sensors over time to reduce the aleatoric uncertainty. This adjusting may generate one or more adjusted 3D object detection models. For example, processing system 100 may adjust one or more model parameters in BEV unit 140 to generate the one or more adjusted 3D object detection models of BEV unit 140.

[0128] Processing system 100 may navigate a vehicle based on output from the one or more adjusted 3D object detection models (408). For example, processing system 100 may use model output 172 of the adjusted object detection models when detecting objects and determining a probabilistic belief state about a 3D environment surrounding the vehicle. Processing system 100 may use the probabilistic belief state when deciding whether to brake, accelerate, turn, or otherwise navigate the vehicle within the 3D environment and / or whether to gather more sensor data to increase the probability that the probabilistic belief state matches a true state of the 3D environment. 1616-438WO01Qualcomm Ref. No.2401738WO 27

[0129] In some examples, each of the plurality of passes includes at least one of different sensor noise or at least one different model parameter from another of the plurality of passes. In some examples, the uncertainty includes a variance. In some examples, the uncertainty further includes an epistemic uncertainty.

[0130] In some examples, as part of adjusting the model parameters, processing system 100 may apply a Markov Decision Process to the determined uncertainty. In some examples, as part of applying the Markov Decision Process, processing system 100 may define a reward as a reduction in total uncertainty across all the respective fields of view at each of a plurality of times (e.g., at each time step t).

[0131] In some examples, processing system 100 may train a policy via enforcement learning to maximize the reward. In some examples, the enforcement learning includes applying a policy gradient or Q-learning to the policy.

[0132] In some examples, the object is a first object of a plurality of objects, the uncertainty is a first uncertainty, and the particular time is a first particular time. In some examples, processing system 100 may determine the plurality of objects in the sensor data using 3D object detection, the plurality of objects being in a 3D environment around the vehicle. Processing system 100 may, for each respective object of the plurality of objects, determine a respective first uncertainty associated with the respective object based on the plurality of passes of the sensor data at the first particular time. Processing system 100 may determine a probabilistic belief state regarding the 3D environment based on a plurality of first uncertainties, the plurality of first uncertainties comprising the respective first uncertainties and the probabilistic belief state representing a probabilistic belief associated with a true state of the 3D environment.

[0133] In some examples, processing system 100 may, for each respective object of the plurality of objects, determine a respective second uncertainty associated with the respective object based on a plurality of passes of the sensor data at a second particular time. Processing system 100 may update the probabilistic belief state through the application of Bayesian filtering based on a plurality of second uncertainties, the plurality of second uncertainties comprising the respective second uncertainties.

[0134] In some examples, the sensor data further comprises LiDAR data from one or more LiDAR sensors (e.g., LiDAR system 102).

[0135] The techniques of this disclosure may provide for more reliable detections by intelligently weighting camera sensor views based on estimated uncertainty. Such a weighting may lead to improved accuracy over single-view or naive fusion approaches. 1616-438WO01Qualcomm Ref. No.2401738WO 28 The techniques of this disclosure may provide an ability to quantify confidence and identify ambiguous or occluded regions requiring further sensing. This may provide potentially valuable information for decision making, such as whether to immediately take an action or to delay taking an action while additional sensor data is gathered to improve confidence and / or reduce ambiguity.

[0136] The techniques of this disclosure may propagate uncertainty through the entire pipeline from raw sensor data to 3D object beliefs over time, thereby capturing effects of noise at every stage of the pipeline. The techniques of this disclosure may learn most informative sensing actions through reinforcement learning to actively reduce uncertainty and optimizes long-term perception instead of optimizing single independent snapshots of data.

[0137] The techniques of this disclosure handle missing and ambiguous detections via probabilistic state representation and may be more robust to real-world sensing challenges than existing techniques. The techniques of this disclosure provide a flexible framework that can incorporate different modalities and fuse on various representations like BEV, 3D points etc.

[0138] The techniques of this disclosure may provide calibrated estimates of aleatoric and epistemic uncertainty to diagnose model and data limitations. The techniques of this disclosure may enable uncertainty-aware applications, such as autonomous driving applications, that require or desire reliability guarantees in uncertain environments

[0139] Additional aspects of the disclosure are detailed in numbered clauses below.

[0140] Clause 1. A system for processing data, the system comprising: one or more memories for storing sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; and one or more processors in communication with the one or more memories, the one or more processors configured to: obtain the sensor data; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models. 1616-438WO01Qualcomm Ref. No.2401738WO 29

[0141] Clause 2. The system of clause 1, wherein each of the plurality of passes includes at least one of different sensor noise or at least one different model parameter from another of the plurality of passes, and wherein the uncertainty comprises a variance.

[0142] Clause 3. The system of clause 1 or clause 2, wherein the uncertainty further comprises an epistemic uncertainty.

[0143] Clause 4. The system of any of clauses 1-3, wherein as part of adjusting the model parameters, the one or more processors are configured to apply a Markov Decision Process to the determined uncertainty.

[0144] Clause 5. The system of clause 4, wherein as part of applying the Markov Decision Process, the one or more processors are configured to define a reward as a reduction in total uncertainty across all the respective fields of view at each of a plurality of times.

[0145] Clause 6. The system of clause 5, wherein the one or more processors are further configured to train a policy via enforcement learning to maximize the reward.

[0146] Clause 7. The system of clause 6, wherein the enforcement learning comprises applying a policy gradient or Q-learning to the policy.

[0147] Clause 8. The system of any of clauses 1-7, wherein the object is a first object of a plurality of objects, the uncertainty is a first uncertainty, and the particular time is a first particular time, and wherein the one or more processors are further configured to: determine the plurality of objects in the sensor data using 3D object detection, the plurality of objects being in a 3D environment around the vehicle; for each respective object of the plurality of objects, determine a respective first uncertainty associated with the respective object based on the plurality of passes of the sensor data at the first particular time; and determine a probabilistic belief state regarding the 3D environment based on a plurality of first uncertainties, the plurality of first uncertainties comprising the respective first uncertainties and the probabilistic belief state representing a probabilistic belief associated with a true state of the 3D environment.

[0148] Clause 9. The system of clause 8, wherein the one or more processors are further configured to: for each respective object of the plurality of objects, determine a respective second uncertainty associated with the respective object based on a plurality of passes of the sensor data at a second particular time; and update the probabilistic belief state through application of Bayesian filtering based on a plurality of second uncertainties, the plurality of second uncertainties comprising the respective second uncertainties. 1616-438WO01Qualcomm Ref. No.2401738WO 30

[0149] Clause 10. The system of any of clauses 1-9, wherein the sensor data further comprises Light Detection and Ranging (LiDAR) data from one or more LiDAR sensors.

[0150] Clause 11. A method for processing data, the method comprising: obtaining sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determining an object in the sensor data using one or more three-dimensional (3D) object detection models; determining an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjusting model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigating a vehicle based on output from the one or more adjusted 3D object detection models.

[0151] Clause 12. The method of clause 11, wherein each of the plurality of passes includes at least one of different sensor noise or at least one different model parameter from another of the plurality of passes, and wherein the uncertainty comprises a variance.

[0152] Clause 13. The method of clause 11 or clause 12, wherein the uncertainty further comprises an epistemic uncertainty.

[0153] Clause 14. The method of any of clauses 11-14, wherein adjusting the model parameters comprises applying a Markov Decision Process to the determined uncertainty.

[0154] Clause 15. The method of clause 14, wherein applying the Markov Decision Process comprises defining a reward as a reduction in total uncertainty across all the respective fields of view at each of a plurality of times.

[0155] Clause 16. The method of clause 15, further comprising training a policy via enforcement learning to maximize the reward.

[0156] Clause 17. The method of clause 16, wherein the enforcement learning comprises applying a policy gradient or Q-learning to the policy.

[0157] Clause 18. The method of any of clauses 11-17, wherein the object is a first object of a plurality of objects, the uncertainty is a first uncertainty, and the particular time is a first particular time, and wherein the method further comprises: determining the plurality of objects in the sensor data using 3D object detection, the plurality of objects being in a 3D environment around the vehicle; for each respective object of the plurality of objects, determining a respective first uncertainty associated with the respective object based on the plurality of passes of the sensor data at the first particular time; and 1616-438WO01Qualcomm Ref. No.2401738WO 31 determining a probabilistic belief state regarding the 3D environment based on a plurality of first uncertainties, the plurality of first uncertainties comprising the respective first uncertainties and the probabilistic belief state representing a probabilistic belief associated with a true state of the 3D environment.

[0158] Clause 19. The method of clause 18, further comprising: for each respective object of the plurality of objects, determining a respective second uncertainty associated with the respective object based on a plurality of passes of the sensor data at a second particular time; and updating the probabilistic belief state through application of Bayesian filtering based on a plurality of second uncertainties, the plurality of second uncertainties comprising the respective second uncertainties.

[0159] Clause 20. Non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to: obtain sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models.

[0160] It is to be recognized that depending on the example, certain acts or events of any of the techniques described herein can be performed in a different sequence, may be added, merged, or left out altogether (e.g., not all described acts or events are necessary for the practice of the techniques). Moreover, in certain examples, acts or events may be performed concurrently, e.g., through multi-threaded processing, interrupt processing, or multiple processors, rather than sequentially.

[0161] In one or more examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored on or transmitted over as one or more instructions or code on a computer-readable medium and applied by a hardware-based processing unit. Computer-readable media may include computer-readable storage media, which corresponds to a tangible medium such as data storage media, or communication media 1616-438WO01Qualcomm Ref. No.2401738WO 32 including any medium that facilitates transfer of a computer program from one place to another, e.g., according to a communication protocol. In this manner, computer-readable media generally may correspond to (1) tangible computer-readable storage media which is non-transitory or (2) a communication medium such as a signal or carrier wave. Data storage media may be any available media that can be accessed by one or more computers or one or more processors to retrieve instructions, code and / or data structures for implementation of the techniques described in this disclosure. A computer program product may include a computer-readable medium.

[0162] By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage, or other magnetic storage devices, flash memory, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Also, any connection is properly termed a computer- readable medium. For example, if instructions are transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of medium. It should be understood, however, that computer-readable storage media and data storage media do not include connections, carrier waves, signals, or other transitory media, but are instead directed to non-transitory, tangible storage media. Disk and disc, as used herein, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc, where disks usually reproduce data magnetically, while discs reproduce data optically with lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0163] Instructions may be applied by one or more processors, such as one or more DSPs, general purpose microprocessors, ASICs, FPGAs, or other equivalent integrated or discrete logic circuitry. Accordingly, the terms “processor” and “processing circuitry,” as used herein may refer to any of the foregoing structures or any other structure suitable for implementation of the techniques described herein. In addition, in some aspects, the functionality described herein may be provided within dedicated hardware and / or software modules configured for encoding and decoding, or incorporated in a combined codec. Also, the techniques could be fully implemented in one or more circuits or logic elements. 1616-438WO01Qualcomm Ref. No.2401738WO 33

[0164] The techniques of this disclosure may be implemented in a wide variety of devices or apparatuses, including a wireless handset, an integrated circuit (IC) or a set of ICs (e.g., a chip set). Various components, modules, or units are described in this disclosure to emphasize functional aspects of devices configured to perform the disclosed techniques, but do not necessarily require realization by different hardware units. Rather, as described above, various units may be combined in a codec hardware unit or provided by a collection of interoperative hardware units, including one or more processors as described above, in conjunction with suitable software and / or firmware.

[0165] Various examples have been described. These and other examples are within the scope of the following claims. 1616-438WO01

Claims

Qualcomm Ref. No.2401738WO 34 CLAIMS What is claimed is:

1. A system for processing data, the system comprising: one or more memories for storing sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; and one or more processors in communication with the one or more memories, the one or more processors configured to: obtain the sensor data; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models.

2. The system of claim 1, wherein each of the plurality of passes includes at least one of different sensor noise or at least one different model parameter from another of the plurality of passes, and wherein the uncertainty comprises a variance.

3. The system of claim 1, wherein the uncertainty further comprises an epistemic uncertainty.

4. The system of claim 1, wherein as part of adjusting the model parameters, the one or more processors are configured to apply a Markov Decision Process to the determined uncertainty. 1616-438WO01Qualcomm Ref. No.2401738WO 35 5. The system of claim 4, wherein as part of applying the Markov Decision Process, the one or more processors are configured to define a reward as a reduction in total uncertainty across all the respective fields of view at each of a plurality of times.

6. The system of claim 5, wherein the one or more processors are further configured to train a policy via enforcement learning to maximize the reward.

7. The system of claim 6, wherein the enforcement learning comprises applying a policy gradient or Q-learning to the policy.

8. The system of claim 1, wherein the object is a first object of a plurality of objects, the uncertainty is a first uncertainty, and the particular time is a first particular time, and wherein the one or more processors are further configured to: determine the plurality of objects in the sensor data using 3D object detection, the plurality of objects being in a 3D environment around the vehicle; for each respective object of the plurality of objects, determine a respective first uncertainty associated with the respective object based on the plurality of passes of the sensor data at the first particular time; and determine a probabilistic belief state regarding the 3D environment based on a plurality of first uncertainties, the plurality of first uncertainties comprising the respective first uncertainties and the probabilistic belief state representing a probabilistic belief associated with a true state of the 3D environment.

9. The system of claim 8, wherein the one or more processors are further configured to: for each respective object of the plurality of objects, determine a respective second uncertainty associated with the respective object based on a plurality of passes of the sensor data at a second particular time; and update the probabilistic belief state through application of Bayesian filtering based on a plurality of second uncertainties, the plurality of second uncertainties comprising the respective second uncertainties.

10. The system of claim 1, wherein the sensor data further comprises Light Detection and Ranging (LiDAR) data from one or more LiDAR sensors. 1616-438WO01Qualcomm Ref. No.2401738WO 36 11. A method for processing data, the method comprising: obtaining sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determining an object in the sensor data using one or more three-dimensional (3D) object detection models; determining an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; adjusting model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigating a vehicle based on output from the one or more adjusted 3D object detection models.

12. The method of claim 11, wherein each of the plurality of passes includes at least one of different sensor noise or at least one different model parameter from another of the plurality of passes, and wherein the uncertainty comprises a variance.

13. The method of claim 11, wherein the uncertainty further comprises an epistemic uncertainty.

14. The method of claim 11, wherein adjusting the model parameters comprises applying a Markov Decision Process to the determined uncertainty.

15. The method of claim 14, wherein applying the Markov Decision Process comprises defining a reward as a reduction in total uncertainty across all the respective fields of view at each of a plurality of times.

16. The method of claim 15, further comprising training a policy via enforcement learning to maximize the reward. 1616-438WO01Qualcomm Ref. No.2401738WO 37 17. The method of claim 16, wherein the enforcement learning comprises applying a policy gradient or Q-learning to the policy.

18. The method of claim 11, wherein the object is a first object of a plurality of objects, the uncertainty is a first uncertainty, and the particular time is a first particular time, and wherein the method further comprises: determining the plurality of objects in the sensor data using 3D object detection, the plurality of objects being in a 3D environment around the vehicle; for each respective object of the plurality of objects, determining a respective first uncertainty associated with the respective object based on the plurality of passes of the sensor data at the first particular time; and determining a probabilistic belief state regarding the 3D environment based on a plurality of first uncertainties, the plurality of first uncertainties comprising the respective first uncertainties and the probabilistic belief state representing a probabilistic belief associated with a true state of the 3D environment.

19. The method of claim 18, further comprising: for each respective object of the plurality of objects, determining a respective second uncertainty associated with the respective object based on a plurality of passes of the sensor data at a second particular time; and updating the probabilistic belief state through application of Bayesian filtering based on a plurality of second uncertainties, the plurality of second uncertainties comprising the respective second uncertainties.

20. Non-transitory computer-readable media storing instructions, which, when executed by one or more processors, cause the one or more processors to: obtain sensor data, the sensor data comprising data from a plurality of camera sensors, each camera sensor of the plurality of camera sensors having a respective field of view; determine an object in the sensor data using one or more three-dimensional (3D) object detection models; determine an uncertainty associated with the object based on a plurality of passes of the sensor data at a particular time, the uncertainty comprising an aleatoric uncertainty; 1616-438WO01Qualcomm Ref. No.2401738WO 38 adjust model parameters of the one or more 3D object detection models to reduce the uncertainty over time, wherein adjusting the model parameters of the one or more 3D object detection models generates one or more adjusted 3D object detection models; and navigate a vehicle based on output from the one or more adjusted 3D object detection models. 1616-438WO01

Images