Fusion of radar and infrared data for object detection and tracking

Abstract

A vehicle computing system may receive and fuse long wave infrared data with radar data to detect and track objects in a low visibility driving environment. In some examples, radar data, including position and velocity data, may be projected on infrared image data to detect, classify, and track infrared emitting objects. An attention machine learning transformer model may also be trained to output object detection based on the combined infrared and radar data. The fusion of infrared and radar data may be used alone and / or may be synchronized with other sensor modalities. In some examples, fusion and analysis of infrared and radar data may be used in a specific low-visibility driving environment, using low-illumination, fog, and shadow areas of the environment to detect and track infrared emitting objects and / or moving objects (e.g., pedestrians and animals), which may be occluded from being detected by other sensor modalities.

Description

Cross Reference to Related Applications

[0001] This PCT international application claims the benefit of priority to U.S. Application No. 18 / 239,860, filed August 30, 2023, entitled “FUSION OF RADAR AND INFRARED DATA FOR OBJECT DETECTION AND TRACKING,” the entire contents of which are incorporated by reference herein in their entirety for all purposes. BACKGROUND

[0002] Autonomous vehicles can include various software-based systems, hardware-based systems, and / or controllers to guide the vehicle through an environment. For example, a controller of an autonomous vehicle can use sensor data captured by a sensor system to perceive objects, predict trajectories, and plan and optimize a route to guide the vehicle through an environment containing static objects and dynamic objects. To ensure the safety of passengers and surrounding people and objects, the autonomous vehicle can receive and analyze sensor data in various ways to make driving decisions when traversing through a congested area with other moving vehicles (autonomous vehicles or other vehicles), moving people, stationary buildings, etc. However, because an autonomous vehicle can include two or more different types (or modalities) of sensors, and the format of the sensor data, sensor placement, sensor characteristics, and content can vary greatly, detections generated by the two different sensor types can differ. Differences between detections determined from different sensor types can cause the object representations created by the vehicle to appear jittery and / or flickering. Moreover, different types of sensors can also be prone to different error variations in size, distance, and depth measurements, which can further complicate object detection and tracking. For example, different types and / or configurations of sensors can perform differently when detecting small objects, moving objects, groups of objects, and / or in various different driving environments. Such performance differences can create technical challenges that hinder the operation of vehicle navigation systems, vehicle safety systems, and / or training machine learning (ML) models. BRIEF DESCRIPTION OF DRAWINGS

[0003] The detailed description is described with reference to the accompanying figures. In the figures, the left-most (one or more) digit(s) of each reference number identifies the figure in which that reference number first appears. The use of the same reference numbers in different figures indicates similar or identical components or features.

[0004] Figure 1 Example techniques of a vehicle configured to fuse infrared sensor data and radar sensor data to perform object detection and tracking capabilities are shown in accordance with one or more examples of the present disclosure.

[0005] Figure 2A ANDFigure 2B Two examples of performing object detection and tracking in low visibility environments using fused infrared and radar data are shown in accordance with one or more examples of the present disclosure.

[0006] Figure 3 An example system including an infrared-radar fusion system and a sensor synchronization component configured to determine object detection based on input from different sensor modalities is shown in accordance with one or more examples of the present disclosure.

[0007] Figure 4 A block diagram of an example technique for generating training data to train a machine learning model for detecting objects is shown in accordance with one or more examples of the present disclosure.

[0008] Figure 5A and Figure 5B Two different example architectures of a machine learning transformer model that can be used to implement various techniques described herein are depicted.

[0009] Figure 6 A block diagram of an example system for implementing various techniques described herein is depicted.

[0010] Figure 7 is a flow diagram showing an example process for performing object detection and tracking in low visibility environments using fused infrared and radar data in accordance with one or more examples of the present disclosure. DETAILED DESCRIPTION

[0011] The technology described herein relates to detecting, classifying, and tracking objects by a vehicle in a driving environment based on fusion of infrared and radar sensor data. An infrared-radar fusion system of a vehicle can receive infrared data (e.g., long-wave infrared data (LWIR) captured by an infrared sensor of the vehicle), as well as radar data captured by a radar sensor covering a corresponding area, to determine a combined infrared-radar sensor view and / or object detections. In some cases, the infrared-radar fusion system can project radar points including position and velocity data onto infrared image data to improve detection of infrared-emitting objects within the environment. Additionally or alternatively, the infrared-radar fusion system can include a machine learning transformer model with attention that is trained to output object detections based on combined infrared data and radar data embeddings. The fusion of infrared and radar data described herein can provide particular advantages for detecting and tracking objects in low-visibility driving environments (e.g., nighttime driving or other low-light driving, fog or rain conditions, or in tunnels or shaded areas where individual sensor modalities can not accurately detect, segment, and classify objects). Other examples described herein include synchronizing the fused infrared and radar data with additional sensor modalities (e.g., RGB cameras, lidar sensors, etc.), as well as conditionally using the fused infrared and radar data in low-visibility driving environments.

[0012] In various examples, an infrared-radar fusion system can operate within a perception component of an autonomous vehicle and can be used to detect, classify, and track objects in an environment of the vehicle. The infrared-radar fusion system can determine and output object detections, which can include a bounding shape (e.g., a bounding box or a bounding contour) and / or other data representing an object detected or perceived based on infrared and / or radar sensor data. Input to the infrared-radar fusion system and / or the perception component can include multi-modal and / or multi-spectral data captured by various sensors of the autonomous vehicle. As described herein, multi-modal data can refer to data captured by different types or modalities of sensors (e.g., RGB (or visible light) cameras, lidar sensors, radar sensors, infrared cameras, etc.), and multi-spectral data can refer to data obtained in different spectral bands. For example, multi-spectral data can be captured by separate image sensors / cameras operating in different spectral bands (e.g., imaging sensors for capturing visible light, infrared sensors for capturing long-wave infrared bands, etc.) and / or multi-spectral sensors for capturing data spanning several discrete spectral bands (e.g., ultraviolet to visible light, visible light to infrared), and / or hyperspectral sensor(s) that can capture nearly continuous wavelengths spanning a wide range of the electromagnetic spectrum.

[0013] As described above, the techniques described herein can be performed by a perception component within the vehicle computing system(s) of a vehicle. The perception component can include one or more object detection components that use trained machine learning (ML) models to detect, segment, and / or classify objects in an environment based on input data from one or more sensor modalities (e.g., sensor types and / or spectral bands). Examples of object detection and tracking techniques performed based on multi-modal sensor data can be found, for example, in U.S. Patent Application No. 18 / 104,082, filed January 31, 2023, entitled “Object Detection and Tracking Using Machine Learning Transformer Models with Attention,” which is incorporated by reference herein in its entirety for all purposes. Additional examples of techniques for determining object detections (e.g., 3D bounding regions) based on object perception from multi-modal sensor data can be found, for example, in U.S. Patent Application No. 15 / 970,838, filed May 3, 2018, entitled “Associating Lidar Data and Image Data,” and U.S. Patent Application No. 16 / 386,249, filed April 16, 2019, entitled “Multi-Modal Sensor Data Association Architecture,” both of which are incorporated by reference herein in their entirety for all purposes. Furthermore, examples of techniques for generating and adjusting parameters to improve object detections associated with long-wave infrared sensor data can be found, for example, in U.S. Patent Application No. 17 / 247,465, filed December 11, 2020, entitled “System for Tuning Parameters of a Thermal Sensor on a Region of Interest,” which is also incorporated by reference herein in its entirety for all purposes.

[0014] To process and analyze multi-modal sensor data, certain existing systems can use multiple processing pipelines (e.g., machine learning pipelines) to process sensor data and / or generate object detections, which can be developed independently for different sensor modalities. For example, a first pipeline can determine object detections based on visible image data, a second independent pipeline can determine object detections based on lidar point cloud data, a third independent pipeline can determine object detections based on radar point cloud data, etc. Within such systems, various techniques can be used to combine object detections (e.g., bounding shapes) from different pipelines of different sensor types / modalities into a combined object detection that can be used to control a vehicle.

[0015] While utilizing different sensor modalities to detect objects and determine additional information in an environment can provide a more accurate representation of the environment, combining sensor data of different modalities can present technical challenges. For example, sensor data captured from the same environment using different sensor modalities can include differences in the number of detected objects and / or the position, size, or other attributes of the detected objects, as different modalities of sensors can capture different data and can be susceptible to different error variations, etc. Moreover, combining sensor data of different modalities (such that sensor data from one modality can be used to modify detection results made in sensor data from other modalities) can be error-prone and computationally expensive.

[0016] Moreover, for certain objects and / or in certain environments, there can not be a sensor modality that can accurately detect, segment, and classify objects individually. For example, in low light or other low visibility conditions, objects such as animals, pedestrians, cyclists, and / or other dynamic objects can be difficult to detect. Visible imaging sensors (e.g., RGB cameras) can not be able to detect such objects in low visibility conditions. Radar and lidar sensors can be able to detect objects in low light environments, but can not be able to detect objects in other low visibility conditions (e.g., through fog or steam). Additionally, for objects in low light environments, radar and lidar sensors can not be able to determine whether the objects are static or dynamic objects, and can not be able to accurately classify, segment, and / or track the objects. Even in low light or low visibility conditions, infrared sensors can detect objects emitting infrared from heat sources (e.g., animals, pedestrians, cyclists, vehicle engines, road glare, etc.). However, infrared sensors can only provide low resolution infrared image data, and thus can not be able to determine accurate position and velocity data for such objects, and can not be able to accurately segment or classify the objects.

[0017] To address the technical problems and inefficiencies of existing object detection systems, the technology described herein can include capturing and fusing infrared data (e.g., LWIR data) and radar data to detect and track objects in low-visibility environments. In some examples, a perception component or system within a vehicle can include an infrared-radar fusion system configured to generate a combined (or fused) infrared-radar sensor view, and / or perform object detection based on a fusion of infrared and radar data within a driving environment. In these examples, infrared data (e.g., LWIR data) can include an infrared image identifying heat-emitting sources within a low-visibility region of the driving environment. This infrared image can be combined / fused with radar data, which can provide higher resolution radar point data for any heat-emitting objects within the infrared data, including point location data (e.g., range, azimuth, and / or elevation) and point velocity data. By combining infrared and radar data at an early stage of the object detection process, the infrared-radar fusion system can more accurately detect and analyze heat-emitting objects (e.g., animals, pedestrians, etc.) within the low-visibility region.

[0018] In some examples, the infrared-radar fusion system can be configured to generate a combined infrared (IR) and radar sensor data view by superimposing radar data onto infrared data. For example, the infrared-radar fusion system can receive radar points from a radar device, and then project the radar points onto an infrared image. To perform such a projection, the infrared-radar fusion system can be configured to initially determine (or receive) overlapping fields of view between various infrared and / or radar sensors used to capture the sensor data. For example, for sensors integrated into and / or mounted on a vehicle, the field of view of a sensor can be based on the sensor’s position relative to the vehicle, orientation angle, and / or range. The vehicle can include any number of infrared sensors and any number of radar sensors, and the various sensors can be located at different positions on the vehicle. The infrared sensors and / or radar sensors on the vehicle can overlap with other sensors of the same modality and / or sensors of different modalities. Thus, to superimpose a set of radar points onto an infrared image, the infrared-radar fusion system can initially determine the overlapping fields of view of the various sensors, which can include stitching overlapping infrared images captured by different infrared sensors on the vehicle, and / or combining multiple radar point clouds captured by different radar devices on the vehicle.

[0019] The infrared-radar fusion system can also use various projection techniques to project radar points onto infrared data images (and vice versa, projecting infrared data onto radar point clouds). For example, the infrared-radar fusion system can spatially align infrared data (e.g., 2D infrared images) with radar data (e.g., 3D radar point clouds) based on the relative positions and / or orientation angles of the sensor devices. The infrared-radar fusion system can then re-project radar points onto 2D infrared images based on the alignment of the sensor devices. In some cases, the infrared-radar fusion system can use ray casting and / or ray tracing to project radar points onto infrared data by casting rays from the position of the radar sensor towards the 2D view of the infrared image to determine the ray intersection point of each radar point relative to the 2D infrared image. Using the sensor data views including radar points projected onto infrared data, the infrared-radar fusion system can execute one or more trained models configured to detect, segment, classify, and / or track objects within the combined sensor data views.

[0020] Additionally or alternatively, the infrared-radar fusion system can include a machine learning transformer model (or other deep neural network) trained to output a combined sensor data view in the form of object detections based on infrared and radar input data. For example, the transformer model can be configured to receive input embeddings including infrared (e.g., LWIR) data and radar data. The transformer model can combine the embeddings, e.g., using an attention mechanism or via concatenation, and then provide the embeddings to an attention head of the transformer model. The transformer model can use an encoder-decoder architecture described in more detail below to determine and output object detection data based on the input embeddings. In various examples, the object detection output data can include any combination of bounding shapes (e.g., 2D or 3D bounding boxes) and / or associated object property data (e.g., object classifications, yaw, velocity, intent, etc.). In one example, the transformer model can generate output including an objectness head, a bounding box regression head, and a bounding box classification head, where the output can be associated with or rendered onto a top-down representation and / or image domain representation of the environment. In some examples, the transformer model can also use previous infrared and radar sensor data, and / or previous object detections or object tracks determined by the transformer model based on previous scenes in the environment, and can associate object detections in a current scene with previously detected objects and / or tracks from previous scenes.

[0021] Infrared data received and used by the infrared-radar fusion system can include long-wave infrared (LWIR) data, which can be an effective modality for detecting IR-emitting objects (e.g., animals, pedestrians, cyclists, or other small and / or moving IR-emitting objects) in low-light or low-visibility conditions. Accordingly, various examples can refer to LWIR sensors and data. However, in other examples, the infrared-radar fusion system can be configured to receive and detect objects based on any other infrared waveband(s), including (but not limited to) short-wavelength infrared, mid-wavelength long-wavelength infrared, far infrared, and / or near infrared wavebands. As noted above, the vehicle can also include multiple infrared sensors, which can include infrared sensors at different locations on the vehicle with different orientations and / or fields of view, as well as infrared sensors of different infrared wavebands.

[0022] Radar data received and used by the infrared-radar fusion system can include radar data detected (or captured) by one or more radar devices mounted on the vehicle or otherwise associated with the environment. In some examples, the radar devices can be configured to generate a plurality of individual radar points, where each radar point includes a plurality of radar data components (or parameters). Different radar data components of a radar point can include, for example, range (or distance to target) data values, elevation data values, azimuth data values, Doppler data values, etc. In some examples, a radar detector (or radar data analyzer) associated with the radar devices (and / or within the infrared-radar fusion system) can use a combination of the data components to determine clusters (e.g., subsets) of radar points that can be associated with the same object in the environment. For example, a subset of radar points that are within similar ranges of range values (distance to target), Doppler values, azimuth values, and elevation values can be associated with the same object and can be evaluated as a cluster to determine object classification, size, pose, trajectory, etc. The radar device aperture can include any number of antennas designed to transmit and / or receive radio waves. The antennas can be spaced apart and positioned within the aperture in a particular arrangement based on the intended use and / or purpose of the radar device. For example, certain radar devices can be designed to receive high-resolution representations of certain radar data parameters (or components), which can be achieved by positioning the transmit and receive antennas in a particular arrangement.

[0023] As shown in various examples herein, by using radar position and / or velocity data in conjunction with infrared data, an infrared-radar fusion system can more accurately detect and differentiate between different types of IR-emitting objects. For example, various machine learning models described herein (which can include transformer model(s) with attention and / or separate object detection models based on combined infrared-radar sensor data) can be trained to use combined infrared and radar data inputs to model various kinematic behaviors of different dynamic objects. For example, different types of animals (e.g., dogs, cats, birds, cows, horses, squirrels, etc.) can have different infrared signatures and / or kinematic behaviors, and ML models can be trained to detect and classify different types of individual animals and / or various types of groups of close animals. Similarly, different types of pedestrians (e.g., individual pedestrians, groups of pedestrians, stationary pedestrians, pedestrians walking or running, pedestrians on bicycles, scooters, skateboards, or other conveyances, etc.) can have different infrared signatures and / or kinematic behaviors, and ML models can be trained to detect and classify different types of pedestrian object detections.

[0024] In some examples, the infrared-radar fusion system can also be configured to track detected objects over time in an environment. For example, the infrared-radar fusion system can determine and store an infrared-radar signature associated with a detected object. The infrared-radar signature can include a combined set of infrared parameters and radar parameters associated with a particular instance of a detected object. Such parameters can include the size and / or shape of the object (e.g., based on infrared detections and / or radar clustering size / shape), the object intensity (e.g., based on infrared readings and / or radar return signals), and the object velocity (e.g., based on Doppler radar data). Additionally or alternatively, the infrared-radar signature can also include the object type / classification, location, yaw, etc. After determining and storing the infrared-radar signature of a detected object, the infrared-radar fusion system can use the signature to track the object through occlusions, crowds, and / or other ambiguous environments. In some instances, the infrared-radar fusion system can use a multi-target tracker implemented with a Kalman filter, a particle filter, or other Bayesian filter.

[0025] The techniques described herein for combining (or fusing) infrared and radar data to perform object detection in low-visibility regions can be used independently (e.g., as standalone object detection systems) or in conjunction with other sensor modalities and / or other object detection systems. For example, an infrared-radar fusion system can use these techniques to determine a first set of object detections (e.g., bounding shapes) that can be combined with other sets of object detections determined based on other sensor modalities (e.g., visible image data, lidar data, etc.). In various examples, object detections determined by the infrared-radar fusion system can be synchronized with other sets of object detections (e.g., outputs of other machine learning pipelines) at various stages in the object detection process. Early synchronization of infrared-radar object detections with object detections from other sensor modalities can refer to synchronization performed immediately after sets of object detections (e.g., bounding boxes) are generated by their respective pipelines. In contrast, late synchronization of infrared-radar object detections with object detections from other sensor modalities can refer to synchronization performed after one or more object segmentation stages, object classification, object tracking, and / or prediction. In these examples, such synchronization can be performed using transformer models and / or other networks using techniques similar or identical to those described in U.S. Patent Application No. 18 / 104,082, filed January 31, 2023, entitled “Object Detection and Tracking Using Machine Learning Transformer Models with Attention.”

[0026] In some examples, the infrared-radar fusion system can be used to combine infrared and radar data and generate object detections in all environments. However, in other examples, the infrared-radar fusion system can be conditionally executed and / or applied in low light or low visibility environments. For example, one or more components within the vehicle computing system of a vehicle can determine lighting and / or visibility conditions of a current driving environment. When a low light or low visibility environment is detected, the infrared-radar fusion system can be executed and used to determine object detections based on combined infrared-radar data. For example, a low light driving environment can include nighttime driving, driving in a tunnel or on a shaded road, and / or driving in foggy conditions. Additionally, low visibility areas within a driving environment can include shaded areas or areas obscured by steam, exhaust, etc. In these environments, the vehicle computing system can be configured to execute and rely on object detections from the infrared-radar fusion system. Additionally, in some instances, when a low light or low visibility condition is detected, object detections output by the infrared-radar fusion system can be used as a substitute (and / or can be weighted more heavily) for object detections determined based on other sensor modalities. For example, under low light or low visibility conditions, object detections output by the infrared-radar fusion system can be weighted more heavily or used in place of object detections based on visible imaging systems.

[0027] Conversely, when a high visibility environment is detected, in some examples, the vehicle computing system can be configured to bypass execution of the infrared-radar fusion system or to ignore (or weight less heavily) object detections output by the infrared-radar fusion system. In these examples, ignoring or weighting less heavily object detections from the infrared-radar fusion system can cause the perception components to rely more on object detection systems from other sensor modalities that perform well in high visibility environments. Additionally, hotter driving environments can result in a greater amount of heat being emitted from road surfaces, buildings, and other static objects. Because these environments can result in infrared detections of heat-emitting dynamic objects (e.g., pedestrians, animals, etc.) being difficult to distinguish from heat emissions of road surfaces or other static objects, the vehicle computing system can be configured to bypass the infrared-radar fusion system and / or to ignore object detections output by the infrared-radar fusion system.

[0028] In some examples, the infrared-radar fusion systems described herein can be included in (or initiated by) an auxiliary (or backup) perception system to use when the primary perception system cannot accurately detect, classify, and / or track objects in the environment. In such examples, a vehicle can include a primary perception system with one or a combination of object detection systems (e.g., machine learning object detection pipelines) in the object detection system, such as ML pipelines based on RGB visible image data, lidar data, radar data, and / or additional sensor modalities. For most objects and most driving environments, the primary perception system can accurately detect, segment, classify, and track objects with sufficient confidence. However, when the primary perception system cannot determine with sufficient confidence whether an object is present in a low-light environment (or a particular low-visibility region of the environment), the perception component can cause the infrared-radar fusion systems described herein to perform object detection within the low-visibility region as part of an auxiliary perception system. Similarly, when the primary perception system of a vehicle detects that an object is present in the environment but cannot classify the object as static or dynamic with sufficient confidence, or cannot accurately segment the object, or cannot track the object with sufficient confidence, etc., the perception component can invoke the infrared-radar fusion systems as an auxiliary perception system to confirm the object detection and classify, segment, and / or track the object detection.

[0029] The techniques described herein can be implemented in a number of ways. Example implementations are provided below with reference to the following figures. Although discussed in the context of autonomous vehicles, the methods, apparatus, and systems described herein can be applied to a variety of systems (e.g., sensor systems or robotic platforms) and are not limited to autonomous vehicles. In examples, the techniques described herein can be used in a driver-controlled vehicle, where such a system can provide an indication to the driver of the vehicle whether it is safe to perform various maneuvers. The techniques can be used in an aerial or nautical environment, or in any system involving objects or entities that can be associated with behaviors unknown to the system.

[0030] Figure 1An example system 100 and techniques are depicted for fusing infrared sensor data and radar sensor data, performing object detection and tracking based on combined sensor data, and controlling a vehicle 102 based on object detection from fused infrared-radar sensor data. In some instances, the vehicle 102 can be an autonomous vehicle configured to operate according to Level 5 classification issued by the Society of Automotive Engineers (SAE) (and / or the National Highway Traffic Safety Administration), which describes a vehicle capable of performing all safety-critical functions for the entire trip, with the driver (or occupant) not expected to control the vehicle at any time. However, in other examples, the vehicle 102 can be a fully or partially autonomous vehicle having any other level or classification. Additionally, the techniques described herein for combining (or fusing) infrared and radar sensor data, performing object detection and tracking, and controlling a vehicle can also be used by non-autonomous vehicles, including land vehicles, air vehicles, water vehicles, etc. The techniques discussed herein can also be applied to non-vehicle sensor-based systems, e.g., location detection and safety systems, manufacturing systems, augmented reality systems, etc.

[0031] As shown in this example, the vehicle 102 can receive sensor data from various sensor systems 104 of the vehicle 102. The sensor systems 104 can include any number of sensors, including sensors capable of sensing various different modalities of the environment proximate to the vehicle 102. Although this example shows an infrared-radar fusion system 106 configured to fuse infrared data 112 from infrared sensors 108 with radar data 114 from radar devices 110, the sensor systems 104 can also include various other sensor modalities / types. For example, the sensor systems 104 can also include image sensors (e.g., cameras), lidar sensors, time-of-flight sensors, environmental sensors, audio sensors, inertial sensors, sonar sensors, location sensors (e.g., GPS), and various other sensors configured to capture data representative of the external environment surrounding the vehicle 102. For the infrared sensors 108, radar devices 110, and / or other sensor modalities within the sensor systems 104, each sensor modality can include any number of individual sensors mounted on and / or integrated into the vehicle 102 in various different locations, angles, orientations, etc.

[0032] In some examples, the vehicle 102 can include separate clusters of sensors (e.g., sensor pods) positioned at each corner of the vehicle 102 and / or on the roof of the vehicle 102. Each cluster of sensors on the vehicle 102 can include one or more infrared sensors 108, one or more radar devices 110, and / or additional sensor devices / modalities. Additionally, each of the sensors within the sensor system 104 can be configured to have different (e.g., overlapping) fields of view and various different data capture characteristics (e.g., detection ranges, capture rates, focal lengths, focal points, color capabilities, etc.) in order to capture various different sensor data of the environment surrounding the vehicle 102. For example, the infrared sensors 108 can include LWIR sensors, short-wavelength wave infrared sensors, mid-wavelength infrared sensors, far-infrared sensors, and / or near-infrared sensors within infrared cameras, thermal imaging cameras, night vision cameras, etc. In some cases, the infrared sensors 108 can be configured to multi-spectral data across several discrete spectral bands (e.g., ultraviolet to visible light, visible light to infrared), and / or the hyperspectral sensor(s) can capture nearly continuous wavelengths across a wide range of the electromagnetic spectrum. Additionally, the radar devices 110 can include different types of radar systems (e.g., continuous wave, Doppler, monopulse, various resolutions and ranges, etc.), each of which can be configured to capture sensor data (e.g., images or point clouds) from the environment.

[0033] As shown in this example, the sensor system 104 including the infrared sensors 108 and the radar devices 110 can provide sensor data 116 to the infrared-radar fusion system 106. In some cases, the sensor data 116 including infrared sensor data, radar point cloud data, and / or any other sensor modalities can be referred to as a “sensor image.” The sensor image can include one or more values (e.g., pixel values, radar point feature data, etc.) of the sensor data at an associated location (e.g., pixel location, point in a radar point cloud, etc.) in the image. The sensor data image(s) can be two-dimensional (2D) or three-dimensional (3D) (e.g., including values along x-direction, y-direction, and z-direction in a 3D grid).

[0034] In some examples, the infrared-radar fusion system 106 can be configured to fuse a single infrared image with a single radar data image (e.g., a radar point cloud) representing the same or overlapping regions of the environment. For example, the infrared data 112 and the radar data 114 can represent the same regions of the environment relative to the vehicle 102. Additionally or alternatively, the infrared-radar fusion system 106 can fuse multiple infrared images with multiple radar data images. For example, the infrared data 112 (e.g., LWIR data) can include multiple infrared images representing infrared data captured by different infrared sensors 108, which can cover different regions of the environment, can have different wavelengths and / or spectral bands, and / or can be captured at slightly different points in time. Similarly, the radar data 114 (e.g., Doppler data) can include multiple radar point clouds representing radar data captured by different radar devices 110, which can cover different regions of the environment, can have different radar characteristics or different resolutions, and / or can be captured at different times.

[0035] The infrared-radar fusion system 106 can include an object detection component 118 and an object tracking component 120. The object detection component 118 can be configured to detect objects in the environment based on a combination of the infrared data 112 and the radar data 114. The object tracking component 120 can be configured to identify and track detected objects over a period of time, including tracking a particular object through occlusions, crowds, or other obscured environments. In some examples, the object detection component 118 and / or the object tracking component 120 can be included within a perception system of the vehicle 102, which can continuously detect, segment, classify, and track objects as the vehicle 102 traverses the environment.

[0036] In some examples, the object detection component 118 and / or the object tracking component 120 can combine the infrared data 112 and the radar data 114, and then execute a trained machine learning (ML) model to detect objects based on the combined sensor data. For example, a ML model within the object detection component 118 can be trained based on a view of a combination of an infrared data image and an overlapping set of radar points, and can output data indicating the object(s) detected within the combined sensor data view. In some examples, the ML model within the object detection component 118 can be configured to receive the combined infrared-radar sensor view and output a set of bounding shapes 122 (e.g., bounding boxes, bounding contours, or other polygonal data) as well as various object attributes (e.g., object type / classification, object yaw, object velocity, etc.). Such a ML model can also output probabilities and / or confidence scores associated with any or all of the detected objects (e.g., bounding shapes) and / or the attributes determined for the detected objects.

[0037] Prior to executing the object detection ML model, the object detection component 118 can generate one or more combined infrared-radar sensor views based on the infrared data 112 and the radar data 114. As described above, the object detection component 118 can use any number of sensor data fusion techniques, including projecting radar points from the radar data 114 onto an infrared image from the infrared data 112. The projection of radar points onto an infrared image can include aligning the infrared data 112 and the radar data 114 based on the relative position and / or orientation angles of the sensor devices, and then reprojecting (e.g., modifying) the radar points based on the alignment differences of the sensor devices.

[0038] In other examples, the object detection component 118 can include one or more ML transformer models trained to output object detections based on combined infrared and radar data embeddings. For example, a transformer model within the object detection component 118 can receive input embeddings based on the infrared data 112 and the radar data 114, use one or more attention mechanisms (or concatenations) to combine the embeddings, and then provide the embeddings to an attention head of the transformer model. Using an encoder-decoder architecture, the transformer model can be trained to output object detection data based on the input embeddings of the infrared and radar data. Like the other trained object detection ML models described herein, the ML transformer model can be trained to output a set of bounding shapes 122, including range data and various other object properties (e.g., object type / classification, object yaw, object velocity, etc.). The ML transformer model can also output probabilities and / or confidence scores associated with any or all of the detected objects (e.g., bounding shapes) and / or properties determined for the detected objects.

[0039] As described above, the ML models and / or ML transformers within the object detection component 118 can be trained to perform semantic classification (e.g., determine object types) for detected objects. Because the combined infrared and radar data can be particularly efficient for detecting objects that emit IR in low-visibility areas, the object detection component 118 can include models (e.g., object detection ML models and / or ML transformers) that are specifically designed and trained to detect and differentiate animals, pedestrians, cyclists, etc. within low-light driving conditions and / or other low-visibility areas. For example, the ML models used by the object detection component 118 can be trained to detect pedestrians based on the combined infrared-radar sensor data, including determining individual pedestrians and / or groups of pedestrians (e.g., including the number of pedestrians in a group), determining pedestrian classifications and / or different sub-classifications (e.g., children vs. adults, seated pedestrians vs. standing pedestrians, jogging pedestrians vs. running pedestrians, etc.), and / or classifications of pedestrians or cyclists that include riding conveyances (e.g., bicycles, scooters, skateboards, etc.) or riding animals. The ML models used by the object detection component 118 can also be trained to detect animals in low-visibility conditions, including determining animal classifications, animal type sub-classifications (e.g., dogs, cats, squirrels, birds, cows, horses, etc.), and the number of animals moving together in a group. In some examples, separate ML model(s) can be trained to recognize different object classes (e.g., pedestrians, animals, etc.) and / or different sub-classes within an object class (e.g., different animal types or groups, different groups or characteristics of pedestrians, etc.), where each ML model generates an output indicating the presence of objects of the respective object class (and / or sub-class), as well as a bounding shape 122 indicating the size / range and location of the objects within the environment, as well as any number of object attributes and / or probability or confidence scores. In other examples, the ML model(s) used by the object detection component 118 can output a probability score at each pixel location of the input(s) representing the probability that the pixel location corresponds to each object class in a set of object classes (e.g., pedestrians, animals, etc.).

[0040] Within the object detection component 118, one or more object classifier models can be configured to classify object detections based on a combination of infrared data (e.g., identifying a shape of emitted IR at a location within the environment) and radar point data for a subset of radar points corresponding to the shape of emitted IR. Each radar point projected onto (or otherwise associated with) an object of emitted IR can include range data, Doppler velocity data, azimuth data, and / or elevation data. By analyzing the data components of radar points projected onto an object of emitted IR, the object detection component 118 can use the models to classify objects, including distinguishing pedestrians from animals, distinguishing animals from different types of animals, distinguishing pedestrians from pedestrians of a riding conveyance, etc. In some examples, the object detection component 118 can use the Doppler velocity component of radar points on an object to classify the object as a dynamic object rather than a static object, and distinguish between different types of dynamic objects based on object velocity, etc. Additionally, in some cases, different radar points can be projected onto (e.g., reflected back from) different parts of a pedestrian or animal, including one or more radar points projected onto the body of the pedestrian / animal and other radar points projected onto an appendage of the pedestrian / animal (e.g., an arm or leg). In these cases, the object detection component 118 can compare the relative differences in Doppler velocity at different points on the object to classify the object as a pedestrian or a particular type of animal, and determine a current behavior of the object (e.g., standing, jogging, running, etc.). For example, a difference in Doppler velocity between a first point on a leg of a pedestrian or animal and a second point on the body (or between points on different legs / appendages of the object) can allow the object detection component 118 to determine that the pedestrian / animal is walking, jogging, or running, etc.

[0041] In some examples, the object detection component 118 can classify detected objects based on temporal infrared and / or radar data. For example, the location of a detected object can be tracked over time, and characteristics of the motion (e.g., velocity, yaw, direction, and uniformity of velocity, etc.) can be used to classify the object as a pedestrian, an animal, a rider, etc.

[0042] The object detection component 118's ability to classify object detections can be based on the quality and / or resolution of the infrared sensor 108, the radar device 110, and / or the object detection range from the vehicle 102. As an example, using a relatively low-resolution radar device 110, the object detection component 118 can apply classification models capable of distinguishing between pedestrians and large animals, as well as large, medium, and small animals. However, using radar data from the low-resolution radar device 110, the object detection component 118 may not be able to distinguish between different types of animals of similar size. As another example, when using a high-resolution radar device 110, the object detection component 118 can apply different classification models capable of distinguishing between different types of animals of similar size at close or medium range (e.g., dogs versus cats, rabbits versus squirrels, etc.). Using higher-resolution radar and / or closer-range object detection can also allow the object detection component 118 to determine the number of pedestrians or animals in close groups. These techniques can provide additional advantages for vehicle object prediction and trajectory planning. For example, distinguishing specific types of animals can allow the prediction and / or planner component 124 of vehicle 102 to determine more accurate predictions of animal movement based on the kinematic abilities (e.g., acceleration, change of direction, etc.) and / or behavioral differences of different types of animals. Similarly, determining the number of animals and / or pedestrians in a group and the specific behaviors of individuals in the group (e.g., walking, running, etc.) can allow the prediction and / or planner component 124 to predict the behavior of the group more accurately. More accurate predictions of object movement can also enable the prediction and / or planner component 124 to determine improved trajectories for vehicle 102 to safely and efficiently traverse the environment.

[0043] Object tracking component 120 may use the same (or similar) techniques used by object detection component 118 to identify and track specific detected objects within an environment over time. For example, object tracking component 120 may include an additional ML model trained to receive a set of object detections (e.g., including boundary shapes, attributes, and confidence scores) from object detection component 118 and to associate the object detections with a previous set of object detections detected during previous time points within the environment. In some examples, object tracking component 120 may store infrared-radar object features for one or more objects detected in the environment, including combined infrared and radar parameters. Infrared-radar features may be associated with object classification (or type) as a whole and / or with individual instances of the detected objects. Parameters stored within the infrared-radar features may include, for example, the x-dimension, y-dimension, and z-dimension of the object extent (e.g., based on the overlap of radar and infrared data), the infrared intensity of the object, and / or the Doppler object velocity. The infrared-radar feature data may also include object type / classification, location, yaw, and any other object detection data described herein. Using the infrared-radar features of detected objects, the object tracking component 120 can track objects through occlusion, crowds, and / or other obscured environments. In some instances, the infrared-radar fusion system can use a multi-target tracker implemented using Kalman filters, particle filters, or other Bayesian filters.

[0044] Object features used for object detection and / or object tracking can be based on any combination of infrared and radar data components described herein. For example, object tracking component 120 can determine the characteristics of an object based on the size, shape, and intensity of an infrared blob generated from a detected object, as well as range data and Doppler velocity data received from a corresponding radar point projected onto the object. In some examples, object features may also include data representing the relative motion of the object, i.e., the motion of certain parts of the object (e.g., arms, legs, body, head, etc.) relative to other parts of the object. As described herein, these object features can be used for object detection, object classification (e.g., distinguishing different types of animals, distinguishing groups of pedestrians or groups of animals, etc.), and / or object tracking. By including additional infrared and radar data features within the object feature data, these features can provide the infrared-radar fusion system 106 with improved capabilities for accurately detecting, classifying, and tracking specific objects in low-visibility environments. Furthermore, although some examples herein describe object features as including only infrared and radar data features, in other examples, infrared-radar object features may also include corresponding visible image components, lidar components, and / or data components from various other sensor modalities.

[0045] In some examples, learned embeddings can be used to generate infrared-radar object features for object tracking. For instance, the learned embedding can be configured to receive infrared image data (e.g., long-wave infrared patches) and the corresponding radar point cloud, and output a vector representing the infrared-radar features of the object. The object tracking component 120 can then compare this vector with other vectors representing other infrared-radar object features in a multidimensional vector space (e.g., using nearest neighbor search or other multidimensional proximity search). Such comparisons allow the object tracking component 120 to determine which infrared-radar object features correspond to the same object at different points in time in the environment.

[0046] The use of infrared-radar object features can offer numerous advantages when performing object tracking. In particular, object features that include the Doppler velocity component of radar data can improve the accuracy and efficiency of object tracking systems compared to tracking systems based on other sensor modalities that do not include a velocity component (e.g., RGB image data, lidar data, etc.). When object features used for object tracking include (or are based on) a Doppler velocity component, each individual sensor reading of the object can include a velocity component and thus indicate the object's likely location at a future time. Conversely, for systems where a velocity component is not included, the object tracking system may be required to capture multiple sensor readings of the object at different times, analyze the readings to determine distance differences, and then infer the object's movement / velocity based on these distance differences. Therefore, this technique, using object features based on combined infrared-radar data (including Doppler velocity data), enables the object tracking component 120 to track objects more quickly and accurately, especially in congested, complex, and / or low-visibility driving environments.

[0047] As illustrated in this example, various boundary shapes 122 determined by the infrared-radar fusion system 106 can be provided to additional components within the computing system of the vehicle 102 for controlling the vehicle's operation within the driving environment. For example, the boundary shapes 122 output from the infrared-radar fusion system 106 may include object detection, object position, object size, object shape, object classification, object yaw, etc. In some instances, these boundary shapes 122 can be sent to a prediction and / or planner component 124 within the computing system of the vehicle 102. The prediction component of the vehicle 102 can determine the predicted position, predicted speed, predicted trajectory, etc., of the detected object represented by the boundary shapes 122 based on object attributes (e.g., pedestrian or animal, pedestrian attributes, animal type, etc.) and / or based on other objects nearby in the environment. The planner component of the vehicle 102 can determine one or more actions to be performed to control the vehicle 102 based on the boundary shapes 122 (and / or additional inputs). For example, the predictor and / or planner component 124 may determine the predicted trajectory for objects detected in the environment, as well as the optimal trajectory (and / or driving maneuvers to be performed by the vehicle 102) to follow, based on the predicted trajectory associated with the boundary shape 122.

[0048] The techniques described herein can improve vehicle functionality and safety by providing enhanced detection of infrared-emitting objects in low-light driving conditions and other low-visibility environments. Techniques including fusing infrared and radar data, and using trained models to analyze the fused infrared-radar view to detect pedestrians, animals, and other IR-emitting objects, can improve the ability of autonomous vehicles to detect, classify, and track such objects in low-visibility environments. The ability to detect, classify, and track such objects can be critical to the overall safety and quality of autonomous driving. Furthermore, in addition to fused infrared-radar sensor data, the techniques disclosed herein can be used in conjunction with any combination of sensor modalities such as vision (e.g., imagery) and / or lidar. Using these techniques, objects can be detected and tracked with greater certainty regarding their position, size, speed, yaw, and classification in low-visibility environments. This is due to the ability to generate and analyze boundary shapes based on the fused infrared-radar sensor modalities using object detection models and / or in ML transformer models to determine boundary shapes with higher accuracy than those generated by individual sensor modalities or other combinations of sensor modalities.

[0049] Figure 2A and Figure 2BTwo examples of object detection using an infrared-radar fusion system 106 are shown. As described above, the techniques described herein for using combined infrared and radar data can provide specific advantages for object detection and tracking in low-light driving environments and other low-visibility areas. Therefore, Figure 2A and Figure 2B Examples of object detection in two different low-visibility driving environments are depicted.

[0050] Figure 2A A nighttime driving environment 200 is depicted, in which vehicle 102 can use an infrared-radar fusion system 106 to perform object detection based on a combination of infrared data 202 and radar data 208. In this example, infrared data 202 includes an infrared image comprising two patches, patch 204 and patch 206, each representing an object emitting IR within environment 200. As shown in this example, although infrared data 202 may include the general size and intensity of the emitting IR patches, these patches have relatively low resolution and cannot be used to determine accurate boundary shapes. Furthermore, the infrared data does not include position or velocity data associated with patches 204 and 206. However, radar data 208 for the corresponding area includes a first radar detection 210 (e.g., a first radar point cloud or a portion thereof) and a second radar detection 212 (e.g., a second radar point cloud or a portion thereof) corresponding to patches 204 and 206 in the infrared data. As described above, the infrared-radar fusion system 106 can be configured to determine a specific set of radar points within radar detections 210 and 212 by aligning radar data 208 with infrared data 202 and projecting radar points onto infrared patches 204 and 206. Based on the combination of infrared data within patches 204 and 206 with the corresponding radar detections 210 and 212, the infrared-radar fusion system 106 can obtain range / size data, position data, and velocity data for various locations (e.g., pixels) within each of the emitted IR patches 204 and 206.

[0051] By fusing infrared-radar data from these locations in the environment, the infrared-radar fusion system 106 can use ML models (and / or other techniques) to analyze the combined sensor data view 214 to determine that IR patches 204 and 206 represent actual heat-emitting dynamic objects (e.g., pedestrians and animals), thereby distinguishing those objects from statically emitting IR objects (e.g., exhaust vents, engines, or metallic objects retaining heat from the previous day). Additionally, the ML models used by the infrared-radar fusion system 106 can execute ML object detection models and / or ML transformer models to generate object detections 216 and 218. Based on the combined infrared-radar data, object detections 216 and 218 can include 2D or 3D range data (e.g., size data) and defined locations within the environment. Object detections 216 and 218 can also include any combination of object attributes described herein, such as object classification, object shape characteristics, object velocity, object yaw, etc. As shown in this example, object detection 216 could represent a pedestrian, and object detection 218 could represent a medium-sized animal near the pedestrian. When object detections 216 and 218 are provided to prediction / planning components (or other downstream sensor data analysis components), these downstream components can determine that object detections 216 and 218 correspond to a dog walker and can perform prediction operations to determine the trajectory of vehicle 102 accordingly.

[0052] Figure 2B A daytime driving environment 220, including an underpass, is depicted, which creates low-visibility shadow areas that may hinder object detection using an RGB camera. As in the previous example, vehicle 102 can use an infrared-radar fusion system 106 to perform object detection within these low-visibility shadow areas based on a combination of infrared data 222 and radar data 226. In this example, infrared data 222 includes an infrared image comprising patches 224 representing objects (or multiple objects) emitting IR within environment 220. Because patches 224 are relatively low-resolution and do not include position data, velocity data, etc., they cannot be used alone for accurate object detection. However, radar data 226 for the corresponding area includes radar detection 228 (e.g., a radar point cloud or a portion thereof) corresponding to infrared patch 224. As described above, infrared-radar fusion system 106 can be configured to determine specific radar points within radar detection 228 by aligning radar data 226 with infrared data 222 and projecting radar points onto infrared patch 224. Based on the combination of infrared data within patch 224 and the corresponding radar detection 228, infrared-radar fusion system 106 can obtain range / size data, position data, and velocity data for various locations (e.g., pixels) within patch 224 that emits IR.

[0053] As in the previous example, the infrared-radar fusion system 106 can fuse infrared and radar data into a combined sensor data view 230, and can use ML models (and / or other techniques) to analyze the combined sensor data view to determine that IR patches 224 represent actual heat-generating dynamic objects (e.g., one or more pedestrians and / or animals), rather than static objects emitting IR. ML models (e.g., ML object detection models and / or ML transformer models) used by object detection component 118 and / or object tracking component 120 can determine object detections 232, 234, and 236. Object detections 232-236 can include 2D or 3D boundary shapes with range / size data, location data, and / or various other object attributes (e.g., classification, velocity, yaw, etc.). As shown in this example, object detections 232-236 can represent closely spaced groups of pedestrians. When object detections 232-236 are provided to prediction / planning components (or other downstream sensor data analysis components), these downstream components can determine that the object detections correspond to a group of people waiting to cross the road at a pedestrian crossing, and can perform prediction operations and determine the trajectory of vehicle 102 accordingly.

[0054] Figure 3 An example system 300 is shown, comprising an infrared-radar fusion system 106 operating within a sensing component configured to perform object detection and tracking based on multimodal sensor data. In this example, system 300 may be associated with a sensing component operating within a vehicle 102, which includes various infrared sensors 302, radar sensors 304, RGB sensors 306, and lidar sensors 308 (and / or additional sensor types / modalities). The sensors of system 300 may capture infrared data 310, radar data 312, image data 314, and lidar data 316 (and / or additional sensor data types / modalities) and provide them to the object detection, segmentation, classification, and tracking components of the sensing component.

[0055] As described above, the sensing component may include multiple object detection systems configured to operate independently and use different sensor modalities to generate object detections. In this example, the infrared-radar fusion system 106 receives infrared data 310 and radar data 312, fuses the infrared and radar data into a combined sensor data view, and generates object detections based on the combined sensor data. As described above, the object detection component 118 of the infrared-radar fusion system 106 may include various sub-components configured to perform various functions. The overlay / projection component 320 may be configured to fuse infrared and radar sensor data using various sensor data fusion techniques (e.g., overlaying sensor images and / or projecting radar points onto an infrared image). Additional techniques performed by the overlay / projection component 320 may include determining overlapping / corresponding sensor data, and spatially and / or temporally aligning the infrared data 310 and radar data 312 based on the relative position orientation angle of the sensor devices and / or the time the sensor data was acquired.

[0056] The object detection model 322 within the object detection component 118 may include a trained ML model for detecting objects and outputting object detection data (e.g., boundary shape and associated object attributes) based on combined infrared-radar sensor data. As described above, the object detection model 322 can be trained to determine the object range and location of static and dynamic objects emitting IR within the environment. The object detection model 322 can also be trained to semantically classify objects into type / subtype categories (e.g., pedestrians, animals, cyclists, etc.) and determine object attributes (e.g., yaw, speed, etc.), which can be used by downstream components to predict object behavior and for planning vehicle trajectories and driving maneuvers.

[0057] In various examples, the object detection model 322 may include convolutional neural networks (CNNs), recurrent neural networks (RNNs), graph neural networks (GNNs), fully convolutional networks (FCNs), etc. However, the object detection model 322 described herein may also refer to a statistical model, a heuristic model, a rule-based model, or a combination thereof. For example, the object detection model 322 may include a statistical model that generates an approximate and / or mathematical function that can be used to determine the probability of the presence and / or classification of one or more objects. In other words, the object detection model 322 may refer to any computational model that learns from training data to improve the accuracy of its output (e.g., object detection and / or classification).

[0058] Alternatively or concurrently, the object detection component 118 may include one or more ML transformer models 324 trained to output object detection based on combined infrared and radar data embeddings. In some examples, the ML transformer model 324 may generate input embeddings based on infrared data 310 and radar data 312, and use an attention mechanism (or concatenation) to combine the embeddings. The ML transformer model 324 may provide the embeddings to the attention head of the ML transformer model. As described above, the ML transformer model 324 may use an encoder-decoder architecture trained to output object detection based on the input infrared and radar embeddings. As described above with respect to object detection model 322, one or more ML transformer models 324 may also be trained to output a set of boundary shapes, classifications, and / or other object attributes.

[0059] In this example, together with the infrared-radar fusion system 106, system 300 may include an additional object detection system that operates independently based on other sensor modalities. For example, image object detection pipeline 328 may be an ML pipeline trained to perform ML-based object detection based on RGB (visible) image data 314. Additionally, lidar object detection pipeline 330 may be a separate ML pipeline trained to perform ML-based object detection based on lidar data 316. Each of the infrared-radar fusion system 106, image object detection pipeline 328, and lidar object detection pipeline 330 can operate independently based on its respective sensor modality and can output separate sets of time-synchronized object detections. For example, infrared-radar fusion system 106 may output boundary shapes 332 (including size, location data, object attributes, confidence scores, etc.), image object detection pipeline 328 may output a separate set 334 of boundary shapes, and lidar object detection pipeline 330 may output another separate set 336 of boundary shapes.

[0060] The object detection synchronization component 338 can receive boundary shapes 332-336 from various independent object detection systems and can combine object detections into a synchronized set of object detections to provide to the downstream prediction and planning components of the vehicle 102. For example, in some cases, two or more of different object detection systems (e.g., infrared-radar fusion system 106, image object detection pipeline 328, lidar object detection pipeline 330, etc.) can output different object detections corresponding to the same real-world objects in the physical environment. Additionally, due to potential errors or inaccuracies in modality-specific ML pipelines (e.g., over- or under-segmentation, misclassification, etc.), one or more object detection systems can output multiple object detections based on a single perceived object, and / or output a single object detection based on multiple perceived real-world objects. One or more modality-specific ML pipelines may also output inaccurate classifications or other attributes for some object detections.

[0061] In various examples, the object detection synchronization component 338 may use heuristic-based algorithms and / or additional trained ML models to determine the optimal set of accurate object detections based on separate sets of object detections received from different object detection systems. Such heuristics and / or ML models may take into account probabilistic data and / or confidence scores associated with boundary shapes 332-336 received from various object detection systems, and / or may be configured or trained to account for accuracy differences between different sensors, sensor modalities, and object detection systems in different environments.

[0062] For example, in this example, the object detection synchronization component 338 includes a visibility component 340 and an environmental data component 342. As described above, the infrared-radar fusion system 106 may be relatively more efficient in determining object detection in low-light or other low-visibility environments. Conversely, when determining object detection in high-visibility environments, the image object detection pipeline 328 may perform as well or better than the infrared-radar fusion system 106. Therefore, the object detection synchronization component 338 can be configured to depend differently on different sets 332-336 of boundary shapes depending on the visibility metric associated with object detection (e.g., determined by the visibility component 340) and / or the current environmental conditions (e.g., determined by the environmental data component 342). For example, when the visibility component 340 determines a low-visibility area of ​​the environment and / or when the environmental data component 342 determines low-light or low-visibility driving conditions, the object detection synchronization component 338 may increase the weight of the boundary shape 332 determined by the infrared-radar fusion system 106 or depend more on that boundary shape 332. Conversely, when the visibility component 340 determines high-visibility areas of the environment and / or when the environmental data component 342 determines daytime driving and high-visibility driving conditions, the object detection synchronization component 338 may reduce the weight of the boundary shape 332 determined by the infrared-radar fusion system 106 or rely less on the boundary shape 332. In these examples, to determine low-visibility driving environments and / or low-visibility areas within those environments, the visibility component 340 may apply one or more visibility thresholds based on various sensor data 302-308 and / or other visibility data received by the vehicle system.

[0063] For example, visibility component 340 can determine a visibility metric (e.g., brightness level) of the driving environment (and / or a specific part of the environment) based on light data received from various sensors (e.g., RGB visible light sensors, infrared sensors, etc.). Alternatively or additionally, visibility component 340 can determine the visibility metric of the driving environment based on the time of day and / or current environmental / weather conditions (e.g., sunny, rainy, foggy, snowy, etc.). In some cases, the visibility metric is also based at least in part on map data. For example, tunnels, underpasses, or narrow streets lined with tall trees or buildings (which may create shadows at certain times of day) identified based on map data may correspond to low visibility conditions, while the absence of such map features may correspond to high visibility conditions. In various examples, visibility component 340 can use any combination of these techniques and various other techniques described herein to determine visibility metrics corresponding to low-visibility and high-visibility driving environments. Any of these techniques can be applied, alone or in combination, to the entire driving environment and / or individual areas within the driving environment. For example, the first region within the environment can be a high-visibility region (e.g., based on light data captured from the first region, time of day, clear weather conditions, etc.), and the second region within the same environment at the same time can be a low-visibility region (e.g., based on light data captured from the second region, detection of tunnels or shadow areas, etc.).

[0064] The trajectory update component 344 can receive a set of object detections from the object detection synchronization component 338 and can determine new / updated trajectories for the detected objects. As shown in this example, the trajectory update component 344 can use object detections 346 received from the object detection synchronization component 338 (and / or from a separate object detection system) and previous object trajectories 348. Previous object trajectories 348 may include object trajectory data of objects previously detected and / or tracked by the system 300 in the environment at previous times. For example, the trajectory update component 344 can retrieve a set of previous object detections that may be similar or identical in type / format to any of the object detections described herein, but may be based on previous sensor data of the environment captured at previous time steps. For example, previous object detections may include object detections based on previous sensor data (e.g., previously captured image frames, LiDAR point clouds, radar point clouds, etc.) for any number of previous times / time steps (e.g., input / output processing cycles, etc.) of the sensing component.

[0065] To determine a new / updated trajectory, trajectory update component 344 can analyze object detection 346 relative to the previous object trajectory 348 for the current driving scenario. In some cases, trajectory update component 344 can use attributes of infrared-radar features associated with various object detections, determined by infrared-radar fusion system 106, to associate the current object detection with the previous object trajectory. Trajectory update component 344 can also use integrated box and contour estimation algorithms to determine optimal, accurate, and consistent object trajectory data based on a combination of previous and current object data. In some instances, trajectory update component 344 can independently determine trajectory updates for different parts of the object (e.g., the front and rear of a vehicle).

[0066] The output of trajectory update component 344 may include perceived object data and trajectory data for any number of objects detected in the current scene within the environment. The object detection and associated trajectory data can be used by any number of downstream processing components for controlling vehicle 102, including prediction component 350 and / or planner component 352. Prediction component 350 includes functionality for generating predicted object trajectories and future states in the environment, and planner component 352 is configured to determine the path that vehicle 102 should follow as it traverses the environment.

[0067] Figure 4 An example block diagram of example technique 400 is shown, which is used to generate training data to train an ML model for detecting objects based on combined infrared-radar sensor data. The techniques described in example 400 can be performed, for example, by one or more remote computing systems (e.g., computing systems(s) separate from vehicle 102). For example, the remote computing system(s) can implement training component 402 to train ML model 404 (which may be similar to or identical to the object detection model 322 and / or ML transformer model 324 discussed above) using training data 406. In some examples, ML model 404 may include a library of ML models (e.g., independent models, each trained to detect objects emitting a specific IR in low-visibility environments). In other examples, ML model 404 may be trained to output one of a set of object category labels (including outputs of "none" or "background") and / or additional object attributes, as well as confidence scores for each category label and / or attribute.

[0068] In this example, training data 406 may include input data labeled with expected output values. These expected output values ​​can often be referred to as “ground truths.” For example, ground truths may include the identification of a specific object in the input data, along with semantic classifications and / or category labels associated with the object (e.g., identifying and labeling the object as a specific type of animal, pedestrian, cyclist, etc.). The accuracy of ML model 404 may be based on the amount and / or accuracy of the data provided in training data 406. Therefore, a suitable dataset for training ML model 404 may include a large amount (e.g., hundreds of thousands) of input data with known or predetermined regions or segments that correspond to objects of interest depicted in various relevant low-light or low-visibility backgrounds (e.g., nighttime driving scenarios, foggy or rainy driving scenarios, driving scenarios with shadows or dark areas in the environment, etc.).

[0069] In the example, training data 406 may include labeled combination of infrared-radar data 420 based on infrared data 414 and radar data 416. Training data 406 may include annotations based on annotator component 418.

[0070] In various examples, the annotator component 418 may include a user interface configured to receive manual annotations of objects of interest from a human operator. For example, in manual annotation techniques, a user interface may be provided depicting a combined view 408 of sensor data based on input data from a single sensor (e.g., an RGB image sensor) or a combination of multiple sensors associated with vehicle 102. The human operator may be instructed to identify objects of interest (e.g., animals, pedestrians, etc.) and annotate them by indicating regions or segments of data corresponding to that object, providing category labels. However, complete manual identification and labeling of objects can be a time-consuming, inefficient, and error-prone technique for generating training data. Furthermore, the amount of manually annotated training data may be limited due to the time and effort required to generate such annotations, and therefore may be insufficient for training an ML model.

[0071] Therefore, alternatively or additionally, the annotator component 418 can be implemented as an automated component on one or more remote computing systems configured to annotate input data with indications of objects of interest via one or more algorithms configured to segment, detect, classify, and / or label objects in ground truth data of infrared data 414, radar 416, and / or other sensor modalities. However, in the example, any combination of manual and automatic annotation can be used to generate training data 406. In some examples, the annotator component 418 can be implemented on one or more computing systems separate from the remote computing systems implementing training component 402.

[0072] In an example where infrared data 414 and radar data 416 are captured by sensors located on the autonomous vehicle (e.g., an infrared sensor capturing infrared data 414 and a radar device capturing radar data 416), training component 402 can align infrared data 414 and radar data 416 in a global reference frame or a reference frame relative to the vehicle based on known geometric relationships between the positions of the sensors on the vehicle, the angles and types of the sensors, and / or external and internal parameters associated with the sensors. In an example where geometric relationships between the sensors are not available, training component 402 can determine the alignment using features of infrared data 414 and radar data 416. In some examples, training component 402 can implement computer vision techniques for image registration. As a non-limiting example, training component 402 can use SIFT-based or SURF-based techniques to detect keypoints in infrared data 414 and match them with corresponding keypoints in radar data 416 to determine the alignment of the sensor data such that the matched keypoints overlap after alignment. In some cases, training component 402 can determine transformations such as homography, thereby associating planar surfaces in infrared data 414 (e.g., building exteriors, road surfaces, etc.) with corresponding planar surfaces in radar data 416 to align the combined sensor data view 408. Proper alignment can be verified by confirming that other permanent structures are located in overlapping areas within the combined sensor data view 408.

[0073] In the example, training component 402 can, after alignment, transfer annotations (e.g., bounding boxes, contours, fragments, pixel-based labels, etc.) and object classification information indicated in the annotations from annotator component 418 to corresponding locations in infrared data 414 and / or radar data 416. In the example shown, object 410 represents a pedestrian, and object 412 represents an animal (or more specifically, a cat), and can be labeled accordingly via annotator component 418. These annotations (e.g., including bounding boxes, classification labels, etc.) can be transferred to corresponding locations / patches in infrared data 414 and to corresponding areas (e.g., radar points) in radar data 416.

[0074] In some examples, training component 402 can further augment the data points in training data 406 by applying data augmentation techniques to labeled infrared data 414 and / or labeled radar data 416. For example, the data points in labeled infrared data 414 and / or labeled radar data 416 can be slightly modified (e.g., by adding small rotations, scaling, shifts, mirroring, etc.) and added to training data 406 along with corresponding modifications to the locations(one or more) in the labels.

[0075] Training component 402 can train ML model 404 based on training data 406, which includes labeled combined infrared-radar sensor data 420 to indicate the expected output for each input data point. To produce an effective output, a machine learning model such as ML model 404 (e.g., CNN, RNN, GNN, FCN, etc.) must first learn a set of parameters, or be "trained". Training is achieved by iteratively adjusting the parameter set of ML model 404 using learning techniques such as backpropagation to make the output of ML model 404 closer to the expected output. During training, labeled combined infrared-radar sensor data 420 can be provided as input to ML model 404, and the corresponding labels included in training data 406 are used as the target output, allowing ML model 404 to learn the appearance associated with objects in the input data of one or more modalities used in training data 406. As understood in the art, in some ML architectures, the ML model only outputs a global class label and probability or confidence score. In such an example, a portion of the labeled combined infrared-radar sensor data 420 (e.g., a portion corresponding to a segment already identified as an object of interest in the annotations) can be used for training instead of data depicting the entire scene. However, in other non-limiting examples, the ML model 404 may include a fully convolutional network (FCN) that returns an output the same size as the input data, where the output at coordinates (x, y) includes one or more class labels and one or more pixel-level confidence scores (or class label probabilities) associated with the pixel at coordinates (x, y) in the input data, thereby locating one or more detected objects within the input data.

[0076] In some examples, the ML model 404 can be a pre-trained model (e.g., pre-trained using a publicly available dataset such as ImageNet), and the training component 402 can implement a self-supervised learning method, where object detection in low-visibility driving scenarios can correspond to a downstream task for the pre-trained model. Self-supervised learning methods do not require a large training dataset; therefore, the ML model 404 can be trained using training data 406 containing a few instances of labeled data, according to a few-shot learning method as described herein. In some examples, when the training component 402 uses self-supervised learning to train the ML model 404, the training data 406 does not need to contain labels (e.g., training data 406 may include infrared data 414 and radar data 416, but does not need to be labeled).

[0077] As discussed, using sensor data with combined infrared and radar modalities to detect and classify objects can improve the robustness and accuracy of object detection, segmentation, and classification. In particular, the fusion of infrared and radar data described herein can provide specific advantages for detecting and tracking IR-emitting objects (e.g., animals, pedestrians, etc.) in low-visibility driving environments (e.g., night driving or other low-light driving, fog or rain conditions) or in tunnels or shadowy regions where objects may not be accurately detected, segmented, and classified by individual sensor modalities. As discussed herein, some objects may be more distinguishable in data from specific modalities (e.g., living or warm objects may be easily detected in infrared images, some materials may have unique characteristics in specific spectral bands and are therefore easier to detect in data captured in those spectral bands, etc.). In such an example, ML model 404 may include a library of ML models, each trained to detect different categories of IR-emitting objects and / or to perform detection in different low-visibility environments.

[0078] In real-world scenarios, combined (and aligned) views of infrared-radar sensor data may not be readily available. For example, for a given scene and / or a given object, data for only one modality may be available. In another example, training data may be insufficient for rare occurrences (e.g., data depicting a specific type of animal crossing a street, pedestrians moving in dark shadow areas, etc.). To address the lack of training data availability, training data 406 may include synthetic data (e.g., computer-generated) or augmented data (e.g., data from a real-world scene modified to add objects of interest).

[0079] Figure 5A and Figure 5B Two example architectures of machine learning transformer models that can be used to implement the infrared-radar fusion system 106 are depicted. As described above, in some examples, the infrared-radar fusion system 106 may include one or more machine learning transformer models (and / or other deep neural networks) trained to output a combined view of sensor data (e.g., bounding boxes or other object detection) based on the corresponding infrared and radar input data.

[0080] In this example, Figure 5AA first example transformer 500 is depicted, configured to receive an input set of infrared and radar sensor data and output an object detection set based on the fused / combined infrared and radar input data. As shown in this example, transformer 500 may include an encoder / decoder architecture with self-attention. In this example, encoder 502 may receive a first set 506 of input embeddings based on infrared sensor data and a second set 508 of input embeddings based on radar sensor data as input. Encoder 502 may also receive previous infrared and / or radar sensor data (e.g., embedded or raw sensor data) representing the driving environment at previous time points. In this example, encoder 502 also includes a multi-head self-attention layer 512, a subsequent addition and normalization layer 514, a feedforward network 516, and another addition and normalization layer 518. In this example, decoder 504 receives the output from encoder 502 and an object query 520 (which may include, for example, an object detection query and / or an object trajectory query). In this example, decoder 504 includes a first multi-head self-attention layer 522, a subsequent addition and normalization layer 524, a second multi-head self-attention layer 526, another addition and normalization layer 528, a feedforward network 530, and another addition and normalization layer 532. The output of decoder 504 can be provided to one or more downstream processing components, including an additional feedforward network 534 configured to determine and output combined object detection (e.g., boundary shape 536) and / or a separate feedforward network 538 configured to determine and output classification data 540 associated with boundary shape 536.

[0081] In addition to using or instead of using previous object detection, in some examples, the infrared-radar fusion system 106 can retrieve and use trajectory data for any number of objects previously detected and tracked by the perception components. Trajectory data may include, for example, data defining driving paths and / or trajectories for dynamic objects in the environment. For example, trajectory data for an object may be defined as a combination of curvature (or yaw) and velocity (or acceleration) of a sequence of one or more time points or locations in the environment. Alternatively or additionally, trajectory data may be defined by a sequence of vehicle states and / or attitudes and / or continuous driving paths with velocity, acceleration, etc. The infrared-radar fusion system 106 can use trajectory data for any previously detected object to project (or otherwise predict) the position and state of the previously detected object at a time corresponding to a set of object detections received from one or more object detection generators.

[0082] Figure 5BA second example transformer 542 is depicted, configured to receive an input set of embeddings based on infrared sensor data and radar sensor data, and output a set of combined / fused object detections (e.g., boundary shapes). Like example transformer 500, transformer 542 also includes an encoder / decoder architecture with self-attention. In this example, ML transformer model 544 is configured to receive a vector 552 of input sensor data embeddings and transform the set 552 of input sensor data embeddings into the output of object detection 560 using an encoder / decoder architecture (e.g., encoder 554, feature 556, and decoder 558). Figure 5B The diagram also illustrates three attention-based ensemble operations: a multi-head attention block (MAB) 546 comprising a multi-head self-attention layer 562 and a feedforward network 564; an ensemble attention block (SAB) 548 comprising a multi-head self-attention layer 566 and a feedforward network 568; and a sensing ensemble attention block (ISAB) 550 comprising a first multi-head self-attention layer 570 and a second multi-head self-attention layer 572.

[0083] Figure 6 A block diagram depicts an example system 600 for implementing the various techniques described herein. In some instances, the example system 600 may include a vehicle 602, which may represent the above-described system. Figures 1-6 The vehicle 602 discussed herein is an example. In some instances, vehicle 602 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration (NHTSA), which describes vehicles capable of performing all safety-critical functions throughout the journey, where the driver (or occupant) is not expected to control the vehicle at any time. However, in other examples, vehicle 602 may be a fully or partially autonomous vehicle with any other level or classification. Furthermore, in some instances, the techniques described herein may also be used by non-autonomous vehicles. These are merely examples, and the systems and methods described herein can be incorporated into any land, air, or water vehicle, ranging from those requiring constant manual control by a driver to those with partial or full autonomous control.

[0084] Vehicle 602 can be configured to perform various techniques described herein, including fusing infrared sensor data and radar sensor data for object detection and tracking, which can be deployed within vehicle 602 when operating in a driving environment. In some examples, vehicle 602 can also be configured to provide log data to one or more separate computing devices 636 configured to train ML transformer models as described herein. Alternatively or additionally, vehicle 602 can receive and execute one or more ML object detection models and / or ML transformer models from separate computing devices 636, which can be integrated within a perception component configured to detect and track objects in the environment based on fused infrared-radar sensor data using various techniques described herein.

[0085] Vehicle 602 may include one or more vehicle computing devices 604, one or more sensors 606, one or more transmitters 608, one or more network interfaces 610, at least one direct connection 612 (e.g., for physical coupling with the vehicle to exchange data and / or provide power), and one or more drive systems 614. In this example, vehicle 602 may correspond to vehicle 102 discussed above. System 600 may additionally or alternatively include one or more computing devices 604.

[0086] In some instances, one or more sensors 606 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compasses), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, magnetometers, gyroscopes), image sensors (e.g., red-green-blue (RGB), infrared (IR), intensity, depth, time-of-flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors), etc. One or more sensors 606 may include multiple instances of each of these or other types of sensors. For example, radar sensors may include individual radar sensors located at corners, front, rear, sides, and / or top of vehicle 602. As another example, cameras may include multiple cameras positioned at various locations around the exterior and / or interior of vehicle 602. One or more sensors 606 may provide input to one or more vehicle computing devices 604 and / or one or more computing devices 636.

[0087] Vehicle 602 may also include one or more transmitters 608 for emitting light and / or sound, as described above. In this example, transmitters 608 may include one or more internal audio and visual transmitters for communicating with passengers of vehicle 602. By way of example, and not limitation, internal transmitters may include speakers, lights, signs, displays, touchscreens, one or more haptic transmitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners, etc.), etc. In this example, transmitters 608 may also include one or more external transmitters. By way of example, and not limitation, the external transmitters in this example include lights or other indicators (e.g., indicator lights, signs, light arrays) for signaling the direction of travel, and one or more audio transmitters (e.g., speakers, speaker arrays, horns) for audible communication with pedestrians or other nearby vehicles, one or more of which include beam steering technology.

[0088] Vehicle 602 may also include one or more network interfaces 610 that enable communication between vehicle 602 and one or more other local or remote computing devices. For example, network interfaces 610 may facilitate communication with one or more other local computing devices and / or one or more drive systems 614 on vehicle 602. Furthermore, network interfaces 610 may additionally or alternatively allow the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Network interfaces 610 may additionally or alternatively enable vehicle 602 to communicate with one or more computing devices 636. In some examples, computing devices 636 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0089] One or more network interfaces 610 may include physical and / or logical interfaces for connecting one or more vehicle computing devices 604 to another computing device or network (e.g., one or more networks 634). For example, network interfaces 610 may enable Wi-Fi-based communication, such as via frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies (e.g., Bluetooth®), cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that enables the respective computing device to engage with one or more other computing devices. In some instances, one or more vehicle computing devices 604 and / or one or more sensors 606 may transmit sensor data to one or more computing devices 636 via one or more networks 634 at a specific frequency, after a predetermined time period, or in near real-time.

[0090] In some instances, vehicle 602 may include one or more drive systems 614 (or drive components). In some instances, vehicle 602 may have a single drive system 614. In some instances, drive system(s) 614 may include one or more sensors to detect the conditions of drive system(s) 614 and / or the surrounding environment of vehicle 602. By way of example and not limitation, the sensors(s) of drive system(s) 614 may include: one or more wheel encoders (e.g., rotary encoders) to sense the rotation of the wheels of the drive component; inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) to measure the orientation and acceleration of the drive component; cameras or other image sensors; ultrasonic sensors to acoustically detect objects in the environment surrounding the drive component; lidar sensors; radar sensors, etc. For drive system(s) 614, some sensors such as wheel encoders may be unique. In some cases, sensors(s) on drive system(s) 614 may overlap with or complement corresponding systems (e.g., sensors(s) 606) of vehicle 602.

[0091] One or more drive systems 614 may include a number of vehicle systems within the vehicle system, including: a high-voltage battery, an electric motor propelling the vehicle, an inverter converting direct current from the battery into alternating current for use by other vehicle systems, a steering system including a steering motor and a steering frame (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system for distributing braking force to mitigate traction loss and maintain control, an HVAC system, lighting (e.g., headlights / taillights for illuminating the exterior environment of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high-voltage junctions, high-voltage cables, charging systems, charging ports, etc.). Additionally, one or more drive systems 614 may include a drive component controller that can receive and preprocess data from one or more sensors and control the operation of various vehicle systems. In some instances, the drive component controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components to perform various functions of one or more drive systems 614. In addition, drive system 614 includes one or more communication connections that enable the respective drive components to communicate with one or more other local or remote computing devices.

[0092] One or more vehicle computing devices 604 may include one or more processors 616 and a memory 618 communicatively coupled to the one or more processors 616. One or more computing devices 636 may also include one or more processors 638 and / or a memory 640. As described above, the memory 640 of the one or more computing devices 636 may store and execute training components 402 (e.g., referenced above). Figure 4 (as described herein), and can be configured to perform any combination of training functions for the ML object detection model and / or ML transformer model described herein.

[0093] The processors 616 and / or 638 may be any suitable processor capable of executing instructions to process data and perform the operations described herein. By way of example and not limitation, the processors 616 and / or 638 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application-specific integrated circuits (ASICs)), gate arrays (e.g., field-programmable gate arrays (FPGAs)), and / or any other device or part of a device that processes electronic data to convert that electronic data into other electronic data that may be stored in registers and / or memory.

[0094] Memory 618 and / or 640 may be examples of non-transitory computer-readable media. Memory 618 and / or 640 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods described herein and the functions belonging to various systems. In various implementations, the memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and various elements described herein may include many other logical, program, and physical components, wherein those shown in the figures are merely examples relevant to the discussion herein.

[0095] In some instances, memory 618 and / or memory 640 may store positioning component 620, sensing component 622, map 624, one or more system controllers 626, prediction component 628, and / or planning component 630. Sensing component 622 may include one or more infrared-radar fusion systems 106 configured to determine object detection based on combined infrared-radar sensor data, and / or object detection synchronization components 338 configured to determine object detection associations and combinations based on various sensor modalities, as described in the examples herein.

[0096] In at least one example, the localization component 620 may include hardware and / or software for receiving data from one or more sensors 606 to determine the position, velocity, and / or orientation (e.g., one or more of x-position, y-position, z-position, roll, pitch, or yaw) of the vehicle 602. For example, the localization component 620 may include one or more maps of the environment and may continuously determine the position, velocity, and / or orientation of the autonomous vehicle within one or more maps. In some instances, the localization component 620 may utilize SLAM (Simultaneous Localization and Mapping), CLAMS (Simultaneous Calibration, Localization, and Mapping), relative SLAM, beamforming, nonlinear least squares optimization, etc., to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position, attitude, and / or velocity of the autonomous vehicle. In some instances, the localization component 620 may provide data to various components of the vehicle 602 to determine the initial position of the autonomous vehicle for generating trajectories and / or for generating map data, as discussed herein. In some examples, the positioning component 620 may provide the planning component 630 and / or the prediction component 628 with the position and / or orientation of the vehicle 602 relative to the environment and / or the associated sensor data.

[0097] The memory 618 may also include one or more maps 624, which can be used by the vehicle 602 for navigation within the environment. For the purposes of this discussion, the map can be any number of data structures modeled in two, three, or N dimensions, capable of providing information about the environment, such as, but not limited to, topology (e.g., intersections), streets, mountains, roads, terrain, and the general environment. In one example, the map may include a three-dimensional mesh generated using the techniques discussed herein. In some instances, the map may be stored in a tile format, such that individual tiles of the map represent discrete portions of the environment and can be loaded into working memory as needed. In at least one example, one or more maps 624 may include at least one map (e.g., an image and / or a mesh) generated according to the techniques discussed herein. In some examples, the vehicle 602 may be controlled at least in part based on the map 624. That is, the map 624 may be used in conjunction with the positioning component 620, the perception component 622, and / or the planning component 630 to determine the location of the vehicle 602, identify objects in the environment, and / or generate routes and / or trajectories for navigation within the environment.

[0098] In some instances, perception component 622 may include a primary perception system and / or prediction system implemented in hardware and / or software. Perception component 622 may detect one or more objects in the environment surrounding vehicle 602 (e.g., identify the presence of objects), classify one or more objects (e.g., determine the object type associated with the detected objects), segment sensor data and / or other representations of the environment (e.g., identify portions of the sensor data and / or environmental representations as associated with the detected objects and / or object types), determine characteristics associated with the objects (e.g., identify trajectories of current, predicted, and / or previous positions, headings, speeds, and / or accelerations associated with the objects), etc. The data determined by perception component 622 is referred to as perception data.

[0099] In some examples, sensor data and / or perception data can be used to generate an environmental state representing the current state of the environment. For example, the environmental state can be a data structure containing object identification data (e.g., object location, area of ​​the environment occupied by the object, object heading, object velocity, historical object data), environmental layout data (e.g., a map of the environment or a sensor-generated layout), environmental condition data (e.g., location and / or area associated with environmental features (e.g., water or ice), whether it is raining, visibility measures), sensor data (e.g., images, point clouds), etc. In some examples, the environmental state can include a top-down two-dimensional representation of the environment and / or a three-dimensional representation of the environment, either of which can be augmented with object data. In yet another example, the environmental state may consist only of sensor data. In still another example, the environmental state can include both sensor data and perception data.

[0100] Prediction component 628 may include functionality for generating predictive information associated with objects in the environment. As an example, prediction component 628 may be implemented to predict the position of a pedestrian in the environment near a crosswalk area (or an area or location associated with the pedestrian crossing the road) when the pedestrian is crossing or preparing to cross a crosswalk area. As another example, the techniques discussed herein may be implemented to predict the positions of other objects (e.g., vehicles, bicycles, pedestrians, etc.) as vehicle 602 crosses the environment. In some examples, prediction component 628 may generate one or more predicted positions, predicted speeds, predicted trajectories, etc., for such a target object based on attributes of the target object and / or other objects near the target object.

[0101] The planning component 630 may receive the position and / or orientation of the vehicle 602 from the positioning component 620, receive sensing data from the sensing component 622, and / or receive a predicted trajectory from the prediction component 628, and may determine instructions for controlling the operation of the vehicle 602 based at least in part on any of these data. In some examples, determining the instructions may include determining the instructions based at least in part on a format associated with the system to which the instructions are associated (e.g., a first instruction for controlling the motion of the autonomous vehicle may be formatted as a message and / or signal (e.g., analog, digital, aerodynamic, kinematic) in a first format that can be parsed / executed by (one or more) system controllers 626 and / or (one or more) drive systems 614, and a second instruction for (one or more) transmitters 608 may be formatted according to a second format associated therewith). In at least one example, the planning component 630 may include a nominal trajectory generation subcomponent that generates a set of candidate trajectories and selects a trajectory for implementation by the drive system(s) 614 based at least in part on determining the cost associated with the trajectory, in accordance with U.S. Patent Application No. 16 / 517,506 filed July 19, 2019 and / or U.S. Patent Application No. 16 / 872,284 filed May 11, 2020 (the entire contents of which are incorporated herein by reference for all purposes).

[0102] Memory 618 and / or 640 may additionally or alternatively store mapping systems (e.g., maps generated at least in part based on sensor data), planning systems, riding management systems, etc. Although positioning component 620, sensing component 622, prediction component 628, planning component 630 and / or (one or more) system controllers 626 are shown as stored in memory 618, any of these components may include processor-executable instructions, (one or more) machine learning models (e.g., neural networks) and / or hardware, and all or part of any of these components may be stored on memory 640 or configured as part of computing device (one or more) 636.

[0103] As described herein, the localization component 620, perception component 622, prediction component 628, planning component 630, and / or other components of system 600 may include one or more ML models. For example, the localization component 620, perception component 622, prediction component 628, and / or planning component 630 may each include different ML model pipelines. The prediction component 628 may use different ML models or combinations of different ML models under different conditions. For example, the prediction component 628 may use different GNNs, RNNs, CNNs, MLPs, and / or other neural networks that are customized to output predictive agent trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajector trajectories ... Each layer in a neural network may also include another neural network, or may include any number of layers (whether convolutional or not). As will be understood in the context of this disclosure, neural networks can utilize machine learning, which can refer to a large class of such algorithms that generate outputs based on learned parameters.

[0104] Although discussed in the context of neural networks, any type of machine learning can be used in accordance with this disclosure. For example, machine learning algorithms can include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, minimum absolute shrinkage and selection operator (LASSO), elastic nets, minimum angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative bisection method 3 (ID3), chi-square automatic interaction detection (CHAID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naive Bayes, Gaussian Naive Bayes, multinomial Naive Bayes, average one-dependency estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g., k-means, k-median, expectation maximization (EM), hierarchical clustering), and association rule learning algorithms. (e.g., perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (mixture), gradient boosting machine (GBM), gradient boosting regression tree (GBRT), random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc.). Additional examples of architectures include neural networks, such as ResNet-50, ResNet-101, VGG, DenseNet, PointNet, etc.

[0105] The memory 618 may additionally or alternatively store one or more system controllers 626, which may be configured to control the steering, propulsion, braking, safety, transmitter, communication and other systems of the vehicle 602. These system controllers 626 may communicate with and / or control corresponding systems of the drive system(s) 614 and / or other components of the vehicle 602.

[0106] In additional or alternative examples, vehicle 602 and / or one or more computing devices 636 may communicate with one or more passenger devices (not shown) (e.g., sending and / or receiving messages via one or more networks 634). Passenger devices may include, for example, smartphones, portable computers (e.g., laptops or tablets), wearable devices (e.g., smart glasses, smartwatches, headphones), etc. Although passenger devices may be passenger-associated and separate from the autonomous vehicle's devices, it is conceivable that passenger devices may be subsystems and / or devices of vehicle 602. For example, passenger devices may additionally or alternatively include displays and / or one or more input / output devices, such as touchscreens, microphones, speakers, etc. In some examples, vehicle 602 may send and / or receive messages from passenger devices.

[0107] It should be noted that, although Figure 6 While shown as a distributed system, in an alternative example, components of vehicle 602 may be associated with computing device(s) 636 and / or components of computing device(s) 636 may be associated with vehicle 602. That is, vehicle 602 may perform one or more functions associated with computing device(s) 636, and vice versa.

[0108] Figure 7 This is a flowchart illustrating an example process 700 for generating combined infrared-radar sensor data and performing object detection in a low-visibility environment using the combined data. As described herein, the operation of process 700 can be described by referring to the above. Figures 1-6 The process 700 is performed by the perception component 622, which includes the infrared-radar fusion system 106 (and / or additional related components or sub-components). In various examples, the process 700 may be performed by the perception component 622 within the vehicle 102, which is configured to perform object detection and / or tracking functions for controlling the vehicle 102 within the driving environment. At operation 702, the sensing component 622 can receive multimodal sensor data of the driving environment. As described herein, the multimodal data may include at least infrared data captured by one or more infrared sensors of the vehicle, and radar data captured by one or more radar devices of the vehicle. Additionally, the multimodal data received at operation 702 may include various supplementary data captured by sensors of different types or modalities, including RGB (or visible light) sensor data, lidar data, etc.

[0109] At operation 704, the sensing component 622 can determine one or more visibility metrics for the driving environment and / or specific areas within the driving environment. As described above, the infrared-radar fusion system 106 can more effectively determine object detection in low-light or other low-visibility environments. Therefore, in some examples, the sensing component 622 can determine visibility metrics (e.g., brightness level) associated with the entire environment and / or visibility metrics associated with specific areas in the environment. For example, the sensing component 622 can use the current time, location, and environmental conditions to determine low-light or low-visibility driving conditions. Such driving conditions may include driving at night, driving in fog or rain, driving through tunnels, etc. The sensing component 622 can also analyze the environment to detect specific low-visibility areas within the environment, such as shadow areas, areas obscured by steam or exhaust fumes, etc.

[0110] At operation 706, the sensing component 622 can determine whether the driving environment includes one or more low-visibility areas. As described above, during nighttime driving, foggy driving, or rainy driving conditions, the sensing component 622 can determine that the environment as a whole is a low-visibility environment (706: Yes). In other examples, the sensing component 622 can determine that one or more specific areas in the environment, rather than the environment as a whole, are low-visibility areas (706: Yes). For example, the determination in operation 706 can be performed based on a brightness threshold and / or an RGB visibility threshold.

[0111] At operation 708, when perception component 622 determines that the driving environment includes one or more low-visibility areas (706: yes), perception component 622 can use infrared-radar fusion system 106 to generate combined infrared-radar sensor data. As described above, infrared-radar fusion system 106 may include various components configured to align and fuse infrared and radar sensor data (e.g., via overlay / projection component 320), and may include multiple models (e.g., object detection model 322 and / or ML transformer model 324) configured to perform object detection and classification based on the fused infrared and radar data. Output from such models may include object detection, which may include boundary shapes based on range / size and object position, classification, and additional attributes such as speed, yaw, etc.

[0112] As shown in this example, fusing infrared and radar data and performing object detection based on the combined data can be conditionally performed by the perception component 622 in low-visibility environments and / or environments with low-visibility areas. However, as mentioned above, in other examples, the infrared-radar fusion system 106 can be used to generate combined infrared-radar sensor data and perform object detection based on the combined data always and / or in all driving environments.

[0113] At operation 710, the sensing component 622 can synchronize object detections output by various sensor modalities and / or various object detection systems. For example, the set of object detections output by the infrared-radar fusion system 106 can be synchronized with a separate set of object detections output by the image object detection pipeline 328, the lidar object detection pipeline 330, and / or any number of additional object detections based on additional sensor modalities.

[0114] At operation 712, perception component 622 can determine a set of object classifications, attributes, and / or trajectories for the object detections determined in operation 710. As described above, the classification and determination of additional attributes of the detected objects can be performed by executing a separate ML model. Alternatively, the additional classification and attribute data can be output by the same ML object detection model and / or ML transformer model configured to output object detections. Examples of object classification models can include separate ML models or combined ML models trained to detect pedestrians in various poses (individually and in groups), various types of animals (individually and in groups), pedestrians on bicycles, scooters, and other means of transportation, etc., based on combined infrastructure radar sensor data.

[0115] At operation 714, perception component 622 can control vehicle 102 based on object detection determined in operation 710 and the associated classification, attributes, and trajectories determined in operation 712. In various examples, the object detection and associated trajectory data determined by perception component 622 can be used by prediction component, planning component, and / or various other downstream processing components for controlling vehicle 102. Such components may include functions for generating predicted object trajectories and future object states within the environment, and for determining driving paths or driving maneuvers that vehicle 102 should follow to navigate the environment safely and efficiently. Example Terms

[0116] A. A system comprising: one or more processors; and one or more non-transitory computer-readable media storing computer-executable instructions, which, when executed, cause the one or more processors to perform operations including: receiving long-wave infrared data associated with an environment, the long-wave infrared data being generated by an infrared sensor; receiving Doppler radar data associated with the environment, the Doppler radar data being generated by a radar sensor; determining a low-visibility region of the environment; determining a combined sensor data view associated with the low-visibility region of the environment based at least in part on the long-wave infrared data and the Doppler radar data; providing the combined sensor data view as input to a trained machine learning model; determining object detection representing objects in the low-visibility region of the environment based at least in part on the output of the trained machine learning model; and controlling the operation of a vehicle based at least in part on the object detection.

[0117] B. The system according to paragraph A, wherein the output of the trained machine learning model includes classification data associated with the object detection, wherein the classification data includes at least one of the following: animal classification; animal type subclassification; animal quantity subclassification; pedestrian classification; pedestrian quantity quantum classification; or pedestrian classification of riding vehicles.

[0118] C. The system according to paragraph A, wherein determining the combined sensor data view comprises: determining an image representation of the environment based on the long-wave infrared data; and projecting multiple radar points from Doppler radar data onto the image representation.

[0119] D. The system according to paragraph A, wherein determining the combined sensor data view comprises: determining a first embedding associated with the long-wave infrared data; determining a second embedding associated with the Doppler radar data; inputting the first embedding and the second embedding into a transformer model; and determining object detection data associated with low-visibility areas of the environment based at least in part on the output of the transformer model.

[0120] E. According to the system described in paragraph A, the operation further includes: determining sensor data characteristics associated with the object based at least in part on the long-wave infrared data and the Doppler radar data; and tracking the object in the environment for a period of time based at least in part on the sensor data characteristics.

[0121] F. A method comprising: receiving infrared sensor data associated with an environment, the infrared sensor data being generated by an infrared sensor; receiving radar sensor data associated with the environment, the radar sensor data being generated by a radar sensor; determining, at least in part, a combined sensor data view associated with a portion of the environment based on the infrared sensor data and the radar sensor data; determining, at least in part, object detection representing an object based on the combined sensor data view; and controlling the operation of a vehicle, at least in part, based on the object detection.

[0122] G. According to the method described in paragraph F, determining the combined sensor data view includes: determining an image representation of the environment based on the infrared sensor data; and projecting multiple radar points from the radar sensor data onto the image representation.

[0123] H. According to the method described in paragraph F, determining the combined view of sensor data includes: determining a first embedding associated with the infrared sensor data; determining a second embedding associated with the radar sensor data; inputting the first embedding and the second embedding into a machine learning transformer model; and determining object detection data associated with a portion of the environment based at least in part on the output of the machine learning transformer model.

[0124] I. The method according to paragraph H, wherein the object detection data includes: a boundary shape representing the object detection within a portion of the environment.

[0125] J. The method according to paragraph F, wherein the radar sensor data includes a set of radar points associated with the environment, and wherein the combined sensor data view includes: range data associated with the set of radar points; azimuth data associated with the set of radar points; Doppler data associated with the set of radar points; and elevation data associated with the set of radar points.

[0126] K. According to the method described in paragraph F, determining the object detection includes: providing the combined view of sensor data as input to a trained machine learning model; and determining a classification associated with the object based at least in part on the output of the trained machine learning model.

[0127] L. According to the method described in paragraph K, the output of the trained machine learning model includes classification data associated with the object, wherein the classification data includes at least one of the following: animal classification; animal type subclassification; animal quantity subclassification; pedestrian classification; pedestrian quantity quantum classification; or pedestrian classification of riding vehicles.

[0128] M. The method according to paragraph F further includes: determining sensor data characteristics associated with the object based at least in part on the infrared sensor data and the radar sensor data; and tracking the object in the environment for a period of time based at least in part on the sensor data characteristics.

[0129] N. According to the method described in paragraph F, determining the combined sensor data view comprises: determining a visibility metric associated with a portion of the environment; comparing the visibility metric with a visibility threshold; and generating the combined sensor data view based at least in part on determining that the visibility metric is less than the visibility threshold.

[0130] O. One or more non-transitory computer-readable media storing instructions executable by one or more processors, wherein, when executed, the instructions cause the one or more processors to perform operations including: receiving infrared sensor data associated with an environment, the infrared sensor data being generated by infrared sensors; receiving radar sensor data associated with the environment, the radar sensor data being generated by radar sensors; determining, at least in part, a combined sensor data view associated with a portion of the environment based on the infrared sensor data and the radar sensor data; determining, at least in part, object detection representing an object based on the combined sensor data view; and controlling the operation of a vehicle based at least in part on the object detection.

[0131] P. According to one or more non-transitory computer-readable media as described in paragraph O, determining the combined sensor data view includes: determining an image representation of the environment based on the infrared sensor data; and projecting a plurality of radar points from the radar sensor data onto the image representation.

[0132] Q. According to one or more non-transitory computer-readable media as described in paragraph O, determining the combined view of sensor data includes: determining a first embedding associated with the infrared sensor data; determining a second embedding associated with the radar sensor data; inputting the first embedding and the second embedding into a machine learning transformer model; and determining object detection data associated with a portion of the environment based at least in part on the output of the machine learning transformer model.

[0133] R. According to one or more non-transitory computer-readable media as described in paragraph O, wherein the radar sensor data comprises a set of radar points associated with the environment, and wherein the combined sensor data view comprises: range data associated with the set of radar points; azimuth data associated with the set of radar points; Doppler data associated with the set of radar points; and elevation data associated with the set of radar points.

[0134] S. According to one or more non-transitory computer-readable media as described in paragraph O, wherein determining the object detection comprises: providing a combined view of sensor data as input to a trained machine learning model; and determining a classification associated with the object based at least in part on the output of the trained machine learning model.

[0135] T. According to one or more non-transitory computer-readable media as described in paragraph O, the operation further includes: determining sensor data characteristics associated with the object based at least in part on the infrared sensor data and the radar sensor data; and tracking the object in the environment for a period of time based at least in part on the sensor data characteristics.

[0136] Although the example clauses above have been described with respect to specific implementations, it should be understood that, in the context of this document, the content of the example clauses may be implemented via methods, devices, systems, computer-readable media, and / or other implementations. Furthermore, any of the example ATs may be implemented alone or in combination with any other one or more of the example ATs. in conclusion

[0137] Although one or more examples of the techniques described herein have been described, various modifications, additions, substitutions, and equivalents thereof are also included within the scope of the techniques described herein.

[0138] In the description of the examples, reference is made to the accompanying drawings, which form part of the description, illustrating specific examples of the claimed subject matter by way of illustration. It should be understood that other examples may be used, and changes or alterations such as structural modifications may be made. Such examples, changes, or alterations do not necessarily deviate from the scope of the claimed subject matter. Although the steps herein may be presented in a certain order, in some cases the order may be changed so that certain inputs are provided at different times or in a different order without altering the function of the described system and method. The disclosed processes may also be performed in a different order. Furthermore, it is not necessary to perform the various calculations herein in the disclosed order, and other examples using alternative orders of calculations can be readily implemented. In addition to being reordered, these calculations may also be decomposed into sub-computations with the same results.

[0139] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms for implementing the claims.

[0140] The components described herein represent instructions that can be stored in any type of computer-readable medium and can be implemented in software and / or hardware. All the methods and processes described above can be embodied in software code modules and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof, and are fully automated via these software code modules and / or computer-executable instructions. Some or all of the methods may alternatively be embodied in dedicated computer hardware.

[0141] Unless otherwise explicitly stated, conditional language (e.g., “can,” “able,” “may,” or “possibly”) is understood in context to indicate that certain features, elements, and / or steps are included in some examples but not in others. Therefore, such conditional language is generally not intended to imply that certain features, elements, and / or steps are required in any way for one or more examples, or that one or more examples must include logic for determining whether certain features, elements, and / or steps are included in or will be executed in any particular example, with or without user input or prompts.

[0142] Unless otherwise explicitly stated, connecting language (e.g., the phrase "at least one of X, Y, or Z") should be understood to mean that items, terms, etc., can be X, Y, or Z, or any combination thereof, including plural of each element. Unless explicitly stated as singular, "a" means both singular and plural.

[0143] Any routine description, element, or block depicted in the flowcharts described herein and / or in the accompanying drawings should be understood as potentially representing a module, segment, or portion of code comprising one or more computable instructions for implementing a particular logical function or element in the routine. Alternative implementations are included within the scope of the examples described herein, wherein elements or functions may be omitted or performed in a different order than shown or discussed, including substantially synchronous execution, execution in reverse order, with additional operations, or omitted operations, depending on the functionality involved, as will be understood by those skilled in the art.

[0144] Many variations and modifications can be made to the above examples, and their elements should be understood as existing in other acceptable examples. All such modifications and variations are intended to be included herein, within the scope of this disclosure, and protected by the appended claims.

Claims

1. A system comprising: One or more processors; as well as One or more non-transitory computer-readable media storing computer-executable instructions, which, when executed, cause the one or more processors to perform operations, including: Receives infrared sensor data associated with the environment, the infrared sensor data being generated by an infrared sensor; Receive radar sensor data associated with the environment, the radar sensor data being generated by radar sensors; A combined view of sensor data associated with a portion of the environment is determined, at least in part, based on the infrared sensor data and the radar sensor data. Object detection representing an object is determined at least in part based on the combined view of sensor data; and The operation of the vehicle is controlled at least in part based on the object detection.

2. The system according to claim 1, wherein, Determining the combined sensor data view includes: Determine an image representation of the environment based on the infrared sensor data; and Multiple radar points from the radar sensor data are projected onto the image representation.

3. The system according to claim 1 or claim 2, wherein, Determining the combined sensor data view includes: Determine the first embedding associated with the infrared sensor data; Determine a second embedding associated with the radar sensor data; The first embedding and the second embedding are input into a machine learning transformer model; and Object detection data associated with a portion of the environment is determined at least in part based on the output of the machine learning transformer model.

4. The system according to claim 3, wherein, The object detection data includes: This represents the boundary shape of the object detected within a portion of the environment.

5. The system according to any one of claims 1-4, wherein, The radar sensor data includes a set of radar points associated with the environment, and wherein the combined sensor data view includes: Range data associated with the set of radar points; Azimuth data associated with the set of radar points; Doppler data associated with the set of radar points; and Elevation angle data associated with the set of radar points.

6. A method comprising: Receives infrared sensor data associated with the environment, the infrared sensor data being generated by an infrared sensor; Receive radar sensor data associated with the environment, the radar sensor data being generated by radar sensors; A combined view of sensor data associated with a portion of the environment is determined, at least in part, based on the infrared sensor data and the radar sensor data. Object detection representing an object is determined at least in part based on the combined view of sensor data. as well as The operation of the vehicle is controlled at least in part based on the object detection.

7. The method according to claim 6, wherein, Determining the combined sensor data view includes: Determine an image representation of the environment based on the infrared sensor data; and Multiple radar points from the radar sensor data are projected onto the image representation.

8. The method according to claim 6 or claim 7, wherein, Determining the combined sensor data view includes: Determine the first embedding associated with the infrared sensor data; Determine a second embedding associated with the radar sensor data; The first embedding and the second embedding are input into a machine learning transformer model; and Object detection data associated with a portion of the environment is determined at least in part based on the output of the machine learning transformer model.

9. The method according to claim 8, wherein, The object detection data includes: This represents the boundary shape of the object detected within a portion of the environment.

10. The method according to any one of claims 6-9, wherein, The radar sensor data includes a set of radar points associated with the environment, and wherein the combined sensor data view includes: Range data associated with the set of radar points; Azimuth data associated with the set of radar points; Doppler data associated with the set of radar points; and Elevation angle data associated with the set of radar points.

11. The method according to any one of claims 6-10, wherein, Determining the object detection includes: The combined sensor data view is fed as input to the trained machine learning model; and The classification associated with the object is determined at least in part based on the output of the trained machine learning model.

12. The method according to claim 11, wherein, The output of the trained machine learning model includes classification data associated with the object, wherein the classification data includes at least one of the following: Animal classification; Animal type subclassification; Animal quantity subclassification; Pedestrian classification; Pedestrian count quantum classification; or Pedestrians using bicycles are classified into categories.

13. The method according to any one of claims 6-12, further comprising: Sensor data characteristics associated with the object are determined at least in part based on the infrared sensor data and the radar sensor data; as well as The objects in the environment are tracked over a period of time, at least in part, based on the characteristics of the sensor data.

14. The method according to any one of claims 6-13, wherein, Determining the combined sensor data view includes: Determine the visibility metric associated with a portion of the environment; The visibility metric is compared with a visibility threshold; and The combined sensor data view is generated at least in part based on determining that the visibility metric is less than the visibility threshold.

15. One or more non-transitory computer-readable media comprising instructions that, when executed by one or more processors, cause the one or more processors to perform the method according to any one of claims 6-14.

Images