Truck Reliability Model

Abstract

Techniques for determining outputs from multiple sensor modalities are described herein. Features from the radar sensor, the lidar sensor, and the image sensor may be input into respective models to determine respective intermediate outputs associated with a track associated with the object and an associated confidence level. Intermediate outputs from the radar model, the lidar model, and the vision model may be input into a fused model to determine a fused confidence level and a fused output associated with the track. The fused confidence level and the individual confidence levels are compared to a threshold to generate a track to send to a planning or predictive system of the autonomous vehicle. Additionally, a vehicle controller may control the autonomous vehicle based on the track and / or the confidence level(s).

Description

[Technical field]

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS This PCT international patent application claims the benefit of priority to U.S. patent application Ser. No. 17 / 553,938, filed Dec. 17, 2021, the disclosure of which is incorporated by reference herein. [Background technology]

[0002] Planning systems for autonomous vehicles can utilize information associated with objects in an environment to determine actions related to those objects. For example, some existing planning systems for autonomous vehicles consider the movement of objects, such as other vehicles on a road surface, when determining maneuvers for the autonomous vehicle to traverse through an environment. Conventional systems may combine data from different modalities, such as radar, lidar, and / or vision, to determine information about an object(s). However, when one or many modalities do not detect an object, the resulting track may have a low confidence level. [Brief description of the drawings]

[0003] [Figure 1] FIG. 1 is a flow diagram of an example system and technique for determining confidence levels of different sensor modalities. [Diagram 2] FIG. 13 is a pictorial flow diagram for determining tracks of a fused modality model. [Diagram 3] 1 is a diagrammatic representation illustrating example systems and techniques that use radar data to track objects in an environment. [Figure 4] 1 includes a block diagram of an example architecture of a vehicle control system and a computing system for determining track of multiple sensor modalities as described herein. [Diagram 5]FIG. 11 is a flow diagram illustrating an example process for determining tracks from multiple sensor modalities for output. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0004] Described herein are techniques for determining confidence level(s) of an object detected and tracked by a sensor in an environment. For example, sensor data can be captured using one or more of radar data, lidar data, image data, and the like. The sensor data can be input to a tracker(s) or model(s) trained to determine a track of the object(s). The technique can include using data generated by each sensor modality to determine a confidence level associated with each sensor modality and comparing the respective confidence level to a threshold. Furthermore, intermediate outputs of the trackers or models associated with each sensor modality can be combined into fused data incorporating data from multiple sensor modalities, and the fused data can include and / or comprise a group of objects detected by one or more sensor modalities. In some examples, a confidence level associated with the fused data can be determined and the confidence level associated with the fused data can be compared to a threshold together with the confidence levels of the individual sensor modalities to determine an output from one of the sensor modalities. Thus, if a first confidence level determined via a single sensor modality is higher than a threshold and a second confidence level determined based on the fused data is lower than a threshold, the second confidence level may be suppressed or replaced with the first confidence level and output to one or more of the vehicle's planner system and / or predictive system along with track data of the group of objects based on the single sensor modality or one of the fused modalities.

[0005] By way of example, and not by way of limitation, a perception system may receive sensor data from multiple modalities and determine object information (e.g., data for tracking an object in an environment). In some cases, data from all sensor modalities may be input into a single model to determine a track and / or confidence level. For example, if a lidar sensor and a vision sensor do not detect an object, but a radar sensor does, a track for the object may have a relatively low confidence (e.g., because training examples where a true positive detection by one modality occurs simultaneously with the modality reporting a false negative may be relatively rare). If downstream systems (e.g., planner systems, prediction systems, and the like) run out of, receive, or otherwise process tracks with low confidence, the systems may not properly consider the object when planning (e.g., planning missions, trajectories, and the like for an autonomous vehicle). In some examples, if a single model determines a low confidence level, the track may not be output to downstream systems. Thus, a predictive system may not properly predict pedestrian movements, or a planner system may not properly consider pedestrians because a downstream system, with a low confidence level, either processed tracks associated with the pedestrian with low confidence or did not receive tracks associated with the pedestrian due to low confidence. However, determining confidence levels for data tracks individually, as well as based on fused data, can lead to a more accurate consideration of available data by making it more certain that tracks associated with objects with high confidence are output to downstream systems.

[0006] In some examples, the multiple sensor configurations may be disposed on a vehicle, e.g., an autonomous vehicle, and operably coupled to a controller and / or a remote computing device. The controller may receive a first signal from a first sensor configuration disposed on the vehicle. The first sensor configuration may include an image capture sensor that captures vision data representative of a first scene (e.g., a portion of an environment visible to the image sensor, e.g., as may be determined by a field of view). In some examples, the first signal may include vision data. In some examples, for example, the vision data may indicate a scene illustrating and / or including a first group of detectable objects in an environment in which the first sensor and / or the vehicle are present. In some examples, the controller may analyze the vision data by using a machine-learned vision model to identify track data of one or more objects associated with the vision data and determine a confidence level associated with the vision data. In some examples, the vision model may be trained using a ground truth indicating whether the vision data is a ground truth object track.

[0007] In some examples, the second sensor configuration may be disposed on the vehicle. In some examples, the second sensor configuration may be a lidar sensor. In some examples, the controller may be configured to receive a second signal from the lidar sensor. In some examples, the second signal may include lidar data. In some examples, the lidar data may be captured contemporaneously (or substantially contemporaneously within technical tolerances) with the vision data described above. In some examples, the lidar data may represent a second scene (e.g., a portion of the environment visible to the lidar sensor) captured by the lidar sensor contemporaneously with the image data described above. In some examples, for example, the lidar data may be indicative of a scene captured by the lidar sensor that includes a second group of objects detectable in the environment. In some examples, the controller may analyze the lidar data by using a machine-learned lidar model to identify one or more object track data associated with the lidar data and determine a confidence level associated with the lidar data. In some examples, the lidar model may be trained using a ground truth indicating whether the lidar data is a ground truth object track. In some examples, depending on the accuracy and / or fidelity of the lidar data, for example, the objects included in the second group of objects may be identical to the objects included in the first group of objects detectable by the image capture device (e.g., at least a portion of the first group of objects and a portion of the second group of objects are both visible to the image capture sensor and the lidar sensor simultaneously).

[0008] In some examples, the third sensor configuration may be disposed on the vehicle. In some examples, the third sensor configuration may be a radar sensor. In some examples, the controller may be configured to receive a third signal from the radar sensor. In some examples, the third signal may include radar data. In some examples, the radar data may be captured simultaneously with the vision data and / or lidar data described above. In some examples, the sensor data may represent a third scene (e.g., a portion of the environment visible to the radar sensor). In some examples, for example, the radar data may indicate a scene captured by the radar sensor that includes a third group of objects detectable in the environment. In some examples, the controller may analyze the radar data by using a machine-learned radar model to identify one or more object track data associated with the radar data and determine a confidence level associated with the radar data. In some examples, the radar model may be trained using a ground truth indicating whether the radar data is a ground truth object track. In some examples, depending on the accuracy and / or fidelity of the radar data, for example, an object included in the third group of objects may be identical to an object included in the first group of objects detectable by the image capture device and / or in the second group of objects detectable by the lidar sensor (e.g., at least a portion of the first group of objects and a portion of the second group of objects and / or a portion of the third group of objects are simultaneously visible together to the image capture sensor, the lidar sensor, and / or the radar sensor).

[0009] In some examples, through the data fusing process, intermediate outputs associated with the vision data, the lidar data, and / or the radar data may be combined as fused sensor data. In some examples, the intermediate outputs may be track data of the individual modalities as described above. In some examples, the fused sensor data may include and / or identify a fourth group of one or more objects predicted, determined, and / or otherwise indicated as being present in the environment based at least in part on the vision data, the lidar data, and / or the radar data. In some examples, the fused sensor data may represent a fourth scene. In some examples, the controller may analyze the fused data by using a machine learning fused model to identify track data associated with the fused data and determine a confidence level associated with the fused data. In some examples, the fused model may be trained using a ground truth indicating whether the fused sensor data is a ground truth object track.

[0010] In some examples, a confidence level associated with the fused sensor data and a confidence level associated with the vision data, the lidar data, and / or the radar data may be compared to a threshold to determine whether to transmit track data associated with the vision data, track data associated with the lidar data, track data associated with the radar data, or track data associated with the fused sensor data to a planner system associated with the vehicle and / or a predictive system associated with the vehicle. For example, if the confidence level associated with the fused sensor data is below the threshold and the confidence level associated with the radar data is above the threshold, the track data associated with the radar data is transmitted to the planner system associated with the vehicle and / or the predictive system associated with the vehicle.

[0011] Techniques for using confidence levels associated with individual sensor modalities and fused sensor data to determine which output of one of the multiple sensor modalities to send to the vehicle's planner system and / or the vehicle's prediction system can enhance vehicle functionality by providing more accurate object detection and improve safe vehicle operation. In some examples, using outputs from individual modalities with low confidence levels, the fused sensor data may output track data with an equally low confidence level, resulting in the vehicle's perception system outputting bounding boxes with low confidence, the vehicle's prediction system being unable to properly predict the movement of objects in the environment (e.g., pedestrians, other vehicles, bicyclists, and the like), or the vehicle's planner system planning a trajectory that is too close to objects in the environment.

[0012] The methods, apparatus, and systems described herein may be implemented in many ways. Exemplary implementations are provided below with reference to the following drawings. Although the present disclosure uses autonomous vehicles in the examples, the techniques described herein are not limited to application in autonomous vehicles. For example, any system that navigates an environment using radar data may benefit from the described radar data processing techniques. In another example, the techniques described herein may be used in aircraft to, for example, identify other aircraft and / or moving objects. Furthermore, non-autonomous vehicles could also benefit from the techniques described herein, for example, with respect to collision detection and / or avoidance systems.

[0013] 1 illustrates an example flow diagram 100 of systems and techniques for determining confidence levels (also called "confidence values" or "confidence scores") of different sensor modalities. The example flow diagram 100 includes a radar detector 102, a lidar detector 110, and a vision detector 118.

[0014] In some examples, the processing may include the radar detector 102 detecting radar features 104. In some examples, the radar features 104 may include radar data associated with the environment. Examples of radar features may be object data such as, but not limited to, bounding boxes, velocity, yaw, centroid, acceleration, classification, dimensionality, track time (e.g., a time period indicating how long an object has been tracked using a modality), and the like. In some examples, the radar features 104 may be input to a radar multi-layer perceptron 106. In some examples, the radar multi-layer perceptron 106 may be a machine learning model. In some examples, the radar multi-layer perceptron 106 may output intermediate radar outputs associated with the radar features. In some examples, the intermediate radar outputs may be associated with track data representing objects in the environment. In some examples, the intermediate radar outputs may be associated with one of the layers of the radar multi-layer perceptron 106 (e.g., embeddings from an intermediate layer). In some examples, the intermediate radar output may include, but is not limited to, a bounding box associated with the object, the object's velocity, the object's attitude, and the like. In some examples, a radar confidence level 108 (also referred to as a "radar confidence value" or "radar confidence score") may be determined based on the radar features 104 using the radar multi-layer perceptron 106. As used herein, a confidence level or confidence value may correspond to a confidence that a modality-specific component detects the object, or to a confidence value that corresponds to a certain characteristic about the object (e.g., size, location, classification, speed, etc.).

[0015] In some examples, the processing may include a lidar detector 110 detecting lidar features 112. In some examples, the lidar features 112 may include lidar data associated with the environment. Examples of lidar features may be object data such as, but not limited to, bounding boxes, velocity, yaw, centroid, acceleration, classification, dimensionality, track time, and the like. In some examples, the lidar features 112 may be input to a lidar multi-layer perceptron 114. In some examples, the lidar multi-layer perceptron 114 may be a machine learning model. In some examples, the lidar multi-layer perceptron 114 may output intermediate lidar outputs associated with the lidar features 112. In some examples, the intermediate lidar outputs may be associated with track data representing objects in the environment. In some examples, the intermediate lidar outputs may be associated with one of the layers of the lidar multi-layer perceptron 114 (e.g., embeddings from an intermediate layer). In some examples, the intermediate lidar outputs associated with the lidar features 112 may include, but are not limited to, a bounding box associated with the object, the velocity of the object, the attitude of the object, and the like. In some examples, a lidar confidence level 116 (also referred to as a “lidar confidence value” or “lidar confidence score”) may be determined based on the lidar features 112 using a lidar multi-layer perceptron 114.

[0016] In some examples, the processing may include the vision detector 118 detecting the vision features 120. In some examples, the vision features 120 may include vision data associated with the environment (e.g., image data, camera data, and the like). Examples of vision features may be, but are not limited to, object data such as, for example, bounding box, velocity, yaw, center, acceleration, classification, dimensionality, track time, and the like. In some examples, the vision features 120 may be input to a vision multi-layer perceptron 122. In some examples, the vision multi-layer perceptron 122 may be a machine learning model. In some examples, the vision multi-layer perceptron 122 may output intermediate vision outputs associated with the vision features 120. In some examples, the intermediate vision outputs may be associated with track data representing objects in the environment. In some examples, the intermediate vision outputs may be associated with one of the layers of the vision multi-layer perceptron 122 (e.g., embeddings from an intermediate layer). In some examples, the intermediate vision outputs may include, but are not limited to, a bounding box associated with the object, the object's velocity, the object's pose, and the like. In some examples, a vision confidence level 124 (also referred to as a "vision confidence value" or "vision confidence score") may be determined based on the vision features 120 using a vision multi-layer perceptron 122.

[0017] In some examples, the intermediate radar output, the intermediate lidar output, and the intermediate vision output may be associated with the same object in the environment. In some examples, the intermediate radar output, the intermediate lidar output, and the intermediate vision output may be input to a fused multi-layer perceptron 126 (also referred to as a "multi-modality multi-layer perceptron" or "multi-modality model"). In some examples, the intermediate radar output, the intermediate lidar output, and the intermediate vision output may be collectively referred to as fused sensor data (also referred to as "multi-modality sensor data"). In some examples, the fused multi-layer perceptron 126 may be a machine learning model. In some examples, the fused multi-layer perceptron 126 may output a fused output (also referred to as a "multi-modality output") associated with the fused sensor data. In some examples, the fused output may be associated with a track representing an object in the environment. In some examples, the fused output may include, but is not limited to, a bounding box associated with the object, the object's velocity, the object's posture, and the like. In some examples, a fused confidence level 128 (also referred to as a "fused confidence value," "fused confidence score," "multi-modality confidence level," "multi-modality confidence value," or "multi-modality confidence score") may be determined based on the fused sensor data using a fused multi-layer perceptron 126.

[0018] In some examples, the process may include a confidence component 130 that receives the radar confidence level 108, the lidar confidence level 116, the vision confidence level 124, and the fused confidence level 128. In some examples, the confidence component 130 may determine an output confidence level based on whether the fused confidence level 128 is higher than the radar confidence level 108, the lidar confidence level 116, and the vision confidence level 124, or based on whether at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 is higher than the fused confidence level 128. In some examples, the just mentioned determination is made by comparing at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 and the fused confidence level 128 to a threshold value. For example, if the fused confidence level 128 is below the threshold and the vision confidence level 124 is above the threshold, the confidence component 130 may determine to output the vision confidence level 124. In some examples, at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 may be compared to a first threshold, and there may be multiple thresholds where the fused confidence level 128 may be compared to a different second threshold. In some examples, if the fused confidence level 128 is above the second threshold, the confidence component 130 may determine to output the fused confidence level 128. In some examples, if the fused confidence level 128 is below the second threshold and the vision confidence level 124 is above the threshold, the confidence component 130 may determine to output the vision confidence level 124.

[0019] It should be understood that various schemes are envisioned for use by the confidence component 130. For example, confidences corresponding to several (e.g., two) modalities may be aggregated and, if a threshold is met, the corresponding data may be used (even though another modality may indicate a low confidence). What has just been described may be useful, for example, in environmental conditions that may be detrimental to a particular modality. In this manner, the confidence component 130 may adaptively output data from multiple modalities based on environmental factors, such as, for example, weather, time of day, sensor orientation, sensor degradation, sensor operating conditions, etc. The confidence component 130 may implement one or more machine learning models, deterministic codes, or combinations. For example, a machine learning model may be trained to determine which environmental factors indicate that the output of a particular corresponding modality is to be used, resulting in a valid perception output. In this manner, training of a model to determine which modalities are beneficial in which conditions can be considered separately from the machine learning perception component trained to determine objects in the environment (which may be overly biased to use certain modalities). In some examples, the confidence component 130 can use information received from one or more perception (e.g., modality-specific) components. As used herein, confidence can apply to the confidence of size, shape, classification, or any other attribute corresponding to an object. In some examples, the confidence component 130 can look at the corresponding results and / or output from a perception component (e.g., a modality-based perceptron) using the techniques disclosed herein. For example, if two modalities have relatively lower confidence but output corresponding classifications, the confidence component 130 may be more likely to use the output from the modality / component.In some examples, the confidence component 130 may use one or more deterministic rules to ensure that certain objects are taken into account. For example, if any pipeline detects a pedestrian within a certain range of the vehicle with a corresponding confidence (e.g., for the range), the vehicle may use that information to avoid unsafe operations with respect to pedestrians.

[0020] In some examples, the process may include an operation 132 configured to output at least one of a radar confidence level 108, a lidar confidence level 116, a vision confidence level 124, or a fused confidence level 128, and a track representative of the object in the environment. In some examples, the operation 132 may generate track data representative of the object in the environment and output the track data to a prediction or planning system of the autonomous vehicle. In some examples, the track data may be based on at least one of the radar intermediate output, the lidar intermediate output, the vision intermediate output, or the fused intermediate output depending on the modality of the confidence level being output. In some examples, the operation 132 may update the track data based on at least one of the radar intermediate output, the lidar intermediate output, the vision intermediate output, or the fused intermediate output depending on the modality of the confidence level being output. In some examples, the track data may include, but is not limited to, an object ID associated with the object and a shape representation associated with the object (e.g., a bounding box, a speed, a contour, a posture, an orientation, and the like).

[0021] Figure 2 illustrates an example pictorial flow diagram 200 for determining tracks representing objects in an environment. Figure 2 includes a vision detector 206, a lidar detector 212, and a radar detector 218. In some examples, the vision detector 206, the lidar detector 212, and the radar detector 218 may be the same as or similar to the vision detector 118, the lidar detector 110, and the radar detector 102.

[0022] In some examples, the vision detector 206, the lidar detector 212, and the radar detector 218 may be used by the vehicle while navigating the environment 202. In some examples, the environment 202 may include an object 204. In some examples, the object 204 is another vehicle. The object 204 is not limited to a vehicle and may be other types of objects such as, for example, a pedestrian, a bicyclist, and the like.

[0023] In some examples, the vision features detected by the vision detector 206 may be input to a vision multi-level perception 208. In some examples, the vision multi-level perception 208 may be the same as or similar to the vision multi-layer perceptron 122. In some examples, an intermediate vision output 210 may be determined using the vision multi-level perception 208. In some examples, the intermediate vision output 210 may include a bounding box associated with the object 204.

[0024] In some examples, the lidar features detected by the lidar detector 212 may be input to a lidar multi-level perception 214. In some examples, the lidar multi-level perception 214 may be the same as or similar to the lidar multi-layer perceptron 114. In some examples, an intermediate lidar output 216 may be determined using the lidar multi-level perception 214. In some examples, the intermediate lidar output 216 may include a bounding box associated with the object 204.

[0025] In some examples, radar features detected by the radar detector 218 may be input to a radar multi-level perception 220. In some examples, the radar multi-level perception 220 may be the same as or similar to the radar multi-layer perceptron 106. In some examples, an intermediate radar output 222 may be determined using the radar multi-level perception 220. In some examples, the intermediate radar output 222 may include a bounding box associated with the object 204.

[0026] In some examples, the intermediate vision output 210, the intermediate lidar output 216, and the intermediate radar output 222 may be input to a fused multilevel perception 224 to generate a fused output 226. In some examples, the fused multilevel perception 224 may be the same as or similar to the fused multilayer perceptron 126. In some examples, the fused output 226 may include a bounding box associated with the object 204. In some examples, track data associated with the object 204 may be generated from the fused output 226, and the track data may be transmitted to a planner system 228. In some examples, the planner system 228 may be a planner system of a vehicle, such as, for example, an autonomous vehicle. In some examples, track data may be generated and / or updated based on the intermediate vision output 210, the intermediate lidar output 216, and the intermediate radar output 222, and confidence levels associated with the radar features, lidar features, vision features, and / or fused features (e.g., using fused multi-level perception 224) may be input to a planner system, a prediction system, another tracking model, and the like. In some examples, the planner system 228 may be the same as or similar to the planning component 426 illustrated in FIG. 4. In some examples, a vehicle control system, such as, for example, vehicle control system 402 of FIG. 4, may control the vehicle based on the track data.

[0027] 3 is a schematic illustration showing an example scenario 300 in which a vehicle 302 is driving on a road surface 304. As shown, a second vehicle 306 is also traveling on the road surface 304. In the example scenario 300, the vehicle 302 and the second vehicle 306 are generally traveling in the same direction and in the same lane of the road. In some examples, the example scenario may be the same scenario illustrated in the environment 202.

[0028] In some examples, the vehicle 302 may be an autonomous vehicle configured to operate in accordance with a Level 5 classification issued by the United States Federal Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions throughout its journey, where the driver (or passenger) is not expected to control the vehicle at any time. In some examples, the vehicle 302 may be configured to control all functions from start to stop, including all parking functions, so that it may not be occupied. However, the vehicle 302 is merely an example, and the systems and methods described herein may be incorporated into any vehicle, including vehicles ranging from those that must be manually controlled by a driver at all times to those that are partially or fully autonomously controlled. In additional implementations, the techniques described herein may be useful in settings other than vehicles. The techniques described herein may be useful in many different applications that use sensor data to determine information about objects in an environment.

[0029] In some examples, the vehicle 302 may include a multiple sensor configuration including a first radar sensor 308 and a second radar sensor 310. As shown, the first radar sensor 308 and the second radar sensor 310 are positioned to propagate waves generally in the direction of travel of the vehicle 302 (e.g., generally along the direction of the illustrated arrow toward the rear of the vehicle 302). As further shown, the first radar sensor 308 and the second radar sensor 310 may have overlapping fields of view. Thus, a first radiated wave 312 emitted by the first radar sensor 308 reflects off the second vehicle 306, returns to the first radar sensor 308, and is detected via a first radar scan. Similarly, a second radiated wave 314 further emitted by the second radar sensor 310 also reflects off the second vehicle 306, returns to the second radar sensor 310, and is detected via a second radar scan. In some examples, the first radar sensor 308 and the second radar sensor 310 may be substantially identical, except for their location on the vehicle 302. In some examples, the radar sensors 308 and 310 may be configured differently. In some examples, the radio waves 312, 314 may be emitted at different frequencies. In some examples, the radar sensors 308, 310 may be configured such that the scans at the radar sensors 308, 310 have different intervals (e.g., Doppler intervals). In some examples, the characteristics of the radar sensors 308, 310 may include, but are not limited to, center frequency, scan type, scan pattern, frequency modulation, pulse repetition frequency, pulse repetition interval, but may be configured to create different Doppler intervals, for example. Thus, the radar sensors 308, 310 may both be positioned to generally sense objects in the same direction with respect to the vehicle 302.

[0030] The radar sensors 308, 310 may receive the radio waves 312, 314 after they reflect off a surface in the environment (e.g., a surface of the second vehicle 306), and the radar sensors 308, 310 may generate radar data based on the reflection. For example, the radar data may include various types of information including, but not limited to, velocity associated with one or more points representing surfaces or objects in the environment of the radar sensor. In some examples, when the radar sensors 308, 310 are pulse Doppler sensors, the velocity of the object relative to the respective sensor may be determined.

[0031] 3 also illustrates a number of radar returns associated with the radar sensors 308, 310, represented diagrammatically by points. In the illustrative example, the first points 316(1), 316(2) (collectively, the first points 316) are illustrated as circles and represent radar returns associated with the radar sensor 308. That is, each of the first points 316 indicates a location of the second vehicle 306 from which the emitted radio wave 312 is reflected. Similarly, the second points 318(1), 318(2) (collectively, the second points 318) are illustrated as "X's" and represent radar returns associated with the radar sensor 310. Therefore, each of the second points 318 indicates a location of the second vehicle 306 from which the emitted radio wave 314 is reflected.

[0032] As also illustrated in FIG. 3, the vehicle 302 may also include a number of additional sensors 320. The additional sensors 320 may be positioned to sense objects in the same direction as the radar sensor 308 and / or the radar sensor 310, typically. Examples of the additional sensors 320 may be, but are not limited to, one or more of an additional radar sensor, a lidar sensor, an imaging sensor (e.g., a camera), a time-of-flight sensor, a SONAR sensor, an infrared imaging device, or the like. Although two instances of the additional sensors 320 are illustrated in FIG. 3, the vehicle 302 may include any number of additional sensors having any number of different modalities. In some examples, the vehicle 302 may include multiple additional sensors positioned to detect objects in other relative positions.

[0033] As illustrated in the accompanying block diagram of FIG. 3, the radar sensors 308, 310 and the additional sensor 320 may represent a type of sensor system 322 of the vehicle 302. The radar sensors 308, 310 may generate radar data 324. In some examples, the radar data 324 may include position data of the respective points 316, 318. In some examples, information associated with the radar returns from the points 316, 318 may include information indicating a location in the environment (e.g., the location of the points 316, 318). Moreover, as in the illustrative example, when the points are associated with the second vehicle 306, the location of the second vehicle 306 may be determined. In some examples, the location information may include range and azimuth for the points 316, 318 or a location in a local coordinate system or a location in a global coordinate system. In some examples, the radar data 324 may include signal strength information. In some examples, the signal strength may include a radar cross-section (RCS) measurement. In some examples, the radar data 324 may also include speed information. For example, the speed of each of the points 316, 318 (and / or of the second vehicle 306) may be based on the frequency of the wireless energy reflected from the points 316, 318 and / or the time the reflected wireless energy was detected.

[0034] Thus, the radar data 324 may include the distance (e.g., range or radial distance) of each of the first points 316 from the radar sensor 308, the velocity (e.g., Doppler velocity) of each of the first points 316 along the distance, a strength measurement (e.g., RCS value), and / or additional information. Similarly, the radar data 324 may also include the distance (e.g., range or radial distance) of each of the second points 318 from the radar sensor 310, the velocity (e.g., Doppler velocity) of each of the second points 318 along the associated distance, strength information, and / or additional information.

[0035] In some examples, the radar data 324 is typically used to track objects. More specifically, Figure 3 illustrates that the vehicle 302 can include one or more vehicle computing devices 326 for performing functionality associated with the radar data 324. The vehicle computing device(s) 326 include a radar processing system 328 having an associated object representation generation component 330, as well as a track association component 332 and a track generation component 334.

[0036] In some examples, the radar processing system 328 typically implements functionality to receive radar data 324 from the radar sensors 308, 310 and generate object representations 336 of objects in the environment of the vehicle 302, such as a representation of the second vehicle 306 and / or other dynamic and / or static objects in the environment. In some examples, the radar processing system 328 may be a radar pipeline that processes only radar data, such as the radar data 324, to the exclusion of other sensor modalities, for example. In some examples, the radar processing system 328 may include functionality to associate returns with each other and / or with particular objects. Thus, for example, the radar processing system 328 may determine that radar returns associated with the first point 316 and the second point 318 are associated with each other and / or with the second vehicle 306. In some examples, the radar processing system 328 may also determine that other returns (e.g., in the same radar scan) are associated with other objects in the environment (e.g., the road surface 304 proximate to the second vehicle 306, other vehicles in the environment, and the like).

[0037] In some examples, the radar processing system 328 may cluster the points (e.g., the first point 316 and the second point 318) based on information from the respective radar returns. For example, the first point 316 and the second point 318 are close (e.g., within a threshold distance), and in some examples, the radar processing system 328 may determine that the points indicate a single object. In some examples, a point cluster may include multiple points that have some likelihood (e.g., level and / or degree of similarity) of identifying a single object or a grouping of objects that should be considered together (e.g., by a planning system of an autonomous vehicle).

[0038] The radar processing system 328 may be embodied as one or more data analysis structures including one or more neural networks. In some examples, the identification of points as associated with the second vehicle 306 may be performed by one or more machine learning networks. In some examples, the radar processing system 328 may include one or more neural networks that process the radar data 324 to group points and associate points with the objects just mentioned. In some examples, the network may classify points (e.g., as associated with a vehicle, a building, a pedestrian, or the like) for each return, identify associations of the points with one or more additional points, associations of the points with objects.

[0039] In some examples, the radar processing system 328 may further include an object representation generation component 330 configured to determine an object representation 336. More specifically, while the radar processing system 328 receives multiple radar points, e.g., from the radar sensors 308, 310, and makes a determination on a point-by-point basis, the object representation generation component 330 generates a single representation of the object based on the point-by-point data. In some examples, the object representation generation component 330 generates a bounding box 338 as the object representation 336. The bounding box 338 can be a two-dimensional representation of the second vehicle 306 generated by the object representation generation component 330 based on the first point 316, the second point 318, and / or other radar points. Although the bounding box 338 is illustrated as a two-dimensional bounding box, other instances of the object representation 336 can include other or different multi-dimensional representations, e.g., a three-dimensional bounding box.

[0040] In some examples, the object representation 336 may further include other attributes or characteristics of the object, such as the second vehicle 306, for example, as determined from the radar data 324. In some examples, the object representation 336 may include the extent of the sensed object (e.g., embodied as a length, width, area, or the like, of a bounding box 338). Additionally, the object representation 336 may also include a position of the bounding box 338. In some examples, the position of the bounding box 338 may be coordinates associated with a point 340 that may represent a center of the bounding box 338. Although the point 340 is illustrated as being the center of the bounding box 338, the point may be other than the center. Additionally, the object representation 336 may also include a speed of the object. In some examples, the object representation 336 may further include one or more of a classification of the object (e.g., a vehicle, a pedestrian, a wheeled pedestrian, a bicyclist, a construction vehicle, an articulated vehicle, a building, or the like). In some examples, the object representation 336 may also include a confidence level associated with the representation.

[0041] In some examples, the object representation 336 may be a unique representation of the object (e.g., the second vehicle 306) based on multiple radar points. The vehicle computing device(s) 326 may use the object representation 336 to track an object, such as the second vehicle 306. As used herein, "tracking an object" generally relates to determining the movement of an object (e.g., with respect to the vehicle 302) over time. In some examples, the vehicle computing device(s) 326 may include functionality to generate and / or receive information about the track of the object (e.g., as track data). In some examples, the track may generally describe attributes of the path or trajectory of the object in the environment of the vehicle 302. In some examples, the track may be a series of measured and / or predicted postures or states of the object (e.g., with respect to the vehicle 302). In some examples, the track may include a series of multi-dimensional representations (e.g., two-dimensional or three-dimensional bounding boxes) generated at a predetermined frequency to represent / predict the movement of the object.

[0042] In some examples, the track association component 332 includes functionality to determine whether the object representation 336 should be associated with an existing track (e.g., of a previously sensed object). For example, the track association component 332 can include functionality to compare the object representation 336 to the track data. For example, attributes of the representation of the second vehicle 306 (e.g., attributes of the bounding box 338) may be compared to the track data to determine whether the second vehicle 306 has already been tracked. In some examples, the comparison of the object representation 336 to the track information can include comparing the track speed to the speed of the object representation, the track position to the position of the object representation, and the like.

[0043] The track generation component 334 can include functionality to update a previously generated track. For example, if the track association component 332 determines that an object representation 336 is associated with a track (e.g., the object representation 336 represents an object that has already been tracked), the track generation component 334 can update the track using the object representation 336 (e.g., by predicting future movements of the second vehicle 306 using the object representation 336).

[0044] In some examples, the track generation component 334 can create a new track (e.g., in instances where an object, such as, for example, the second vehicle 306, is newly detected). For example, in instances where the object representation 336 does not match an existing track, the track generation component 334 can use the object representation 336 to generate a new track. In some examples, the track generation component 334 can receive multiple instances of the object representation 336 to generate a new track (e.g., based on different radar scans and / or radar scans from different times). In some examples, the updated track information and / or new track information generated by the track generation component 334 can be used to control the vehicle 302 (e.g., to proceed with respect to a tracked object, such as, for example, the second vehicle 306). Additional examples of associating tracks and generating new tracks can be found, for example, in U.S. patent application Ser. No. 17 / 364,491, entitled “Associating Radar Data with Tracked Objects,” filed June 30, 2021, the entirety of which is incorporated herein by reference in its entirety for all purposes.

[0045] 4 is a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, the system 400 may include a vehicle control system 402 of a vehicle, such as an autonomous vehicle.

[0046] The vehicle control system 402 may include one or more vehicle computing devices 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, one or more drive systems 414, and a user interface 416.

[0047] The vehicle computing device(s) 404 may include one or more processors 418 and a memory 420 communicatively coupled to the one or more processors 418. In the illustrated example, the vehicle control system 402 is an autonomous vehicle, however, the vehicle control system 402 could be any other type of vehicle. In the illustrated example, the memory 420 of the vehicle computing device 404 stores a localization component 422, a perception component 424, a planning component 426, one or more system controllers 428, a sensor processing system 430, a track association component 432, and a track generation component 434. Although depicted in FIG. 4 as residing in memory 420 for illustrative purposes, it is contemplated that the localization component 422, the perception component 424, the planning component 426, the one or more system controllers 428, the sensor processing system 430, the track association component 432, and / or the track generation component 434 may additionally or alternatively be accessible to the vehicle control system 402 (e.g., stored in or otherwise accessible by memory separate from the vehicle control system 402).

[0048] In at least one example, the localization component 422 can include functionality to receive data from the sensor system(s) 406 to determine a position and / or orientation of the vehicle control system 402 (e.g., one or more of an x-position, a y-position, a z-position, a roll, a pitch, or a yaw). For example, the localization component 422 can include and / or request / receive a map of the environment and can continuously determine the location and / or orientation of the autonomous vehicle within the map. In some cases, the localization component 422 can utilize simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneous techniques, relative SLAM, bundle adjustment, non-linear least squares optimization, or the like to receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, and the like to accurately determine the location of the autonomous vehicle. In some cases, the localization component 422 can provide data to various components of the vehicle control system 402 to determine an initial position for the autonomous vehicle in order to generate candidate trajectories, as described herein.

[0049] In some cases, the perception component 424 can include functionality for performing object detection, segmentation, and / or classification. In some examples, the perception component 424 can provide processed sensor data indicative of the presence of an entity proximate to the vehicle control system 402 and / or the classification of the entity as a type of entity (e.g., car, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). In additional and / or alternative examples, the perception component 424 can provide processed sensor data indicative of one or more characteristics associated with the detected entity and / or the environment in which the entity is located. In some examples, the characteristics associated with the entity can include, but are not limited to, x position (global position), y position (global position), z position (global position), orientation (e.g., roll, pitch, yaw), type of entity (e.g., classification), velocity of the entity, acceleration of the entity, extent (magnitude) of the entity, etc. Without limitation, characteristics associated with the environment may include the presence of other entities in the environment, the state of other entities in the environment, the time of day, the day of the week, the season, weather conditions, darkness / light indications, etc. As a non-limiting example, the perception component 424 may generate object representations 336 from the radar data, as described herein.

[0050] The planning component 426 can determine a path that the vehicle control system 402 will follow to traverse through an environment. The planning component 426 can determine various routes and trajectories and various levels of detail. For example, the planning component 426 can determine a route to travel from a first location (e.g., a current location) to a second location (e.g., a target location). For purposes of this discussion, the route can be a sequence of waypoints to travel between the two locations. As non-limiting examples, the waypoints include streets, intersections, GPS (Global Positioning System) coordinates, etc. Furthermore, the planning component 426 can generate instructions to guide the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning system 426 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the instructions can be a trajectory, or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon technique, and one of the multiple trajectories is selected for the vehicle control system 402 to proceed.

[0051] In at least one example, the vehicle computing device 404 may include one or more system controllers 428 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of the vehicle control system 402. The just mentioned system controller(s) 428 may communicate with and / or control corresponding systems of the drive system(s) 414 and / or other components of the vehicle control system 402.

[0052] The sensor processing system 430 can be the radar processing system 328 described above, but can also be, for example, the lidar multi-layer perceptron 114 and / or the vision multi-layer perceptron 122. Typically, the sensor processing system 430 can include functionality to receive sensor data (e.g., vision, lidar, radar, and the like) and generate a representation of an object from the sensor data, for example, as the object representation 336. For example, the sensor processing system 430 may receive sensor data including a plurality of points and information associated with the points, including location information, signal strength information, speed information, or the like, for the points. The sensor processing system 430 may use one or more processing models, algorithms, or the like on the received sensor data to determine an object representation, such as, for example, the object representation 336. Each of the object representations can be a single representation generated from multiple radar points associated with the same sensed object. In other words, the sensor processing system 430 generates a single representation of an object based on the sensor data deemed to be associated with the object. The sensed object representation may be a multi-dimensional, e.g., two-dimensional or three-dimensional bounding box, having associated attributes of the sensed object, including, but not limited to, speed, position, classification, and / or other aspects of the orientation or state of the sensed object. Moreover, the sensor processing system 430 may generate one or more probabilities, confidence values, and / or the like associated with the object representation 336 and / or aspects or attributes of the object representation 336.

[0053] The track association component 432 may be the same as or similar to the track association component 332. The track association component 432 typically includes functionality to associate a sensed object representation (e.g., generated from radar data, lidar data, vision data, and the like) with track information of an object that has already been tracked. For example, the track association component 432 may include functionality to compare aspects of a sensed object representation (e.g., one of the object representations 336) with a tracked object representation that may be part of a track. Additional details associated with the operation of the track association component 432 are described in connection with FIG. 3 and throughout this disclosure as well.

[0054] The track generation component 434 can be the same as or similar to the track generation component 334. The track generation component 434 typically includes functionality to receive object representations 336 from radar data, lidar data, vision data, and the like, and update existing tracks or create new tracks based thereon. For example, if the track association component 432 determines that the sensed object is associated with an existing track, the track generation component can generate updated track information (e.g., to add or update the existing track). In some examples, if the object representation associated with the sensed object does not correspond to an existing track, the track generation component 434 generates a new track to associate with the sensed object. Additional details associated with the operation of the track generation component 434 are described in connection with FIG. 3 and throughout this disclosure as well.

[0055] While shown separate from other components for clarity and ease of reference, the functionality of the sensor processing system 430, track association component 432, and / or track generation component 434 may be performed by other aspects of the vehicle control system 402. Without limitation, one or more of the components may be incorporated into the perception component 424. Aspects of the present disclosure provide improved functionality due at least in part to the use of a unique representation of multiple radar returns, regardless of the system, component, or system that uses the data in accordance with the techniques detailed herein.

[0056] In at least one example, the sensor system(s) 406 can include a radar sensor as described herein. In further examples, the sensor system(s) 406 can include a lidar sensor, an ultrasonic transducer, a SONAR sensor, a location sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), a camera (e.g., RGB, IR, intensity, depth, time-of-flight, etc.), a microphone, a wheel encoder, an environmental sensor (e.g., temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), etc. The sensor system(s) 406 can include multiple instances of each of the just mentioned or other types of sensors. For example, as described herein, implementations of the present disclosure can use multiple scans from multiple sensors with overlapping fields of view, e.g., from multiple radar sensors. Thus, for example, the autonomous vehicle control system 402 can include multiple radar sensors. In an additional example, the lidar sensors may include individual lidar sensors located on the corners, front, rear, sides, and / or top of the vehicle control system 402. As another example, the camera sensors may include multiple cameras positioned at various locations around the exterior and / or interior of the vehicle control system 402. The sensor system(s) 406 may provide input to the vehicle computing device 404. Additionally or alternatively, the sensor system(s) 406 may send sensor data over one or more networks 436 to one or more computing devices at a particular frequency, at predetermined times, in near real-time, etc.

[0057] The emitter(s) 408 may be configured to emit light and / or sound. The emitter(s) 408 in the example just described include internal audio and visual emitters that communicate with the passengers of the vehicle control system 402. By way of example and not limitation, the internal emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. In some examples, one or more of the internal emitters may be used to signal passengers that the vehicle is approaching or has arrived in an unmapped area and that continued travel in the unmapped area requires permission and / or manual control. Additionally or alternatively, the internal emitters may alert the passenger(s) that a teleoperator or other external source (e.g., a waiting passenger) has taken manual control of the vehicle control system 402. Additionally, the emitter(s) 408 in the examples just described can also include external emitters. By way of example and not limitation, the external emitters in the examples just described can include one or more lights that signal a direction to go or other indications regarding the operation of the vehicle (e.g., indicator lights, signs, light arrays, etc.), and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that audibly communicate with pedestrians or other nearby vehicles, including acoustic beam steering technology.

[0058] The communication connection(s) 410 may enable communication between the vehicle control system 402 and one or more other local or remote computing devices. For example, the communication connection(s) 410 may facilitate communication with other local computing device(s) in the vehicle control system 402 and / or with the drive system(s) 414. Additionally, the communication connection(s) 410 may also enable the vehicle to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.). Additionally, the communication connection(s) 410 may enable the vehicle control system 402 to communicate with remote teleoperated computing devices or with other remote controllers.

[0059] The communication connection(s) 410 may include physical and / or logical interfaces for connecting the vehicle computing device(s) 404 to another computing device or to a network, such as, for example, network(s) 436. For example, the communication connection(s) 410 may enable Wi-Fi-based communications, such as, for example, frequencies defined by the IEEE 802.11 standard, short-range wireless frequencies such as, for example, Bluetooth, cellular communications (e.g., 2G, 4G, 4G LTE, 5G, etc.), or via any suitable wired or wireless communications protocol that enables each computing device to interface with other computing device(s).

[0060] In at least one example, the vehicle control system 402 may include a drive system(s) 414. In some examples, the vehicle control system 402 may have a single drive system 414. In at least one example, if the vehicle control system 402 has multiple drive systems 414, the individual drive systems 414 may be located at opposing ends (e.g., front and rear, etc.) of the vehicle control system 402. In at least one example, the drive system(s) 414 may include one or more sensor systems that detect conditions of the drive system(s) 414 and / or conditions surrounding the vehicle control system 402. By way of example and not limitation, the sensor system(s) may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive system, cameras or other imaging sensors, ultrasonic sensors that acoustically detect objects surrounding the drive system, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive system(s) 414. In some cases, the sensor system(s) 406 of the drive system(s) 414 may overlap or supplement a corresponding system (e.g., sensor system(s) 406) of the vehicle control system 402.

[0061] The drive system(s) 414 can include many vehicle systems, including a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use by other vehicle systems, a steering system including a steering motor and steering rack (which may be electrically powered), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., a cooling system, a safety system, an on-board charging system, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). In addition, the drive system(s) 414 can receive and preprocess data from the sensor system(s) 406 and can include a drive system controller to control the operation of various vehicle systems. In some examples, the drive system controller can include one or more processors and a memory communicatively connected to the one or more processors. The memory may store one or more systems that perform various functionality of the drive system(s) 414. Additionally, the drive system(s) 414 also include one or more communication connections that enable each drive system to communicate with one or more other local or remote computing devices.

[0062] In at least one example, the direct connection 412 can provide a physical interface connecting one or more drive systems 414 to the body of the vehicle control system 402. For example, the directional connection 412 can allow for the transfer of energy, fluid, air, data, etc. between the drive system(s) 414 and the vehicle. In some cases, the direct connection 412 can also releasably secure the drive system(s) 414 to the body of the vehicle control system 402.

[0063] The user interface 416 may include one or more devices, buttons, and / or control panels by which a passenger can communicate with the vehicle control system 402. In a non-limiting example, a passenger of the vehicle control system 402 may control the functionality of the vehicle control system 402 through interaction(s) with the user interface 416. In other examples, the user interface 416 may include a microphone configured to receive verbal or spoken input. Typically, the user interface 416 may provide a means by which a passenger can interface with the vehicle computing device(s) 404.

[0064] In at least one example, vehicle control system 402 may be in communication with one or more computing devices 438 over one or more networks 436. For example, as described herein, vehicle control system 402 may communicate with one or more computing devices 438 over network(s) 436. In some examples, vehicle control system 402 may receive control signals from computing device(s) 438. In other examples, vehicle control system 402 may send information to computing device(s) 438.

[0065] In some examples, the computing device(s) 438 may include a processor(s) 440 and a memory 442 communicatively coupled to the processor(s) 440. In the illustrated example, the memory 442 of the computing device(s) 438 includes a radar component 444, a lidar component 446, a vision component 448, and a fused component 450. In some examples, the radar component 444 may include software necessary to process radar data such as illustrated in FIGS. 1-3. In some examples, the lidar component may include software necessary to process lidar data such as illustrated in FIGS. 1 and 2. In some examples, the vision component may include software necessary to process vision data such as illustrated in FIGS. 1 and 2. In some examples, the fused component may include software necessary to process fused sensor data such as illustrated in FIGS. 1 and 2.

[0066] In some cases, aspects of some or all of the components described herein may include any models, algorithms, and / or machine learning algorithms. For example, in some cases, aspects of the components in memory 420 and / or 442 may be implemented as neural network(s).

[0067] As described herein, a typical neural network is a biologically inspired algorithm that generates an output through input data through a series of connected layers. Furthermore, each layer in a neural network can include another neural network, or can include any number of layers (whether convolutional or not). As can be understood in the context of the present disclosure, a neural network can use machine learning, which can refer to a broad class of algorithms mentioned above in which an output is generated based on a learning parameter.

[0068] Although described in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, but not limited to, machine learning algorithms include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter plot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage and selection operator (LASSO), elastic nets, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CA), etc.), and other algorithms (e.g., classification and regression trees (CA), etc.). RT), Iterative Dichotomy 3 (ID3), Chi-squared Automatic Interaction Detection (CHAID), Decision Cut, Conditional Decision Tree), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average One Dependence Estimators (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), Clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms Examples of algorithms that may be used include: algorithms (e.g., perceptron, backpropagation, Hopfield networks, radial basis function networks (RBFN)), deep learning algorithms (e.g., deep Boltzmann machines (DBM), deep belief networks (DBN), convolutional neural networks (CNN), stacked autoencoders), dimensionality reduction algorithms (e.g., principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Sammon mapping, multidimensional scaling (MDS), projection pursuit, linear discriminant analysis (LDA), mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g., boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting machines (GBM), gradient boosted regression trees (GBRT), random forests), support vector machines (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc.

[0069] Additional examples of architectures include neural networks such as, for example, ResNet50, ResNet101, VGG, DenseNet, PointNet, and the like.

[0070] 5 depicts an example process 500 for determining tracks from multiple sensor modalities for output. For example, some or all of the process 500 may be performed by one or more components in FIG. 4 as described herein. For example, some or all of the process 500 may be performed by the radar component 444, the lidar component 446, the vision component 448, and / or the fused component 450.

[0071] At operation 502, the process may include receiving a radar signature from a radar sensor arrangement. Additional details associated with operation 502 are described in connection with FIGS. 1-3, as well as throughout this disclosure.

[0072] At operation 504, the process may include inputting the radar signature into a radar model. In some examples, the radar model may be a machine learning model. Additional details associated with operation 504 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0073] In operation 506, processing may include determining a confidence level (also referred to as a "confidence value" or "confidence score") associated with the radar feature and an output associated with the radar feature. In some examples, the output associated with the radar feature may be associated with track data associated with objects in the environment. In some examples, the output may be an intermediate radar output. Additional details associated with operation 506 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0074] At operation 508, the process may include receiving lidar signatures from the lidar sensor arrangement. Additional details associated with operation 508 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0075] At operation 510, the process may include inputting the lidar characteristics into a lidar model. In some examples, the lidar model may be a machine learning model. Additional details associated with operation 510 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0076] In operation 512, processing may include determining a confidence level (also referred to as a "confidence value" or "confidence score") associated with the lidar feature and an output associated with the lidar feature. In some examples, the output associated with the lidar feature may be associated with track data associated with objects in the environment. In some examples, the output may be an intermediate lidar output. Additional details associated with operation 512 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0077] At operation 514, the process may include receiving vision features from a vision sensor arrangement (e.g., an image sensor). Additional details associated with operation 514 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0078] At operation 516, the process may include inputting the vision features into a vision model. In some examples, the vision model may be a machine learning model. Additional details associated with operation 516 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0079] At operation 518, processing may include determining a confidence level (also referred to as a "confidence value" or "confidence score") associated with the vision feature and an output associated with the vision feature. In some examples, the output associated with the vision feature may be associated with track data associated with objects in the environment. In some examples, the output may be an intermediate vision output. Additional details associated with operation 518 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0080] In operation 520, the process may include receiving and combining the intermediate radar output, the intermediate lidar output, and the intermediate vision output as a fused feature. Additional details associated with operation 520 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0081] At operation 522, the process may include inputting the fused features into a fused model. In some examples, the fused model may be a machine learning model. Additional details associated with operation 522 are described in connection with FIGs. 1 and 2, as well as throughout this disclosure.

[0082] At operation 524, processing may include determining a confidence level (also referred to as a "confidence value" or "confidence score") associated with the fused feature and an output associated with the fused feature. In some examples, the output (also referred to as a "fused output" or "multi-modality output") associated with the fused feature may be associated with track data associated with objects in the environment. Additional details associated with operation 524 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0083] At operation 526, the process may include determining whether a confidence level associated with one of the radar, lidar, or vision features is above a threshold and a confidence level associated with the fused feature is below a threshold. If yes, at operation 528, the process may include transmitting an output associated with one of the radar, lidar, or vision features (e.g., track data associated with the object and based on one of the sensor modalities) to a planner system of the vehicle. If no, at operation 530, the process may include transmitting an output associated with the fused feature (e.g., track data associated with the object and based on the fused modality) to a planner system of the vehicle. In some examples, a vehicle control system may use the track data to control the vehicle. Additional details associated with operations 526-530 are described in connection with FIGS. 1 and 2, as well as throughout this disclosure.

[0084] Example clauses A: one or more processors; and when executed, the instructions include receiving first feature data from a first detector trained to detect objects based on radar data, receiving second feature data from a second detector trained to detect objects based on at least one of lidar data or camera data, inputting the first feature data into a first model, inputting the second feature data into a second model, receiving a first confidence score from the first model based at least in part on the first feature data, receiving a second confidence score from the second model based at least in part on the second feature data, receiving a first intermediate output from the first model, receiving a second intermediate output from the second model, and calculating the first intermediate output and the second intermediate output as a third intermediate output. and one or more non-transitory computer-readable media storing instructions executable by one or more processors to cause the system to perform operations including: inputting into a model of the third model a third confidence score based at least in part on the first intermediate output and the second intermediate output, receiving from the third model a third confidence score based at least in part on the first intermediate output and the second intermediate output, determining that the third confidence score is below a threshold, determining that the first confidence score is above the threshold, determining that the first confidence score is above the threshold and the third confidence score is below the threshold, and outputting tracks representative of objects in the environment based at least in part on determining that the first confidence score is above the threshold and the third confidence score is below the threshold and based at least in part on the first intermediate output.

[0085] B: The system of paragraph A, wherein the operations further include training a first model based at least in part on a ground truth indicating whether the first feature data is associated with a ground truth object track.

[0086] C: The system of paragraph A, wherein the track includes an object identifier, and the operation further includes updating the track based at least in part on the camera data.

[0087] D: The system of paragraph A, further including outputting the track to at least one of a predictive system or a planning system, and controlling the autonomous vehicle based at least in part on the track and the first confidence score.

[0088] E: The first intermediate output is the embedding from the intermediate layer of the first model, the system of terms A.

[0089] F: A method comprising: receiving first feature data from a sensor operating via a first modality; inputting the first feature data into a first model; receiving from the first model a first modality confidence score associated with the object based at least in part on the first feature data; receiving a first intermediate output from the first model; inputting the first intermediate output into a multi-modality model; receiving from the multi-modality model a multi-modality confidence score associated with the object based at least in part on the first intermediate output; and outputting data associated with the object based at least in part on the first modality confidence score and the multi-modality confidence score.

[0090] G: The method of paragraph F, further including: determining that the first modality confidence score is higher than the multi-modality confidence score; and outputting data associated with the object based at least in part on the first modality confidence score being higher than the multi-modality confidence score.

[0091] H: The method of term G, wherein determining that the first modality confidence score is higher than the multi-modality confidence score includes determining that the multi-modality confidence score is below a threshold and determining that the first modality confidence score is above a threshold.

[0092] I: The method of paragraph F, where the data associated with the object is a track representing the object.

[0093] J: The method of paragraph I, further including receiving a multi-modality output from the second model and generating a track representing the object based at least in part on the first intermediate output or the multi-modality output.

[0094] K: The method of paragraph I, further including: receiving a multi-modality output from the second model; updating a track representing the object with at least one of the first intermediate output or the multi-modality output based at least in part on the first modality confidence score and the multi-modality confidence score; and outputting the track representing the object to at least one of the prediction system or the planning system based at least in part on the first modality confidence score and the multi-modality confidence score.

[0095] L: The method of claim F, further including controlling the autonomous vehicle based at least in part on the data associated with the object.

[0096] M: The first model is a multi-layer perceptron, the method of section F.

[0097] N: The method of paragraph F, wherein the first feature data is associated with at least one of radar data, lidar data, or camera data.

[0098] O: The method of clause F, wherein the sensor is a first sensor, further comprising: receiving second feature data from a second sensor operating in a second modality different from the first modality; inputting the second feature data into a second model; receiving from the second model a second modality confidence score associated with the object based at least in part on the second feature data; receiving a second intermediate output from the second model; inputting the second intermediate output into a multi-modality model; receiving from the multi-modality model a multi-modality confidence score associated with the object based at least in part on the first intermediate output and the second intermediate output; and outputting data associated with the object based at least in part on the first modality confidence score, the second modality confidence score, and the multi-modality confidence score. P: One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the processor to perform operations including receiving first feature data from a sensor operating via a first modality; inputting the first feature data into a first model; receiving from the first model a first modality confidence score associated with the object based at least in part on the first feature data; receiving a first intermediate output from the first model; inputting the first intermediate output into a multi-modality model; receiving from the multi-modality model a multi-modality confidence score associated with the object based at least in part on the first intermediate output; and outputting data associated with the object based at least in part on the first modality confidence score and the multi-modality confidence score.

[0099] Q: A non-transitory computer-readable medium of term P, wherein data associated with an object is a track that represents the object.

[0100] R: A non-transitory computer-readable medium of term Q, the operations further including receiving a multi-modality output from the second model and generating a track representing the object based at least in part on the first intermediate output or the multi-modality output.

[0101] S: A non-transitory computer-readable medium of term Q, wherein the operations further include receiving a multi-modality output from the second model, updating a track representing the object with at least one of the first intermediate output or the multi-modality output based at least in part on the first modality confidence score and the multi-modality confidence score, and outputting the track representing the object to at least one of the prediction system or the planning system based at least in part on the first modality confidence score and the multi-modality confidence score.

[0102] The non-transitory computer-readable medium of term P, further comprising: receiving second feature data from a second sensor operating in a second modality different from the first modality; inputting the second feature data into a second model; receiving from the second model a second modality confidence score associated with the object based at least in part on the second feature data; receiving a second intermediate output from the second model; inputting the second intermediate output into a multi-modality model; receiving from the multi-modality model a multi-modality confidence score associated with the object based at least in part on the first intermediate output and the second intermediate output; and outputting data associated with the object based at least in part on the first modality confidence score, the second modality confidence score, and the multi-modality confidence score.

[0103] Although the exemplary clauses described above are described with respect to one particular implementation, it should be understood in the context of this document that the contents of the exemplary clauses may also be implemented via a method, device, system, computer-readable medium, and / or another implementation. Additionally, any of the examples A-T may be implemented alone or in combination with any other one or more of the examples A-T.

[0104] termination Although one or more examples of the techniques described herein have been described, various alternatives, additions, permutations and equivalents fall within the scope of the techniques described herein.

[0105] In describing the examples, reference is made to the accompanying drawings, which form a part hereof, which show, by way of example, specific examples of the claimed subject matter. It is understood that other examples can be used and that modifications or alternatives, such as structural changes, can be made. The above examples, modifications, or alternatives do not necessarily depart from the scope of the intended and claimed subject matter. Although steps herein may be given in a certain order, in some cases the order can be changed such that certain inputs are provided at different times or in a different order without changing the functionality of the described systems and methods. Furthermore, the disclosed procedures could be performed in a different order. Additionally, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orderings for the calculations could be implemented without difficulty. In addition to being reordered, in some cases the calculations could also be decomposed into sub-calculations having the same results.

Claims

1. receiving first feature data from a sensor operating via a first modality; inputting the first feature data into a first model; receiving, from the first model, a first modality confidence score associated with the object based at least in part on the first feature data; receiving a first intermediate output from the first model; inputting the first intermediate output into a multi-modality model; receiving, from the multi-modality model, a multi-modality confidence score associated with the object based at least in part on the first intermediate output; outputting data associated with the object based at least in part on the first modality confidence score and the multi-modality confidence score; A method comprising:

2. determining that the first modality confidence score is higher than the multi-modality confidence score; outputting the data associated with the object based at least in part on the first modality confidence score being higher than the multi-modality confidence score; The method of claim 1 further comprising:

3. Determining that the first modality confidence score is higher than the multi-modality confidence score includes: determining that the multi-modality confidence score is below a threshold; determining that the first modality confidence score is above the threshold; and 3. The method of claim 2, comprising:

4. The data associated with the object is a track representing the object, and the method comprises: receiving a multi-modality output from the second model; generating the track representing the object based at least in part on the first intermediate output or the multi-modality output; The method of claim 1 further comprising:

5. receiving a multi-modality output from the second model; updating the track representing the object with at least one of the first intermediate output or the multi-modality output based at least in part on the first modality confidence score and the multi-modality confidence score; outputting the track representing the object to at least one of a prediction system or a planning system based at least in part on the first modality confidence score and the multi-modality confidence score; The method of claim 4 further comprising:

6. controlling an autonomous vehicle based at least in part on the data associated with the object. The method of claim 1 further comprising:

7. 2. The method of claim 1, wherein the first model is a multi-layer perceptron.

8. The method of claim 1 , wherein the first feature data is associated with at least one of radar data, lidar data, or camera data.

9. the sensor is a first sensor; receiving second feature data from a second sensor operating in a second modality different from the first modality; inputting the second feature data into a second model; receiving, from the second model, a second modality confidence score associated with the object based at least in part on the second feature data; receiving a second intermediate output from the second model; inputting the second intermediate output into the multi-modality model; receiving, from the multi-modality model, a multi-modality confidence score associated with the object based at least in part on the first intermediate output and the second intermediate output; outputting the data associated with the object based at least in part on the first modality confidence score, the second modality confidence score, and the multi-modality confidence score; The method of claim 1 further comprising:

10. One or more non-transitory computer-readable media storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations that implement the method of any one of claims 1 to 9.

11. 1. A system comprising: one or more processors; one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to: receiving first feature data from a sensor operating via a first modality; inputting the first feature data into a first model; receiving, from the first model, a first modality confidence score associated with the object based at least in part on the first feature data; receiving a first intermediate output from the first model; inputting the first intermediate output into a multi-modality model; receiving, from the multi-modality model, a multi-modality confidence score associated with the object based at least in part on the first intermediate output; outputting data associated with the object based at least in part on the first modality confidence score and the multi-modality confidence score; one or more non-transitory computer-readable media for causing the computer to perform operations including: A system comprising:

12. The system of claim 11 , wherein the data associated with the object is a track representing the object.

13. The operation is receiving a multi-modality output from the second model; generating the track representing the object based at least in part on the first intermediate output or the multi-modality output; The system of claim 12 further comprising:

14. The operation is receiving a multi-modality output from the second model; updating the track representing the object with at least one of the first intermediate output or the multi-modality output based at least in part on the first modality confidence score and the multi-modality confidence score; outputting the track representing the object to at least one of a prediction system or a planning system based at least in part on the first modality confidence score and the multi-modality confidence score; The system of claim 12 further comprising:

15. the sensor is a first sensor, and the operation includes: receiving second feature data from a second sensor operating in a second modality different from the first modality; inputting the second feature data into a second model; receiving, from the second model, a second modality confidence score associated with the object based at least in part on the second feature data; receiving a second intermediate output from the second model; inputting the second intermediate output into the multi-modality model; receiving, from the multi-modality model, a multi-modality confidence score associated with the object based at least in part on the first intermediate output and the second intermediate output; outputting the data associated with the object based at least in part on the first modality confidence score, the second modality confidence score, and the multi-modality confidence score; 15. The system of claim 11, further comprising:

Images