Object detection and tracking

Abstract

Tracking the current and / or previous position, velocity, acceleration, and / or orientation of an object using sensor data may include determining whether to associate a current object detection generated from recently received (e.g., current) sensor data with a previous object detection generated from previously received sensor data. In other words, tracking may identify an object detected in the previous sensor data as the same object detected in the current sensor data. However, various types of sensor data may be used to detect objects, and some objects may not be detected or may be detected differently by different sensor types, confounding attempts to track the objects. An ML model may be trained to receive outputs related to tracking associated with different sensor types and / or objects and determine a data structure including a region of interest, an object classification, and / or a pose associated with the object.

Description

[Technical field]

[0001] The present disclosure relates to object detection and tracking. [Background technology]

[0002] This application claims the benefit of U.S. Provisional Application No. 62 / 926,423, filed October 26, 2019, and U.S. Nonprovisional Application No. 16 / 779,576, filed January 31, 2020, both of which are incorporated herein in their entireties.

[0003] An autonomous vehicle may use sensors to capture data about the environment it traverses. The autonomous vehicle uses this sensor data to detect objects in the environment to avoid collisions. However, an autonomous vehicle may include two or more different types of sensors, and accordingly, the sensor data may vary significantly in form and content, such that detections produced by two different sensor types may differ. Small discrepancies between detections determined in association with two different sensor types may cause jitter (i.e., "flying around") and / or flicker (i.e., appearing and disappearing) in the object representation created by the vehicle. Also, some sensor types, such as depth cameras, may be subject to large errors in depth measurements, further complicating tracking of objects. This may hinder the safe navigation of the vehicle and the training of machine learning (ML) models. Furthermore, techniques for reducing discrepancies and / or smoothing object representations or data associated therewith may consume computing bandwidth and / or memory. [Brief description of the drawings]

[0004] The detailed description will be set forth with reference to the accompanying drawings, in which the leftmost digit of a reference number identifies the drawing in which the reference number first appears. Use of the same reference number in different drawings indicates similar or identical components or functions.

[0005] [Figure 1] FIG. 1 illustrates an example scenario in which an autonomous vehicle configured with perception and tracking components may track previous and current positions, speeds, and / or orientations of objects in an environment surrounding the autonomous vehicle and generate, at least in part, a trajectory for controlling the autonomous vehicle based on the tracking. [Diagram 2] 1 is a block diagram of an example system including a perception component and a tracking component. [Diagram 3] FIG. 1 is a block diagram illustrating at least a portion of a perception architecture including components for achieving the improved object tracking techniques discussed herein. [Figure 4] FIG. 1 is a pictorial flow diagram of an exemplary process for determining putative object detections based on a perception pipeline object environment representation / object detection that may be generated based at least in part on sensor data. [Diagram 5] FIG. 1 is a pictorial flow diagram of an example process for tracking objects in an environment based at least in part on a putative object detection determined by an ML model, as discussed herein. [Figure 6] FIG. 1 is a block diagram of an example architecture of an ML model for generating putative object detections. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0006] Techniques for tracking current and / or previous positions, velocities, accelerations, or orientations of objects using sensor data may include determining whether to associate a current object detection generated from recently received (e.g., current) sensor data with a previous object detection generated from previously received sensor data. In other words, tracking may identify that an object detected in the previous sensor data is the same object detected in the current sensor data. In some examples, tracking may additionally or alternatively include various current and / or previous data regarding the object that is useful for a planning component of an autonomous vehicle to predict the object's motion / behavior and determine a trajectory and / or path for controlling the autonomous vehicle. For example, tracking may additionally or alternatively include a representation of an area of ​​the environment currently and / or previously occupied by the object, an object classification associated with the object (e.g., vehicle, large vehicle, pedestrian, bicycle), a current and / or previous direction associated with the object, a current and / or previous velocity and / or acceleration of the object, and a current position and / or velocity of the object, although any other parameters are contemplated.

[0007] In some examples, one or more sensors of a sensor type may be associated with a pipeline (e.g., a sequence of operations; steps; a network or layers thereof; machine learning models; hardware such as analog-to-digital converters, amplifiers, field programmable gate arrays (FPGAs), application specific integrated circuits (ASIC(s), and / or the like)) associated with that sensor type and used to determine information about objects contained in the associated sensor data. The sensor data may be received from one or more sensors of that type, and the pipeline (sometimes referred to herein as a perception pipeline) may generate a representation of an environment based at least in part on the sensor data. For simplicity, the collective output of the pipelines is referred to herein as an environment representation. The environment representation may include one or more object detections and may include one or more output types. For example, the video pipeline 302 may output an environment representation 308 based at least in part on video data 310 (e.g., sensor data including one or more RGB images, thermal images).

[0008] The environment representation may include object detection, which may include an indication and / or tolerance that a portion of the environment is occupied by an object, a region of interest (ROI) associated with the detected object (e.g., a bounding box, mask, segmentation, or some other identification that an area of ​​the environment is occupied by an object), an object classification of the object, an orientation of the object, a velocity and / or acceleration of the object, a height of the object, and / or the like. The environment representation may additionally or alternatively include a voxelized representation (a discretized three-dimensional representation of such data) and / or a top-down segmentation of the sensor data, as well as a top-down representation of the environment, as discussed in more detail in U.S. Patent Application No. 15 / 963,833, which is incorporated herein in its entirety. It is noted that U.S. Patent Application No. 15 / 963,833 may additionally or alternatively be modified to include the regressions discussed herein, and may be modified by backpropagating losses associated with these additional or alternative regressions. In some examples, the regions of interest discussed herein may be three-dimensional ROIs and / or two-dimensional ROIs (e.g., a top-down / bird's-eye view of the ROI). In some examples, additional environmental representations may be generated by perception pipelines associated with different sensor types.

[0009] In some examples, multiple object detections may be generated associated with the same object in the environment. These multiple object detections may be generated by different perception pipelines, which may be associated with different sensor types. For example, a lidar perception pipeline may receive lidar data and determine an object detection associated with the object, a hybrid lidar video perception pipeline may receive lidar and video data and generate different object detections associated with the same object, a video perception pipeline may receive imagery from a camera and generate additional object detections associated with the same object, etc. Due to differences in the sensors and perception pipelines, the object detections associated with the same object generated by the different pipelines may differ from each other. For example, a first object detection may indicate a ROI associated with the object that has different dimensions than a ROI indicated by a second object detection associated with the same object. Furthermore, one pipeline may generate an object detection associated with an object while another pipeline may not generate a detection at all. For example, a radar or lidar pipeline may detect an object, but the object may be out of the camera's field of view.

[0010] In some examples, the techniques discussed herein may include aggregating at least a portion of the environment representations associated with different sensor types and providing the aggregated data as input to a trained ML model to output a putative object detection. In some examples, the aggregated data may be represented in a multi-channel image, with different channels associated with different sensor types and / or different types of perception data for which the perception data was generated. For example, the aggregated data may include a lidar, video, and / or radar occupancy grid {e.g., a grid having discrete portions, such as pixels, indicating whether a corresponding location in the environment is occupied or not according to a perception pipeline associated with the respective sensor data type}, a top-down display of ROIs generated in association with the lidar, video, and / or radar, an object classification associated with a portion of the environment, a probability that a portion of the environment is occupied, a yaw of a detected object, and / or the like. See U.S. Patent Application Serial No. 16 / 591,518 for occupancy maps, which is incorporated herein by reference in its entirety. In some examples, the occupancy grid may extend to a maximum height that may correspond to the height of the autonomous vehicle plus a buffer. In other words, the occupancy grid may indicate occupancy of a portion of the environment below the maximum height. For example, traffic lights and signs placed on a road may exceed a maximum height and therefore the occupancy grid will not show that they occupy part of the environment.

[0011] In some examples, the aggregated data may additionally or alternatively include data from a remote computing device and / or non-cognitive based data such as, for example, map data (e.g., road data, driveable surface locations, destinations), weather data, traffic notifications (e.g., congestion, collisions, lane changes, construction, speed changes), safety notifications (e.g., environmental hazard locations, disaster locations, road surface conditions, visibility conditions), and / or the like. In some examples, the remote computing device may be another autonomous vehicle, a third party service, a distributed computing device, a remote sensor, and / or the like.

[0012] In some examples, the aggregated data from the different pipelines may further include at least a portion of the environment representation for one or more prior times. For example, perception pipelines associated with different sensor types may be synchronized to generate environment representations at the same frequency (e.g., every 100 ms, every 500 ms, every 1 sec). In examples where the perception pipelines are synchronized to output environment representations at substantially the same time, every 500 ms, the aggregated data may include data for 0 ms (i.e., the set of aggregated data corresponding to the most recently received set of environment representations), -500 ms, 1 sec, etc., as described above. In at least one example, the aggregated data may include a multi-channel image including aggregated data for time steps 0 ms, -100 ms, -200 ms, -300 ms, and / or 400 ms, and / or 1 sec, -2 sec, and / or 3 sec, although any suitable time steps may be used. For example, the aggregated data may include object detections received from different pipelines for the current and the last n time steps, where n is a positive integer (e.g., 1, 2, 3, 4, 5, 10, and / or the like).

[0013] In some examples, one or more previously generated tracks may be provided as input to the ML model in addition to the aggregated data (e.g., as channels of the aggregated data). For example, tracks may be generated in association with every time step or at intervals (e.g., every eleventh time step). The previously generated tracks may be included as channels in a multi-channel image. In some examples, the tracks may include the center, range, and / or yaw of a ROI associated with the object, as well as an identifier that associates the ROI with other previous regions of interest generated in association with the object.

[0014] In some examples, the ML model may be trained to output a final environment representation that may include one or more putative object detections. As described above, object detections associated with an object may differ in dimension, location, or even presence between different pipelines. The final environment representation may include a single putative object detection for each object that is determined based at least in part on the object detections received from the different pipelines (e.g., received as inputs to the ML model as part of the aggregated data). For example, a putative object detection generated by an ML model may include a ROI (e.g., a region associated with an object) that identifies a portion of the environment as being occupied, a predicted ROI associated with a future time, a speed associated with the ROI, an object classification associated with the ROI (e.g., vehicle, pedestrian, large vehicle, bicycle), a classification of the speed of the ROI (e.g., static or dynamic), an orientation (e.g., yaw) associated with the ROI, and / or orientation bins (e.g., two bins centered at 0 degrees and 180 degrees; four bins centered at 0 degrees, 90 degrees, 180 degrees, and 270 degrees; this output may further include a distance from the bin center), and / or a height (e.g., height of the detected object) associated with the ROI. In some examples, any region of interest may be generated based at least in part on the output of the confidence layer, such as following a non-maximum suppression technique. See, for example, U.S. Patent Application Nos. 16 / 013764 and 16 / 013729, which are incorporated herein in their entireties.

[0015] In some examples, the output of the ML model may include an image in which each pixel stores the above information. For example, the final environment representation may include a top-down representation of the environment, such as an image. The image may indicate the data described above. The pixels of the image may indicate that a portion of the environment is occupied, the object classification of the object occupying the portion, the velocity of the object, etc. Object detections may be indicated in the final environment as regions of pixels (or other discrete portions of the final environment) that are indicated as occupied and / or associated with the above data (e.g., object classification, object velocity, object orientation).

[0016] In some examples, the techniques may further include determining whether to associate a previously generated track with one of the putative object detections of the final environment representation and / or whether to generate a new track. In some examples, the techniques may include receiving a previous track and projecting an ROI associated with the previous track from a previous time frame associated with the previous track to a current and / or future time frame. For example, this may include modifying the representation of the position and / or orientation of the ROI. In additional or alternative examples, the predicted ROI may include a predicted ROI output by the ML model at a previous time step. The techniques may include determining the relevance of the projected ROI with any of the putative object detections generated by the ML model. For example, the degree of relevance may include determining the degree of overlap (e.g., Intersection Over Union (IOU)), Munkres assignment score, size comparison, feature comparison (e.g., is the projected ROI associated with the same object classification as the putative object detection, is there a large disparity between the speed associated with the projected ROI and the predicted speed associated with the putative object detection), and / or the like. In some examples, the determination of the degree of relevance may be encoded into a scoring function, which may be machine learning and / or deterministic. For example, the IOU may be determined deterministically and used as an input to an ML model that determines a score for the degree of relevance, along with a representation of the location and size of the projected ROI relative to the estimated object detection ROI.

[0017] If the association meets or exceeds the threshold, the techniques may include associating the previous tracking with the putative object detection. However, if the association does not meet the threshold, the techniques may include testing the putative object detection with any other projected ROIs (e.g., other ROIs that overlap or are within a threshold distance of the putative object detection) and / or associating and generating a new tracking if no projected ROI matches the putative object detection. In additional or alternative examples, additional or alternative data may be used to determine when the putative object detection corresponds to a previous object detection / tracking, although a previous tracking may be associated with the putative object detection or a new tracking may be more simply generated based on the IOU alone.

[0018] The techniques discussed herein may improve the vehicle's safety by improving the vehicle's ability to predict the motion and / or behavior of objects around the vehicle and / or by continuing to account for objects that may be temporarily occluded. The techniques may also increase the operating range of the perception system (e.g., increased to 40 meters, 60 meters, or 80 meters from the vehicle). In at least some examples, the techniques may further reduce the computational bandwidth for determining a track associated with an object, especially when different sensor modalities generate regions of interest of different sizes, when depth estimates associated with the sensor modalities vary, when objects are over-segmented and / or under-segmented. While other techniques using diverse sensor modalities to track objects may require comparing object detections from each sensor modality to previous tracks, the instant technique involves comparing estimated object detections determined by an ML model to previous tracks. The techniques may improve the accuracy of object detection (e.g., object location, segmentation) and / or tracking. Tracks generated using the techniques discussed herein have less jitter because the underlying object detection is more stable.

[0019] (Example scenario) FIG. 1 illustrates an exemplary scenario 100 including a vehicle 102. In some examples, the vehicle 102 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire trip with no driver (or passenger) expected to control the vehicle at any time. However, in other examples, the vehicle 102 may be a fully or partially autonomous vehicle having any other level or classification. It is contemplated that the techniques discussed herein may be applied to more robotic, such as for autonomous vehicles. For example, the techniques discussed herein may be applied to mining, manufacturing, augmented reality, and the like. Additionally, although the vehicle 102 is depicted as a land vehicle, the vehicle 102 may be a spacecraft, a seaplane, and / or the like. In some examples, the vehicle 102 may be represented in a simulation as a simulated vehicle. For simplicity, the discussion herein does not distinguish between simulated and real-world vehicles. Thus, references to a "vehicle" may refer to a simulated vehicle and / or a real-world vehicle.

[0020] In accordance with the techniques discussed herein, the vehicle 102 may receive sensor data from sensors 104 of the vehicle 102. For example, the sensors 104 may include position sensors (e.g., Global Positioning System (GPS) sensors), inertial sensors (e.g., acceleration sensors, gyro sensors, etc.), magnetic sensors (e.g., compasses), position / speed / acceleration sensors (e.g., speedometers, driveline sensors), depth position sensors (e.g., lidar sensors, radar sensors, sonar sensors, Time of Flight (ToF) cameras, depth cameras, ultrasonic and / or sonar sensors, and / or other depth-sensing sensors), image sensors (e.g., cameras), audio sensors (e.g., microphones), and / or environmental sensors (e.g., barometers, hygrometers, etc.).

[0021] The sensors 104 may generate sensor data, which may be received by a computing device 106 associated with the vehicle 102. However, in other examples, some or all of the sensors 104 and / or computing devices 106 may be separate from and / or located remotely from the vehicle 102, and data capture, processing, command, and / or control may be communicated to / from the vehicle 102 by one or more remote computing devices via wired and / or wireless networks.

[0022] The computing device 106 may include a memory 108 that stores the perception component 110, the planning component 112, the tracking component 114, and / or the system controller 116. Although depicted in FIG. 1 for illustrative purposes, it should be understood that the tracking component 114 may reside on a separate computing device (or otherwise) from any one or more of the other components. Typically, the perception component 110 may determine what is in the environment surrounding the vehicle 102, and the planning component 112 may determine how to operate the vehicle 102 according to the information received from the perception component 110. For example, the planning component 112 may determine a trajectory 118 based at least in part on the perception data and / or other information, such as, for example, one or more maps, orientation information (e.g., the location of the vehicle 102 in the environment relative to the maps and / or features detected by the perception component 110), etc. Trajectory 118 may include instructions for controller 116 to operate drive components of vehicle 102 to achieve a steering angle and / or steering rate, which may result in a vehicle position, vehicle speed, and / or vehicle acceleration. For example, trajectory 118 may include a target heading, a target steering angle, a target steering rate, a target position, a target speed, and / or a target acceleration for controller 116 to track. Perception component 110, planning component 112, and / or tracking component 114 may include one or more machine learning (ML) models and / or other computer-executable instructions.

[0023] In some examples, controller 116 may include software and / or hardware for operating drive components of vehicle 102 sufficient to track trajectory 118. For example, controller 116 may include one or more Proportional-Integral-Derivative (PID) controllers.

[0024] In some examples, the perception component 110 may receive sensor data from the sensors 104 and determine data related to objects in the vicinity of the vehicle 102 (e.g., object classification, instance segmentation, semantic segmentation, two-dimensional and / or three-dimensional bounding boxes, tracking related to detected objects), route data specifying the vehicle's destination, global map data identifying road features (e.g., features detectable by different sensor modalities useful for localizing an autonomous vehicle), local map data identifying features detected in the vicinity of the vehicle (e.g., positions and / or dimensions of buildings, trees, fences, fire hydrants, stop signs, and any other features detectable by various sensor modalities), tracking data (e.g., environment representations, object detection and / or tracking as discussed herein), and / or the like.

[0025] In some examples, the perception component 110 may include a hardware and / or software pipeline, which may include one or more GPUs, ML models, Kalman filters, and / or the like. In some examples, the perception component 110 may monitor as much of the environment around the autonomous vehicle as possible, which may be limited by sensor capabilities, object and / or environmental occlusions (e.g., buildings, elevation differences, objects in front of other objects), and / or environmental effects such as fog, snow, etc. For example, sensor data may include lidar and / or radar data, which the perception component 110 may receive as input. The perception component 110 may be configured to detect as much information about objects and the environment as possible to avoid not considering events or object behaviors that should be considered by the planning component 112 in determining the trajectory 118.

[0026] The object classification determined by the perception component 110 may distinguish between different object types, such as, for example, a passenger vehicle, a pedestrian, a bicycle, a delivery truck, a semi-truck, a traffic sign, and / or the like. Tracking may include past, current, and / or predicted object position, speed, acceleration, and / or orientation. The data generated by the perception component 110 may be referred to as perception data. Once the perception component 110 generates the perception data, the perception component 110 may provide the perception data to the planning component 112. In some examples, the perception data may include the output of a sensor specific pipeline (e.g., video, lidar, radar) and / or a hybrid sensor pipeline (e.g., video-lidar, radar-lidar). The perception data may additionally or alternatively include tracking, estimated object detection, and / or a final environment representation, as discussed herein.

[0027] The planning component 112 may use the sensory data received from the perception component 110 to determine one or more trajectories to control the movement of the vehicle 102 to traverse a path or route and / or otherwise control the operation of the vehicle 102, although any such operation may be performed by various other components (e.g., localization may be performed by the localization component, which may be based at least in part on the sensory data). For example, the planning component 112 may determine a path for the vehicle 102 from a first location to a second location; substantially simultaneously, and based at least in part on the sensory data and / or simulated sensory data (which may further include predictions regarding objects detected in the data), generate a number of potential trajectories for controlling the operation of the vehicle 102 according to a Receding Horizon technique (e.g., one microsecond, half a second) to control the vehicle to traverse the route (e.g., to avoid any of the detected objects); and select one of the potential trajectories as a trajectory 118 of the vehicle 102 that may be used to generate drive control signals that may be transmitted to drive components of the vehicle 102. FIG. 1 shows an example of such a trajectory 118, represented as arrows indicating heading, speed, and / or acceleration, but the trajectory itself may include instructions to the controller 116, which in turn may operate the drive system of the vehicle 102.

[0028] The tracking component 114 may receive one or more environment representations and / or object detections from a respective perception pipeline (e.g., a video pipeline, a lidar pipeline, a hybrid video lidar pipeline, a radar pipeline, a hybrid video lidar radar pipeline, and / or the like) and may determine whether to associate a previously generated track with the current object detection or to generate a new track associated with the current object detection. The tracking component 114 may additionally or alternatively determine a final environment representation and / or estimated object detection, which may serve as a summary of the different object detections. In some examples, the tracking component 114 discussed herein may determine an estimated object detection by inputting object detections and / or environment representations generated by one or more sensor types (e.g., discrete sensor types such as lidar, RGB camera, thermal imaging, sonar, radar, etc.; and / or hybrid sensor types such as video-lidar association, lidar radar association, etc.) into an ML model discussed herein that is trained to output a final environment representation that may include an estimated object detection. In some examples, if the relevance of the putative object detection to the projection of the previous tracking (e.g., a score based at least in part on Intersection Over Union (IOU), size and / or feature comparison) meets or exceeds a threshold, the putative object detection may be associated with the previous tracking rather than making a decision based on the raw object detection. If the IOU is below the threshold, the raw object detection may be included in the decision.

[0029] 1 illustrates a top-down representation 120 of an environment that may be part of a final environment representation determined by the ML model of tracking component 114. Top-down representation 120 illustrates object detections, in this case illustrated as estimated ROI 122. Top-down representation 120 and / or estimated ROI 122 may be determined by the ML model of tracking component 114 based at least in part on object detections received from one or more perception pipelines. For example, the object detections provided as input to the ML model may include three-dimensional ROIs associated with image 124, one of which is illustrated as ROI 126, and top-down ROIs associated with lidar data 128 (which may be, for example, two-dimensional and / or three-dimensional), one of which is illustrated as ROI 130.

[0030] The estimated ROI 122 may be part of an estimated object detection generated by the ML model in association with an object identified by at least one of the perception pipelines. The estimated ROI 122 is depicted to include a center 132 and an extent (black line). The extent may be defined based at least in part on an anchor box associated with an object classification determined by the ML model in association with the estimated object detection. In additional or alternative examples, the estimated ROI may include any other identification of a portion of the top-down representation, such as another form of bounding shape or mask.

[0031] FIG. 1 also illustrates a previous ROI 134 associated with a track 136 previously generated by the tracking component 114 in association with one or more object detections. In some examples, the track 136 may associate one or more previous object detections and / or indicate data associated therewith, such as speed, acceleration, orientation, object classification, unique identifier, occlusion state (e.g., whether an object is currently / previously partially or fully occluded from one or more sensors). The previous ROI 134 may be associated with any of the data discussed herein as being associated therewith (e.g., occupancy map, top-down segmentation, object classification, object orientation, object velocity, and / or acceleration). For example, the previous ROI 134 may be associated with the object 138 and the object classification 140 "vehicle". For illustration, the previous ROI 134 may be associated with a recent object detection associated with the track 136.

[0032] Techniques discussed herein may include projecting the previous ROI 134 to a time relative to the estimated ROI 122, which may include predicting a position, orientation, velocity, acceleration, and / or the like of an object relative to the previous ROI 134 based on the track 136 and data associated therewith. Figure 1 depicts such a projection 142 of the previous ROI 134 with a dashed line. In the depicted example, the estimated ROI 122 is also associated with the vehicle, such that the IOU between the estimated ROI 122 meets an association threshold and / or the IOU between the projection 142 and the estimated ROI 122 meets or exceeds an IOU threshold.

[0033] Regardless, the track may include at least the previous ROI, whose center, range, and estimated object detection may include estimated ROI 122. In some examples, the techniques discussed herein may include projecting the previous ROI based at least in part on the speed and / or orientation associated with the track. If the IOU, center-to-center distance, inset-to-inset distance, or Munkres allocation of the projected ROI 142 relative to the estimated ROI 130 meets a threshold, the estimated object detection may be added to the track 136. If not, the estimated object detection may be associated with a new track and / or an alternative tracking component may use the raw detection to generate a track.

[0034] (Example System) FIG. 2 illustrates a block diagram of an example system 200 for implementing the techniques discussed herein. In some examples, the example system 200 may include a vehicle 202, which may represent the vehicle 102 of FIG. 1. In some examples, the vehicle 202 may be an autonomous vehicle configured to operate according to a Level 5 classification issued by the National Highway Traffic Safety Administration, which describes a vehicle capable of performing all safety-critical functions for the entire journey with no driver (or passenger) expected to control the vehicle at any time. However, in other examples, the vehicle 202 may be a fully or partially autonomous vehicle having any other level or classification. Additionally, in some examples, the techniques described herein may also be usable by non-autonomous vehicles.

[0035] Vehicle 202 may include vehicle computing device 204, sensors 206, emitters 208, network interfaces 210, and / or drive components 212. Vehicle computing device 204 may represent computing device 106, and sensor 206 may represent sensor 104. System 200 may additionally or alternatively include computing device 214.

[0036] In some examples, the sensors 206 may represent the sensors 104 and may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, position sensors (e.g., Global Positioning System (GPS), compass, etc.), inertial sensors (e.g., Inertial Measurement Units (IMU), accelerometers, magnetometers, gyroscopes, etc.), image sensors (e.g., Red Green Blue (RGB), Infrared (IR), intensity, depth, time of flight cameras, etc.), microphones, wheel encoders, environmental sensors (e.g., thermometers, hygrometers, light sensors, pressure sensors, etc.), etc. The sensors 206 may include multiple instances of each of these sensors or other types of sensors. For example, the radar sensors may include individual radar sensors located at the corners, front, back, sides, and / or top of the vehicle 202. As another example, the cameras may include multiple cameras located at various locations on the exterior and / or interior of the vehicle 202. The sensors 206 may provide input to the vehicle computing device 204 and / or the computing device 214.

[0037] The vehicle 202 may also include emitters 208 for emitting light and / or sound, as described above. The emitters 208 in this example may include interior audio and video emitters for communicating with passengers of the vehicle 202. By way of example and not by way of limitation, the interior emitters may include speakers, lights, signs, display screens, touch screens, haptic emitters (e.g., vibration and / or force feedback), mechanical actuators (e.g., seat belt tensioners, seat positioners, head rest positioners, etc.), and the like. The emitters 208 in this example may also include exterior emitters. By way of example and not by way of limitation, the exterior emitters in this example may mean lights (e.g., indicator lights, signs, light arrays, etc.) for announcing a direction of travel or other indication of the vehicle's operation, and one or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) for audibly communicating with pedestrians or other nearby vehicles, one or more of which may include acoustic beam steering technology.

[0038] Vehicle 202 may include a network interface 210 that enables communication between vehicle 202 and one or more other local or remote computing devices. For example, network interface 210 may facilitate communication with other local computing devices on vehicle 202 and / or drive components 212. Network interface 210 may also additionally or alternatively enable the vehicle to communicate with other nearby computing devices (e.g., other nearby vehicles, traffic signals, etc.). Network interface 210 may additionally or alternatively enable vehicle 202 to communicate with computing device 214. In some examples, computing device 214 may include one or more nodes of a distributed computing system (e.g., a cloud computing architecture).

[0039] Network interface 210 may include physical and / or logical interfaces for connecting vehicle computing device 204 to another computing device or network, such as network 216. For example, network interface 210 may enable Wi-Fi based communications, such as over frequencies defined by the IEEE 200.11 standard, short-range wireless frequencies such as Bluetooth, cellular communications (e.g., 2G, 3G, 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol by which each computing device can interface with other computing devices. In some examples, vehicle computing device 204 and / or sensors 206 may transmit sensor data over network 216, at specific frequencies, after a predetermined period of time, in near real-time, to computing device 214.

[0040] In some examples, the vehicle 202 includes one or more drive components 212. In some examples, the vehicle 202 may have a single drive component 212. In some examples, the drive component 212 may include one or more sensors for detecting conditions surrounding the drive component 212 and / or the vehicle 202. By way of example and not limitation, the sensors of the drive component 212 may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive component, inertial sensors (e.g., inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.) that measure the orientation and acceleration of the drive component, cameras or other image sensors, ultrasonic sensors that acoustically detect objects in the vicinity of the drive component, lidar sensors, radar sensors, etc. Some sensors, such as wheel encoders, may be unique to the drive component 212. In some cases, the sensors on the drive component 212 may overlap or supplement corresponding systems (e.g., sensors 206) of the vehicle 202.

[0041] The drive components 212 may include a high voltage battery, a motor to propel the vehicle, an inverter to convert direct current from the battery to alternating current for use in other vehicle systems, a steering system including a steering motor and a steering rack (which may be electric), a braking system including hydraulic or electric actuators, a suspension system including hydraulic and / or pneumatic components, a stability control system to distribute braking force to mitigate loss of traction and maintain control, an HVAC system, lighting (e.g., lighting such as head / tail lights that illuminate the exterior surroundings of the vehicle), and one or more other systems (e.g., cooling systems, safety systems, on-board charging systems, other electrical components such as DC / DC converters, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, the drive components 212 may include a drive components controller to receive and pre-process data from sensors and control the operation of various vehicle systems. In some examples, the drive components controller may include one or more processors and a memory communicatively coupled to the one or more processors. The memory may store one or more components for performing various functions of the drive components 212. Additionally, drive components 212 may include one or more communication connections that enable each drive component to communicate with one or more other local or remote computing devices.

[0042] Vehicle computing device 204 may include a processor 218 and a memory 220 communicatively coupled to one or more of processors 218. Memory 220 may represent memory 108. Computing device 214 may also include a processor 222 and / or memory 224. Processor 218 and / or 222 may be any suitable processor capable of processing data and executing instructions to perform operations as described herein. By way of example and not limitation, processor 218 and / or 222 may include one or more central processing units (CPUs), graphics processing units (GPUs), integrated circuits (e.g., application specific integrated circuits (ASICs)), gate arrays (e.g., field programmable gate arrays (FPGAs)), and / or any other device or portion of a device that processes electronic data and transforms the electronic data into other electronic data that may be stored in a registry and / or memory.

[0043] Memory 220 and / or 224 may be an exemplary non-transitory computer-readable medium. Memory 220 and / or 224 may store an operating system and one or more software applications, instructions, programs, and / or data to implement the methods and functions attributed to the various systems described herein. In various implementations, memory may be implemented using any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile / flash memory, or any other type of memory capable of storing information. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, of which the ones shown in the accompanying figures are merely illustrative and relevant to the discussion herein.

[0044] In some examples, memory 220 and / or memory 224 may store localization component 226, perception component 228, planning component 230, tracking component 232, map 234, and / or system controller 236. Perception component 228 may represent perception component 110, planning component 230 may represent planning component 112, and / or tracking component 232 may represent tracking component 114.

[0045] In at least one example, the localization component 226 may include hardware and / or software that receives data from the sensors 206 to determine the position, velocity, and / or orientation of the vehicle 202 (e.g., one or more of x-, y-, z-position, roll, pitch, or yaw). For example, the localization component 226 may include and / or request / receive a map 234 of the environment and can continually determine the position, velocity, and / or orientation of the autonomous vehicle within the map 234. In some examples, the localization component 226 may receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., utilizing simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc., to accurately determine the location, attitude, and / or velocity of the autonomous vehicle. In some examples, the orientation component 226 may provide data to various components of the vehicle 202 to determine an initial position of the autonomous vehicle for generating a trajectory and / or for generating map data as discussed herein. In some examples, the orientation component 226 may provide the tracking component 232 with the position and / or orientation of the vehicle 202 relative to the environment and / or sensor data associated therewith.

[0046] In some examples, perception component 228 may include a primary perception system and / or a predictive system implemented in hardware and / or software. Perception component 228 detects objects in the environment surrounding vehicle 202 (e.g., identifies that an object is present), classifies objects (e.g., determines an object type associated with a detected object), segments sensor data and / or other representations of the environment (e.g., identifies portions of sensor data and / or representations of the environment as associated with a detected object and / or object type), determines characteristics associated with the objects (e.g., tracks that identify current, predicted, and / or previous positions, headings, speeds, and / or accelerations associated with an object), and / or the like. Data determined by perception component 228 is referred to as perception data.

[0047] The planning component 230 may receive the position and / or orientation of the vehicle 202 from the orientation component 226 and / or the sensory data from the perception component 228, and may determine instructions for controlling operation of the vehicle 202 based at least in part on any of this data. In some examples, determining the instructions may include determining the instructions based at least in part on a format associated with a system with which the instructions are associated (e.g., a first instruction for controlling operation of the autonomous vehicle may be formatted in a first format of messages and / or signals (e.g., analog, digital, aerodynamic, kinematic) that the system controller 236 and / or the drive component 212 can analyze / execute, a second instruction of the emitter 208 may be formatted according to a second format associated therewith).

[0048] The tracking component 232 may operate on the vehicle 202 and / or on the computing device 214. In some examples, the tracking component 232 may be upstream (providing inputs) in the pipeline from the planning component 230 and downstream (receiving inputs) from at least some of the perception component 228. The tracking component 232 may be configured to pass all, some, or all of the output of the tracking component 232 to the planning component 230. In some examples, the tracking component 232 may be part of the perception component 228. In some examples, the tracking component 232 may include an ML model as discussed herein that determines a final environment representation that may include an estimated object detection. The tracking component 232 may additionally or alternatively include a first component that receives the estimated object detection generated by the ML model and at least one "raw" object detection generated in association with a sensor type (e.g., one of the inputs to the ML model) and / or a second component that receives one or more object detections associated with the estimated object detection and tracking. In any event, the first component and / or the second component of the tracking component may determine whether to associate the putative object detection with a track or whether to generate a new track in association with the object detection. The tracking component 232 may additionally or alternatively output speed, orientation, and / or other historical or current characteristics of one or more object detections, all or some of which may be at least a part of a track.

[0049] Memory 220 and / or 224 may additionally or alternatively store a mapping system (e.g., generating a map based at least in part on the sensor data), a planning system, a ride management system, etc. Although localization component 226, perception component 228, planning component 230, tracking component 232, map 234, and / or system controller 236 are illustrated as being stored in memory 220, any of these components may include processor-executable instructions, machine learning models (e.g., neural networks), and / or hardware, and any of these components may be stored in whole or in part on memory 224 or configured as part of computing device 214.

[0050] As described herein, the localization component 226, the perception component 228, the planning component 230, the tracking component 232, and / or other components of the system 200 may include one or more ML models. For example, the localization component 226, the perception component 228, the planning component 230, and / or the tracking component 232 may each include a different ML model pipeline. In some examples, the ML model may include a neural network. An exemplary neural network is a biologically inspired algorithm that passes input data to a series of connected layers to generate an output. Each layer of the neural network may include another neural network or may include any number of layers (convolutional or not). As may be understood in the context of the present disclosure, the neural network may utilize machine learning, which may refer to a broad class of such algorithms in which an output is generated based on learned parameters.

[0051] Although discussed in the context of neural networks, any type of machine learning consistent with this disclosure may be used. For example, machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally weighted scatterplot smoothing (LOESS)), instance-based algorithms (e.g., ridge regression, least absolute shrinkage selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g., classification and regression trees (CART), iterative binary tree 3 (ID3), chi-squared automated interaction detection (CH2), etc.). AID), decision stumps, conditional decision trees), Bayesian algorithms (e.g., Naïve Bayes, Gaussian Naïve Bayes, Multinomial Naïve Bayes, Average Ordinary Dependent Classifier (AODE), Bayesian Belief Networks (BNN), Bayesian Networks), clustering algorithms (e.g., k-means, k-medians, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g., Perceptron, Backpropagation, Hopfield Networks, Radial Basis Function Network (RBFN)), deep learning algorithms (Deep Boltzmann Machine (DBM), Deep Belief Networks (DBN), Convolutional Neural Networks (CNN), Stacked Auto-Encoders), dimensionality reduction algorithms (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Sammon Mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA), Mixture Discriminant Analysis (MDA), Quadratic Discriminant Analysis (QDA), Flexible Discriminant Analysis (FDA)), ensemble algorithms (e.g., Boosting, Bootstrapped Aggregation (Bagging), AdaBoost, Stacked Generalization (Blending), Gradient Boosting Machines (GBM), Gradient Boosted Regression Trees (GBRT), RandomExamples of architectures may include, but are not limited to, Recurrent Forest, Support Vector Machine (SVM), supervised learning, unsupervised learning, semi-supervised learning, etc. Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, etc. In some examples, the ML models discussed herein may include PointPillars, SECOND, top-down feature layers (see, e.g., U.S. Patent Application No. 15 / 963833, which is incorporated herein in its entirety), and / or VoxelNet. Latency optimization of architectures may include MobilenetV2, Shufflenet, Channelnet, Peleenet, and / or the like. The ML models may include residual blocks such as Pixor in some examples.

[0052] Memory 220 may additionally or alternatively store one or more system controllers 236 that may be configured to control steering, propulsion, braking, safety, emitter, communication, and other systems of vehicle 202. These system controllers 236 may communicate with and / or control corresponding systems of drive component 212 and / or other components of vehicle 202. For example, planning component 230 may generate instructions based at least in part on sensory data generated by perception component 228, validate the sensory data, and / or send instructions to system controller 236. System controller 236 may control operation of vehicle 202 based at least in part on instructions received from planning component 230.

[0053] 2 is illustrated as a distributed system, it should be noted that in alternative examples, components of vehicle 202 may be associated with computing device 214 and / or components of computing device 214 may be associated with vehicle 202. That is, vehicle 202 may perform one or more of the functions associated with computing device 214, and vice versa.

[0054] (Example System) FIG. 3 is a block diagram of an example perception system 300 for determining an estimated object detection from one or more object detections received from different perception pipelines and / or determining an update or new track based at least in part on the estimated object detection. The example perception system 300 may include any number of pipelines for determining characteristics of the surroundings of the perception system 300. For example, the pipelines may determine what is in the environment and / or sensors of the autonomous vehicle and / or characteristics of objects within the environment. Although FIG. 3 depicts three pipelines: a video pipeline 302, a radar pipeline 304, and a lidar pipeline 306, any number and types of pipelines are contemplated. For example, the pipelines may be associated with the same type of sensor (e.g., one or more cameras, one or more lidar sensors, one or more radar sensors), direction of field of view (e.g., one or more cameras with overlapping fields of view), diverse types of sensors (e.g., a hybrid video lidar pipeline), data from other sources (e.g., data from a remote computing device, map data stored in memory), and / or the like.

[0055] Regardless, one or more of the pipelines may be configured to receive sensor data and determine one or more outputs. For simplicity, the collective outputs of the pipelines are referred to herein as an environment representation. The environment representation may include one or more object detections and may include one or more output types. For example, the video pipeline 302 may output an environment representation 308 based at least in part on the video data 310 (e.g., sensor data consisting of one or more RGB images, thermal images). The environment representation 308 generated by the video pipeline 302 may include one or more object detections, at least one of which includes the following: - 2D ROI associated with an object A three-dimensional ROI associated with the object, such as one of those depicted at 312, as described in more detail in U.S. Patent Application Nos. 15 / 970,838 and 16 / 386,249, the entireties of which are incorporated herein by reference. Depth estimates associated with objects and / or ROIs (e.g., estimates of distance from the sensor / vehicle to the object), as discussed in U.S. Patent Application Serial No. 15 / 970,838 -The height of the object, as discussed in US patent application Ser. No. 15 / 970838. - Object classifications and / or margins associated with the objects, where margins indicate the posterior probability that the sensor data is associated with an object of a type specified by the object classification (e.g., vehicle, pedestrian, articulated vehicle, animal, child) - In some examples, the object classifications and / or margins may be associated with individual portions of the sensor data, such as pixels -Inserting video The position (e.g., center and / or extent) of the object relative to the environment representation (e.g., a voxelization of the environment, a mesh representation of the environment). - The object's pose (which may include the object's position and / or orientation). an occupancy map 314 that may contain parts that are indicated as being occupied by objects (e.g. whether there is an object occupying a pixel / voxel, instance segmentation); and / or - An occlusion grid (e.g., containing the probability that parts of the environment are hidden from line of sight to one or more of the visual sensors).

[0056] The radar pipeline 304 may determine an environment representation (not shown to save drawing space) that includes: - an occupancy map, which may include the parts shown as being occupied by the object - an occlusion grid (e.g., including the probability that a portion of the environment is hidden from line of sight to one or more of the radar sensors), as discussed in more detail in U.S. Patent Application No. 16 / 407,139, which is incorporated herein by reference in its entirety - The yaw rate associated with the object, as discussed in more detail in U.S. Patent Application No. 16 / 416,686, which is incorporated herein by reference in its entirety. - Velocity relative to the object (e.g. ground range rate relative to the object) - Radar Cross Section (RCS), and / or - ROI and / or object classification

[0057] The lidar pipeline 306 may determine, based at least in part on the lidar data 316, an environment representation 318 that includes: - an occupancy map 320 that may include portions that are indicated as being occupied by objects; -Occlusion grid (e.g., including the probability that parts of the environment are hidden from line of sight to one or more of the lidar sensors) - Classification of the ROI and / or object associated with the object; and / or - Top-down segmentation (e.g., a top-down indication of whether a portion of the environment is occupied and / or a top-down ROI (e.g., the lidar top-down ROI depicted at 322, etc.) as discussed in more detail in U.S. Patent Application Serial No. 15 / 963,833, which is incorporated herein in its entirety)

[0058] The pipelines illustrated and discussed with respect to FIG. 3 are merely exemplary. Perception system 300 may include more, fewer, and / or different pipelines than those illustrated and discussed herein. Additionally, the pipelines may be intermixed rather than neatly separated as depicted. For example, at least a portion of the environment representation described above may be a product of portions of two different pipelines. It should be noted that, in the depicted example, specific details regarding a radar environment representation are not depicted due to limited space and expanding the visual and lidar environment representations for clarity, but that radar pipelines and radar environment representations are additionally or alternatively contemplated.

[0059] The various environment representations generated by the different pipelines may be aggregated into a multi-channel data structure 324. For example, this aggregation may include projecting the data into a common representation of the environment, such as a common reference frame and / or a voxel space having the same dimensions, a mesh representation, etc. The aggregation may additionally or alternatively include determining a top-down representation for one or more portions of the environment representation, such as projecting a three-dimensional ROI from a top-down perspective into a two-dimensional ROI, and / or determining a top-down two-dimensional ROI based at least in part on a two-dimensional ROI associated with a sensor perspective, depth, and / or object classification. In some examples, the multi-channel data structure 324 may include a multi-channel image, where each channel of the image may be associated with a different pipeline and / or a different type of output (e.g., occupancy map, occlusion grid, ROI, object classification). For example, a first channel of the image may include pixels that indicate whether each portion of the environment is occupied / unoccupied as determined by the vision pipeline, a second channel of the image may include pixels that indicate whether each portion of the environment is occupied / unoccupied as determined by the lidar pipeline, a third channel may include pixels that indicate whether each portion of the environment is associated with a certain object classification, etc.

[0060] The multi-channel data structure 324 may include the output of one or more perception pipelines, and / or data received from other sources (e.g., from a memory of the autonomous vehicle, from a remote computing device). For example, the multi-channel data structure 324 may include an environment representation (which may be, e.g., two-dimensional, three-dimensional, or higher dimensional), which may include a top-down representation of the environment, an occupancy map of the environment, a representation of occluded portions of the environment, an ROI associated with the object, an object classification associated with the object, a sensor data segmentation, a three-dimensional discretized representation of the sensor data, a yaw and / or yaw rate associated with the object, a velocity and / or acceleration associated with the object, a set of ranges (e.g., dimensions), and / or a ground height estimate associated with the object, as discussed in more detail in U.S. patent application Ser. No. 16 / 698,055, filed Nov. 27, 2019, which is incorporated herein in its entirety.

[0061] As used herein, an object detection may be part of one or more environment representations that indicate the presence of an object, such as an ROI, a positive occupancy indication, an object classification, etc.

[0062] In some examples, the previous tracking 326 and / or previous environment representation 328 may be part of the multi-channel data structure 324 or may otherwise be provided as input to the ML architecture 330. For example, the previous tracking 326 and / or previous environment representation 328 may be associated with the last n (n is a positive integer) time steps in which an environment representation was generated by the pipeline and / or a tracking was determined. For example, the system may be synchronized to determine the environment representation and / or tracking at a time interval (e.g., every 10 ms, every 100 ms, every 500 ms) or every time interval. The tracking and / or environment representation determined at the previous n time steps. In additional or alternative examples, for example when the perception system is semi-synchronous or asynchronous, the previous tracking 326 and / or previous environment representation 328 may be associated with a time window. For example, this data may be accumulated on a rotational basis, over a 2 second period, a 3 second period, a 5 second period, and / or any other time window suitable for the application.

[0063] The multi-channel data structure 324 may be provided as an input to an ML architecture 330, which may be trained to determine a final environment representation including one or more putative object detections 332. For example, the ML architecture 330 may determine a top-down representation of the environment including an indication that a portion of the environment is occupied, an ROI and / or object classification associated with the occupied portion (e.g., an object), an orientation of the object (e.g., yaw and / or yaw / orientation bins), a velocity associated with the object (e.g., stationary / moving, lateral and / or longitudinal velocity), a height associated with the object, and / or a predicted ROI associated with a future time step.

[0064] The final environment representation and / or its putative object detections 332 may be provided as input to a tracking component 334. The tracking component 334 may be trained and / or configured to determine whether to associate the putative object detections with a previous tracking 336, a new tracking, or forward one or more object detections from the pipeline and / or the putative object detections 332 to the contingent tracking component. In some examples, the tracking component 334 may output updated or new tracking 338 to the planning component based at least in part on the decisions discussed herein.

[0065] In some examples, the incidental tracking component may be a tracking component configured to receive raw environment representations from the pipelines and determine tracking therefrom. The incidental tracking component may require more computation and / or memory in some cases because it uses more data across different pipelines to determine whether an object detection is a false positive and / or whether to associate an object detection with a previous track or a new track.

[0066] (Example Process) 4 is a pictorial flow diagram of an example process 400 for determining a final environment representation and / or putative object detection from multiple object detections and / or environment representations generated by one or more perception pipelines. The example process 400 may additionally or alternatively track objects within the environment based at least in part on determining to associate the putative object detection with a previous track, determining to generate a new track for association with the putative object detection, and / or determining to forward data discussed herein to a contingent tracking component.

[0067] At operation 402, the example process 400 may include receiving a first object detection associated with a first sensor type and a second object detection associated with a second sensor type, according to any of the techniques discussed herein. For example, the first object detection 404 may be part of a first environment representation 406 received from a first pipeline 408, and the second object detection 410 may be part of an nth object environment representation 412 received from an nth pipeline 414.

[0068] While the object detection and environment representation are depicted as being top-down ROIs in the figures, in at least one example, the object detection may include lidar-based object detection, radar-based object detection, and / or video-based object detection. Lidar-based object detection may include lidar occupancy maps, lidar-based 2D or 3D ROIs, top-down segmentation ROIs, top-down segmentation maps, lidar occlusion grids, and / or object classification. Radar-based object detection may include radar occupancy maps, ground-based range rates, RCS, radar-based ROIs and / or object classifications, and / or radar occlusion grids. Video-based object detection may include video occlusion grids, instance segmentation, 2D ROIs (e.g., in image space), 3D ROIs, depth, object classification, video embedding, and / or object pose (e.g., position and / or orientation). Additional or alternative object detection may be used. In some examples, some of the object detection components described above may include hybrid components, such as, for example, radar-based ROI, pose data, or 3D ROI, any of which may be lidar, radar, and / or video-based.

[0069] At operation 416, the example process 400 may include receiving a track associated with an object in the environment according to any of the techniques discussed herein. For example, the track 418 may be a track previously determined in relation to a previous time step. In the depicted example, the track 418 is associated with an object detected by the first pipeline 408 and the nth pipeline 414, as shown by object detection 404 and object detection 410. The track 418 may associate any of the object detection data from different pipelines and / or may include estimated object detections generated by the ML model from previous time steps, at least as far back as a time window set statically or set by the speed of the vehicle, the number of objects being tracked (e.g., a smaller time window to preserve data for more objects, or the same or a larger time window to track objects through occlusions for more objects), or a combination thereof. The track ultimately indicates that two different object detections generated by two different pipelines are associated with the same object and / or that two different object detections in time generated by the same or different pipelines are associated with the same object.

[0070] The depicted tracking 418 associates previous estimated object detections generated by the ML model, and in the depicted example includes four two-dimensional (top-down) ROIs (420) associated with the same vehicle in the environment surrounding the autonomous vehicle, the vehicle's pose, and the historical location of the center of the vehicle, illustrated as a line through the ROIs. The estimated object detections may additionally or alternatively include an object classification, in this case "vehicle," the vehicle's historical and / or (previous) predicted speed, per-pixel occupancy (e.g., a mask of pixels associated with the vehicle), an estimated height of the vehicle, and / or the like.

[0071] At operation 422, the example process 400 may include inputting the first object detection, the second object detection, and / or the tracking into an ML model in accordance with any of the techniques discussed herein. In some examples, the object detections and / or tracking may be aggregated into a multi-channel data structure. In some examples, some portions of the object detections may be projected into voxel space and / or rendered in a two-dimensional top-down view prior to operation 422.

[0072] It should be noted that the object detections may be provided to the ML model as part of the environment representation. For example, the environment representation 406 includes various object detections, as well as data not shown, such as object speed, estimated height, etc., as described above. In some examples, the environment representation may be aggregated and provided as an input to the ML model. In some examples, the object detections may be separated from the rest of the environment representation and provided as an input. For example, the environment representation may be a common frame of reference or may be transformed to a common frame of reference during aggregation. The pipeline may be configured to output positive object detections along with their coordinates in the common frame of reference. For example, these positive object detections may be portions of the environment representation that are associated with a likelihood of meeting or exceeding a threshold confidence. Each and any of the object detection components described above may be associated with a regressed confidence score. For example, object classification may be associated with a confidence score, ROIs may be determined based at least in part on confidence scores associated with different pixels via non-maximum suppression techniques, occupancy may be determined based at least in part on the likelihoods associated with each pixel as determined by the ML model of the respective pipeline, etc.

[0073] At operation 424, the example process 400 may include receiving an estimated object detection 426 as an output from an ML model, according to any of the techniques discussed herein. In some examples, the ML model may be trained to output a final environment representation 428 and / or an estimated object detection 426. The estimated object detection 426 may be part of the final environment representation 428 due to properties of the final environment representation 428 indicating that a portion of the environment is occupied and / or associated with an object classification. The ML model may be trained to output a ROI (e.g., center and / or extent) associated with the object, an object classification associated with the object, an estimated pose (e.g., position and / or orientation) of the object, an estimated velocity of the object, and / or an estimated height of the object. The ML model may additionally or alternatively determine a confidence score associated with any of these outputs. In some examples, the ROI may be generated based at least in part on an anchor box or any other canonical object shape associated with the object classification on which the ML model was trained.

[0074] In some examples, the final environment representation may include a multi-channel image of the top-down representation of the environment, which may include, in different channels, discrete portions such as distance to the center of the nearest ROI, extent of the ROI, the tolerance that the pixel is associated with an object (e.g., used for non-maximal suppression to generate the ROI), the tolerance that the pixel is associated with a particular object classification of multiple object classifications, orientation, velocity, and / or estimated height.

[0075] In some examples, the ML model may determine a classification associated with the orientation that indicates the bin into which the orientation falls (e.g., north, southeast, west; any other number and type of orientation bins such as 45° bins, 90° bins, 180° bins, etc.) and / or the orientation-related regression (e.g., the degree of variation from the center of the bin determined in relation to the pixel).

[0076] In some examples, the ML model may determine the speed using a classification algorithm, resulting in an output, such as static or dynamic, or speed bins (e.g., 5, 10, 15, or such mph increments, such that the bins may indicate that the object is moving at 1-10 mph in at least one example). Additionally or alternatively, the ML model may determine the speed using a regression task as an offset from the center of the speed bin (e.g., the speed value may be determined by adding / subtracting a regression from the value of the bin into which the object was classified) or as a standalone regression (e.g., the regression task directly calculates the speed). In some examples, the speed binning and / or regression task may be based at least in part on one or more previous object detections and / or tracking that are part of the input multi-channel image. Similarly, the ML model may determine an estimated height using a classification task (e.g., as a height bin) and / or a regression task (e.g., as an offset from a height bin or directly regressed height).

[0077] In some examples, training the ML model may include determining a classification loss based at least in part on a difference between the object classification, orientation bin, velocity bin, and / or estimated height bin and the respective ground truth (e.g., semantic label, ground truth instance segmentation, and / or ground truth ROI for object classification; ground truth orientation indication for orientation bin; ground truth velocity for velocity bin; ground truth height for height bin). In some examples, a total classification loss may be determined by summing these losses. In some examples, determining the classification loss may include determining a cross-entropy focal loss, as discussed in more detail in U.S. Patent Application No. 16 / 013729, which is incorporated by reference in its entirety, although other forms of losses are contemplated. In some examples, determining the classification loss of the bins may be based at least in part on a cross-entropy loss or a Huber loss.

[0078] Training the ML model may additionally or alternatively include determining one or more regression losses based at least in part on the difference between the distance to the center of the ROI, the extent of the ROI, which may be expressed in logarithmic space, the estimated direction or offset from a direction bin, the estimated velocity or offset from a velocity bin, and / or the estimated height or offset from a height bin and their respective associated ground truths. Determining the regression loss may include determining an L1 (or any other loss) loss for each of the one or more regression outputs described above, and summing the regression losses to determine a total regression loss.

[0079] Training the ML model may additionally or alternatively include determining a total loss by summing a total classification loss and a total regression loss. In some examples, the ML model may be trained to output an uncertainty associated with each task. The losses determined for the tasks may be weighted based at least in part on the uncertainty output in association with the output task, such that the greater the uncertainty, the greater the weight applied to the loss and the greater the resulting loss. This weighting may occur prior to determining the total classification loss and / or total regression loss, but may occur at another time. In some examples, the uncertainty may be based at least in part on covariance and / or may be heteroscedastic.

[0080] In some examples, the total loss, whether weighted or not, may be backpropagated through various regression and / or classification components of the ML model, which may include adjusting one or more parameters of the components to minimize the total loss. Additionally or alternatively, the total loss may be backpropagated through one or more of the pipelines or portions of the pipelines. For example, the total loss may be used to adjust one or more parameters of a top-down segmentation component, a ROI component, a direction component, a velocity component, and / or a height component of a vision pipeline, a lidar pipeline, and / or the like.

[0081] In some examples, the losses discussed herein may be determined per object detection instead of or in addition to per-pixel losses. Determining the loss per object detection may improve the accuracy of the ML model in determining a putative object detection for small objects. In some examples, the losses may be determined based at least in part on a ground truth that includes a representation of the object's outline, an instance segmentation that identifies pixels and / or sub-pixels associated with the object, and / or the like.

[0082] In some examples, the ML model may additionally or alternatively output an indication of a track with which the estimated object is associated. In some examples, the ML model may output an embedding in association with the object detection for use in determining whether to generate a new track or associate the object detection with a previous track. The embedding may in some examples be based at least in part on other outputs of the ML model. For example, the embedding may be determined by a component of the ML model that receives other outputs of the ML model (e.g., ROI, object classification, estimated direction, estimated speed, estimated height, estimated direction), but the embedding may additionally or alternatively be determined at least in part on outputs of one or more of the perception pipelines.

[0083] In additional or alternative examples, the ML model may be configured to output object detections as discussed herein for different height bins. For example, a first set of channels of a multi-channel output image may include object detections for a first height range (e.g., 0.25 meters to 2 meters), a second set of channels may include object detections for a second height range (e.g., 2 meters to 4 meters), etc. Any other height intervals may be used.

[0084] At operation 432, the example process 400 may include determining an updated (432) or new track associated with the object based at least in part on the putative object detection according to any of the techniques discussed herein. In some examples, operation 432 may additionally or alternatively be based at least in part on one or more previous tracks and / or putative object detections. In some examples, the example process 400 may include determining to update all previous tracks currently stored in memory. Updating a track may include associating one of the putative object detections with the track, indicating that the track is associated with an object that is likely to be partially or fully occluded (e.g., an occluded status flag may be changed), and / or retiring the track. Retiring a track may include indicating that the object associated with the track has been hidden for at least a threshold amount of time, that the object is likely no longer in view, and / or removing the track. Once all tracks have been updated, any remaining putative object detections not associated with the track may be passed to an alternative tracking component and / or new tracks may be generated in association therewith.

[0085] 5 is a pictorial flow diagram of an example process 500 for tracking an object in an environment based at least in part on a putative object detection determined by an ML model discussed herein. Example process 500 may be at least a portion of operation 432 of example process 400 or may be an independent process. Operation 432 may include determining an updated or new tracking associated with the object based at least in part on the putative object detection 432 according to any of the techniques discussed herein.

[0086] At operation 502, the example process 500 may include projecting at least a portion of a previous track associated with a previous time to a current time according to any of the techniques discussed herein. For example, projecting at least a portion of a previous track may include determining a projected ROI 504 based at least in part on a previous track (e.g., track 418). In some examples, the projected ROI may be based at least in part on a proximate ROI associated with the previous track 418. FIG. 5 is illustrated by filling the proximate ROI of track 418 with a black shade. The projected region of interest 504 is depicted with a dotted line and may be based at least in part on a past speed, direction, and / or position associated with track 418. In additional or alternative examples, the projected ROI may include a predicted ROI output by the ML model at a previous time step.

[0087] It should be noted that while FIG. 5 illustrates a projected ROI 504, any of the other adaptive attributes associated with the putative object detection may be projected. For example, the object classification should not change between frames, but the speed, orientation, height (e.g., due to a pedestrian sitting or ducking), ROI location, etc. may change and may be predicted / projected at least in part based on the tracking 418 and / or data associated therewith. For example, the tracking 418 may associate any of the previous putative object detections generated by the ML model and data associated therewith. Thus, while the projected ROI 504 is discussed herein, it is understood that the object classification, predicted speed, predicted direction, predicted height, predicted position, predicted orientation, and / or predicted immersion may be associated therewith. Determining the projected ROI 504 may include determining an updated / predicted immersion based at least in part on the immersion associated with the projected ROI 504 and the predicted speed, predicted orientation, predicted height, predicted position, predicted orientation, depending on which data the immersion is based on. For example, if the fitting is based on adaptive data such as velocity and position, the fitting may be updated, but if the fitting is based on raw sensor data and / or perception pipeline data, the fitting may be used without being updated.

[0088] At operation 506, the example process 500 may include determining an association between the estimated object detection and the projected ROI according to any of the techniques discussed herein. For example, the estimated object detection may be at least a portion of the estimated object detection 426 determined by the ML model discussed herein. In some examples, operation 506 may include determining a score based at least in part on an IOU between the projected ROI 504 and the estimated object detection 426, a Euclidean distance between a center of the estimated object detection 426 and the projected region of interest 504 (or previous ROI), a Munkres assignment between the projected ROI 504 and the estimated object detection 426, and / or the like. In at least one example, determining the degree of association may include determining a difference (e.g., Euclidean distance) between an immersion associated with the projected region of interest 504 and an immersion associated with the estimated object detection 426. In some examples, the immersion may take into account the object classification, size, and / or other characteristics of the object detection. The relevance may, in at least one example, include one or a combination of IOU, center-to-center distance, Munkres assignment score, and / or a sum or weighted sum of distances between embeddings, and the like.

[0089] In some examples, the relevance may include a sum or weighted sum of the IOU and the distance between the inlays. In such examples, such sum may include the first IOU, and since a high IOU indicates similarity, but a high distance between inlays indicates dissimilarity, the (normalized) distance may be inverted after it is calculated. The IOU may be inverted by subtracting the IOU from 1. Since the IOU may range between 0 and 1, and the Euclidean distance between inlays may be significantly greater than 1, the Euclidean distance between inlays may be normalized, at least in part, based on the minimum and maximum distance between the inlays in the learned inlay space. In additional or alternative examples, the logit of the IOU may be determined and scaled, at least in part, based on the maximum distance between the inlays in the learned inlay space. In additional or alternative examples, the relevance may include a score determined, at least in part, based on one or more relevances. For example, an ML model may be trained to receive the IOU, the distance between the fittings, at least a portion of the tracking, and / or at least a portion of the estimated object detection data, and output a score indicative of the margin to which the estimated object detection data is associated with a previous tracking.

[0090] At operation 508, the example process 500 may include determining whether the relevance meets a relevance threshold, according to any of the techniques discussed herein. Relevance that meets the relevance threshold indicates that the object associated with the putative object detection 426 is / is likely to be the same object as indicated by the track 418. If the relevance meets the relevance threshold, the example process 500 may proceed to operation 510. On the other hand, if the degree of relevance determined at operation 506 does not meet the degree of relevance threshold, the example process 5008 may continue to operation 442.

[0091] The threshold relevance may be met by meeting or exceeding a threshold IOU value if the relevance measures include an IOU, by finding a Munkres assignment, by the distance between embeddings being less than a threshold distance, by the distance between centers being less than a threshold distance, and / or by the combined score satisfying a threshold combined score. In accordance with the latter example, when IOU and distance between embeddings are combined to determine the score, meeting the threshold composite score may be achieved as described above by determining that the composite score is less than a composite score threshold if the IOU is inverted, and vice versa if the distance between embeddings is normalized and then inverted. In examples where multiple relevance measures are determined (e.g., ML model generated scores, IOU, distance between embeddings), operation 508 may include a rule set that specifies one or more of the relevance thresholds that must be met to pass transition to operation 510 (e.g., all relevance measures must exceed each relevance threshold, such as at least one, two out of three, etc.).

[0092] At operation 510, the example process 500 may include associating the putative object detection 426 with the track 418 as an updated track 430 according to any of the techniques discussed herein.

[0093] At operation 512, the example process 500 may include generating a new track associated with the putative object detection according to any of the techniques discussed herein and / or providing the first object detection, the second object detection, and / or the putative object detection to an alternative tracking component. Generating a new track indicates that the putative object detection is associated with an object not previously detected by any of the perception pipelines and / or that was not within the field of view of its associated sensor. In additional or alternative examples, raw object detections from one or more pipelines and / or tracking 418 may be provided as inputs to the alternative tracking component. In some examples, the alternative tracking component may be configured to determine a new track or update a previous track based at least in part on comparing object detections from the various perception pipelines.

[0094] (Example Architecture) 6 is a block diagram of an example architecture 600 of the ML model discussed herein for generating a putative object detection. In some examples, the example architecture 600 may be associated with one object classification or multiple object classifications. In other words, the example architecture 600 may receive and / or be trained on object detections related to a particular object classification, or the example architecture 600 may receive and / or be trained on object detections of various types of object classifications. In examples where the example architecture 600 is associated with one object classification, the ML model may include multiple instances of the example architecture 600 trained in association with different object classifications.

[0095] Regardless, the example architecture 600 may include a first set of layers 602, a second set of layers 604, and / or a third set of layers 606, although additional or alternative numbers of sets of layers may be used. The layers may be part of an ML model, a sub-network, and / or a separate network. In at least one example, at least one of the set of layers 602-606 may include one or more convolutional layers, which may include learning parameters adjusted at least in part based on the losses described above, and / or a deconvolutional layer (e.g., that upsamples its input). In some examples, the multi-channel image described above may be provided as an input to any one or more of the first set of layers 602, the second set of layers 604, and / or the third set of layers 606. As described above, the multi-channel image may include data from a current and / or previous time step. In some examples, the outputs of the first set of layers 602 may be provided as inputs to the aggregation layer 608 and the second set of layers 604, the outputs of the second set of layers 604 may be provided as inputs to the aggregation layer 608 and the third set of layers 606, and the outputs of the third set of layers 606 may be provided as inputs to the aggregation layer 608.

[0096] In some examples, the aggregation layer 608 may concatenate or upsample and sum element-wise the output of each layer to form a data structure. In some examples, the data structure generated by the aggregation layer 608 may be provided as input to the classification layer 610 and / or the regression layer 612. The classification layer 610 may include one or more sets of convolutional layers or other components for classification tasks discussed herein. In some examples, the output layer for a classification task may output a tensor (or other data structure) of variances, where discrete portions of the field indicate the variances with which the associated portion of the environment is associated with a classification (e.g., occupancy space, object classification, velocity bins, orientation bins, height bins). For example, a first set of layers in the classification layer 610 may determine whether each portion of the environment is occupied or unoccupied and / or associated with a respective object classification, and another set of layers in the classification layer 610 may determine whether the environment is associated with an estimated height bin, and so on. In some examples, the discrete portions of the set of object classification layers may additionally or alternatively include a margin associated with each of the object classifications on which the exemplary ML architecture 600 was trained. In other words, the classification output head may output a binary indication that a portion of the environment is or is not relevant to a classification (e.g., height bin, object classification, occupancy), or the classification output head may output a regressed value to which the NMS algorithm may be applied to determine the classification. The output layer of the regression layer 612 may output fields of values ​​of those tasks being regressed, such as ROI center offset, ROI range, velocity offset, direction offset height offset, and / or the like.

[0097] Various outputs of the classification layer 610 and / or regression layer 612 may be aggregated into an estimated object detection 614. For example, this may include subtracting an offset from the bin values, determining an ROI based at least in part on a non-maximum suppression technique applied to the classification and / or regression output, determining that two or more ROIs are associated with an articulated object based at least in part on a center offset associated with the two or more ROIs, grouping the two or more ROIs into a single object detection, normalizing the output, and / or formatting the output into a common object detection format, such as a message for transmission over a publish-subscribe network. In examples where the classification output includes bins and the regressed output includes offsets, the classification may indicate bins that may be a range of values ​​(e.g., 0-5 meters, 0-10 miles per hour) and the (regression) offset may specify a positive or negative value to be subtracted from the bin's reference point, such as may be used for yaw, yaw rate, height, velocity, acceleration, object range, and / or the like. For example, the reference point may be a bin center value (e.g., 3 meters in a bin specifying a range of 0 to 5 meters) or a bin end (e.g., 0 meters or 5 meters in a bin specifying a range of 0 to 5 meters). Note that the poses indicated as part of the data structures output by the ML models discussed herein may include the object's location, center, and / or orientation (e.g., orientation may include yaw). In some examples, the poses may additionally or alternatively include the object's range.

[0098] (Example section) A. A method comprising: receiving a first object detection associated with a first sensor type and a second object detection associated with a second sensor type, the first object detection and the second object detection identifying an object in an environment surrounding the autonomous vehicle; receiving a track associated with the object, the track identifying at least one of an estimated previous position of the object, a previous region of interest, or a previous velocity of the object; inputting at least a portion of the first object detection, the second object detection, and the track into a machine learning (ML) model; receiving from the ML model a data structure including a region of interest associated with the object, an object classification, and a pose, the pose indicating at least one of a position or yaw associated with the object; determining an updated track associated with the object based at least in part on the data structure, the updated track including at least a portion of the track, a current position, and at least one of the region of interest or yaw associated with the object; and controlling the autonomous vehicle based at least in part on the updated track.

[0099] B. The method of paragraph A, wherein the data structure further includes at least one of an indication that the object is static or dynamic, a top-down segmentation of the environment.

[0100] C. The method of any of paragraphs A or B, wherein determining the updated tracking comprises determining a degree of alignment of the region of interest to a previous region of interest, and determining that the degree of alignment meets or exceeds a degree of alignment threshold.

[0101] D. The method of any one of paragraphs A to C, wherein the first object detection and the second object detection include an initial estimate associated with the object, and the data structure comprises one or more refined estimates associated with the object.

[0102] E. The method of any one of paragraphs A to D, further including receiving a first prior object detection associated with a time prior to a time at which the first object detection was generated, receiving a second prior object detection associated with a time prior to a time at which the second object detection was generated, and inputting the first prior object detection and the second prior object detection into an ML model in addition to the first object detection, the second object detection, and the tracking.

[0103] F. The method of any one of paragraphs A to E, wherein inputting at least a portion of the first object detection, the second object detection, and the tracking includes generating a multi-channel image based on at least a portion of the first object detection, the second object detection, and the tracking, and inputting the multi-channel image to an ML model.

[0104] G. A system including one or more processors and a memory storing processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations including receiving a first output and a second output; receiving a track associated with an object in an environment, the track identifying at least one of an estimated previous position of the object, a previous region of interest, or a previous velocity of the object; inputting at least a portion of the first output, the second output, and the track into a machine learning (ML) model; receiving from the ML model a data structure including a region of interest, an object classification, and a pose associated with the object, the pose indicating at least one of a position or a yaw associated with the object; and determining an updated track associated with the object based at least in part on the data structure, the updated track including at least a portion of the track, a current position, and at least one of the region of interest or yaw associated with the object.

[0105] H. The system of paragraph G, wherein the operation further includes controlling the vehicle based at least in part on the updated tracking.

[0106] I. The system of either paragraph G or paragraph H, wherein the first output includes an initial estimate associated with the object, and the data structure includes a refined estimate associated with the object.

[0107] J. A system described in any one of paragraphs G to I, wherein the data structure further includes at least one of an indication that the object is static or dynamic, a top-down segmentation of the environment, a yaw rate, a velocity associated with the object, or an acceleration associated with the object.

[0108] K. The system of any one of paragraphs G through J, wherein the first output is an output of a first perception pipeline, the second output is an output of a second perception pipeline, and the third output indicates that portions of the environment associated with the first output and the second output are unoccupied, and the third output is provided as an input to the ML model in addition to the first output and the second output.

[0109] L. The system of any one of paragraphs G to K, wherein determining the updated tracking includes determining a degree of alignment of the region of interest relative to a previous region of interest and determining that the degree of alignment meets or exceeds a degree of alignment threshold.

[0110] M. The system of any one of paragraphs G to L, wherein at least one of the first output or the second output includes at least one of a representation of the environment from a top-down perspective, an indication that a portion of the environment is occupied, a representation of the portion of the environment that is occupied, a region of interest associated with the object, a classification associated with the object, a sensor data segmentation, a three-dimensional discretized representation of the sensor data, a yaw associated with the object, a yaw rate associated with the object, a ground height estimate, a set of ranges associated with the object, a velocity associated with the object, or an acceleration associated with the object.

[0111] N. The system of any one of paragraphs G through M, further comprising the operations of receiving a first prior environmental representation associated with a time prior to a time at which the first output was generated, the first prior environmental representation being associated with a first perception pipeline; receiving a second prior environmental representation associated with a time prior to a time at which the second output was generated, the first prior environmental representation being associated with the first perception pipeline; and inputting the first output, the second output, and the tracking, as well as the first prior environmental representation, and the second prior environmental representation, into an ML model.

[0112] O. The system of any one of paragraphs G to M, wherein inputting the first output, the second output, and at least a portion of the tracking further includes generating a multi-channel image based at least in part on the first output, the second output, and at least a portion of the tracking.

[0113] P. A non-transitory computer readable medium having processor-executable instructions stored thereon that, when executed by one or more processors, cause at least one or more processors to perform operations including receiving a first output associated with a first process and a second output associated with a second process; receiving a track associated with an object in an environment, the track identifying at least one of an estimated previous position of the object, a previous region of interest, or a previous velocity of the object; inputting at least a portion of the first output, the second output, and the track into a machine learning (ML) model; receiving from the ML model a data structure including a region of interest, an object classification, and a pose associated with the object, the pose indicating at least one of a position or yaw associated with the object; and determining an updated track associated with the object based at least in part on the data structure, the updated track including at least a portion of the track, a current position, and at least one of the region of interest or yaw associated with the object.

[0114] Q. The non-transitory computer-readable medium of paragraph P, wherein the first output includes an initial estimation associated with the object, and the data structure includes a final estimation associated with the object.

[0115] R. The non-transitory computer-readable medium of either paragraph P or paragraph Q, wherein the data structure further includes at least one of an indication that the object is static or dynamic, a top-down segmentation of the environment, a yaw rate, a velocity associated with the object, or an acceleration associated with the object.

[0116] S. The non-transitory computer-readable medium of any one of paragraphs P to R, wherein at least one of the first output or the second output includes at least one of a representation of the environment from a top-down perspective, an indication that a portion of the environment is occupied, a representation of the portion of the environment that is occupied, a region of interest associated with the object, a classification associated with the object, a sensor data segmentation, a three-dimensional discretized representation of the sensor data, a yaw associated with the object, a yaw rate associated with the object, a ground height estimate, a set of ranges associated with the object, a velocity associated with the object, or an acceleration associated with the object.

[0117] T. The non-transitory computer-readable medium of any one of paragraphs P to S, wherein inputting at least a portion of the first output, the second output, and the tracking further includes generating a multi-channel image based at least in part on at least a portion of the first output, the second output, and the tracking.

[0118] U. An autonomous vehicle comprising a system according to any one of paragraphs G through O.

[0119] V. A system including one or more processors and memory capable of processor-executable instructions that, when executed by the one or more processors, cause the system to perform operations including the methods described in any one of paragraphs A through F.

[0120] (Conclusion) Although the subject matter has been described in language specific to structural features and / or methodological acts, it is understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

[0121] The components described herein represent instructions that may be stored on any type of computer-readable medium and may be implemented in software and / or hardware. All of the methods and processes described above may be embodied in and fully automated via software code components and / or computer-executable instructions executed by one or more computers or processors, hardware, or some combination thereof. Parts or all of the methods may alternatively be embodied in specialized computer hardware.

[0122] At least some of the processes discussed herein are illustrated as logical flow graphs, each operation of which represents a sequence of operations that may be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more non-transitory computer-readable storage media that, when executed by one or more processors, cause a computer or an autonomous vehicle to perform the referenced operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations can be combined in any order and / or in parallel to perform a process.

[0123] In particular, conditional language such as "may," "would," "may," and the like, unless otherwise noted, will be understood within the context to indicate that a particular example includes a particular feature, element, and / or step and another example does not. Thus, such conditional language is not intended to generally imply that a particular feature, element, and / or step is somehow required for one or more examples, or that one or more examples necessarily include logic for determining whether a particular feature, element, and / or step should be included or performed in any particular example, with or without user input or prompting.

[0124] Conjunctions such as "at least one of X, Y, or Z" shall be understood to indicate that the item, term, etc. can be either X, Y or Z, or any combination thereof (including multiples of each element), unless expressly stated otherwise. "a" refers to the singular as well as the plural.

[0125] It should be understood that any routine descriptions, elements, or blocks in the flow diagrams described herein and depicted in the accompanying figures may represent modules, segments, or portions of code that include one or more computer-executable instructions for performing particular logical functions or elements within the routine. Alternative implementations are within the scope of the examples described herein, in which elements or functions may be omitted or performed in a different order than shown or discussed, including substantially synchronized, in reverse order, with additional operations, or omission of operations, depending on the functionality involved, as will be appreciated by those skilled in the art.

[0126] It is understood that many variations and modifications can be made to the examples described above, elements of which are among the other acceptable examples, and all such modifications and variations are intended to be included within the scope of this disclosure and protected by the following claims.

Claims

1. 1. A system comprising: one or more processors; When executed by the one or more processors, the system receiving a first output from a first perception pipeline including a first object detection and a second output from a second perception pipeline including a second object detection; receiving a previous track associated with an object in an environment, the previous track including at least one of a previous position, a previous velocity, a previous acceleration, and / or a previous orientation of the object; inputting the first output, the second output, and at least a portion of the previous tracking into a machine learning (ML) model; receiving a data structure from the ML model including a region of interest, an object classification, and a pose associated with the object, the pose indicating at least one of a position or a yaw associated with the object; determining an updated track associated with the object based at least in part on the data structure, the updated track including at least a portion of a current position of the object and one or more of the region of interest or the yaw associated with the object; a memory storing processor-executable instructions for performing operations including: A system comprising:

2. The system of claim 1 , wherein the action further comprises controlling a vehicle based at least in part on the updated tracking.

3. The system of claim 1 or 2, wherein the data structure further comprises at least one of an indication that the object is static or dynamic, a top-down segmentation of the environment, a yaw rate, a velocity associated with the object, or an acceleration associated with the object. a third output received from a third perception pipeline indicating that portions of the environment associated with the first output and the second output are unoccupied; The system of claim 1 , wherein the third output is provided as an input to an ML model in addition to the first output and the second output.

5. determining the updated trace determining a degree of alignment of said region of interest with respect to a previous region of interest; determining whether the degree of alignment meets or exceeds the degree of alignment threshold; The system according to claim 1 , comprising:

6. At least one of the first output or the second output is a representation of said environment from a top-down perspective; an indication that the portion of the environment is occupied; a representation of the portion of said environment that is occupied; a previous region of interest associated with the object; A classification associated with the object; Sensor Data Segmentation, 3D discretized representation of sensor data, A previous yaw associated with the object; A previous yaw rate associated with the object; a set of ranges associated with said object; the previous velocity associated with the object; or the previous acceleration associated with the object; The system according to claim 1 , further comprising at least one of:

7. The operation, receiving a first prior representation of the environment associated with a time prior to a time at which the first output was generated, the first prior representation of the environment being associated with the first perception pipeline; receiving a second prior representation of the environment associated with a time prior to a time at which the second output was generated, the second prior representation of the environment being associated with the second perception pipeline; inputting the first output, the second output, at least a portion of the previous tracking, as well as the first prior environment representation and the second prior environment representation into an ML model; The system of claim 1 , further comprising:

8. 8. The system of claim 1 , wherein inputting the first output, the second output, and at least a portion of the previous tracking further comprises inputting a multi-channel image based at least in part on the first output, the second output, and at least a portion of the previous tracking.

9. A non-transitory computer-readable medium having processor-executable instructions stored thereon that, when executed by one or more processors, receiving a first output from a first perception pipeline, the first output including a first object detection, and a second output from a second perception pipeline, the second output including a second object detection; receiving a previous track associated with an object in an environment, the previous track including at least one of a previous position, a previous velocity, a previous acceleration, and / or a previous orientation of the object; inputting the first output, the second output, and at least a portion of the previous tracking into a machine learning (ML) model; receiving a data structure from the ML model including a region of interest, an object classification, and a pose associated with the object, the pose indicating at least one of a position or a yaw associated with the object; determining an updated track associated with the object based at least in part on the data structure, the updated track including at least a portion of a current position of the object and one or more of the region of interest or the yaw associated with the object; 23. A non-transitory computer-readable medium that causes the at least one or more processors to perform operations comprising:

10. 10. The non-transitory computer-readable medium of claim 9, wherein the data structure further comprises at least one of an indication that the object is static or dynamic, a top-down segmentation of the environment, a yaw rate, a velocity associated with the object, or an acceleration associated with the object.

11. At least one of the first output or the second output is a representation of said environment from a top-down perspective; an indication that the portion of the environment is occupied; a representation of the portion of said environment that is occupied; a previous region of interest associated with the object; A classification associated with the object; Sensor Data Segmentation, 3D discretized representation of sensor data, A previous yaw associated with the object; A previous yaw rate associated with the object; a set of ranges associated with said object; the previous velocity associated with the object; or the previous acceleration associated with the object; 11. The non-transitory computer readable medium of claim 9 or 10, comprising at least one of:

12. 12. The non-transitory computer readable medium of claim 9, wherein inputting the first output, the second output, and at least a portion of the previous tracking further comprises inputting a multi-channel image based at least in part on the first output, the second output, and at least a portion of the previous tracking.

Images