System and method for generating information related to measured resides using sensor data

Abstract

A method includes obtaining sensor data from an environment of an autonomous vehicle, the sensor data including at least one of lidar data, radar data, or camera data. A set of features is extracted from the sensor data and processed using a machine learning model to generate, for at least a subset of the sensor data, outputs that classify respective sensor returns as corresponding to objects or non-objects and outputs indicative of attributes of the objects. The non-object includes at least one of rain, snow, fog, smoke, or exhaust gas. The sensor return classified as corresponding to the object is compared to the pre-classified object to generate a general object classification. The autonomous vehicle is controlled based on the general object classification. In response to determining that the respective sensor return does not correspond to any pre-classified object, the respective sensor return is classified as corresponding to a general object.

Description

Background Art

[0001] A fundamental challenge in autonomous vehicle technology is to collect and interpret information about the environment surrounding the vehicle and execute commands to appropriately control the vehicle's motion to safely guide the vehicle through the environment. For example, one or more sensors of an autonomous vehicle (or a vehicle equipped with autonomous vehicle sensors) can capture measurement data and use the measurement data to track and / or identify dynamic objects within the environment surrounding the vehicle. As the state of the object changes dynamically, improvements in efficiently updating the tracking and / or identification of the object by using the new measurement data are still desired. Summary of the invention

[0002] Embodiments of the present disclosure relate to systems and methods for generating information from sensor data using machine learning (ML) models, and more particularly, to systems and methods for generating object presence information and object motion information about objects that do not belong to a predetermined classification of objects (e.g., "vehicle", "pedestrian", "cyclist", "road sign", etc.) In the present disclosure, such objects may be referred to as "residual objects" or "general objects".

[0003] Embodiments related to systems and methods for improving latency and avoiding redundancy in collecting and interpreting information about the vehicle's surroundings and executing commands to appropriately control vehicle motion are disclosed herein. In some embodiments, a residual interpreter system can implement a "no measurement missed" strategy by explaining or predicting "unexplained" sensor data and performing graceful degradation to cover false negatives even when other components of the perception system (e.g., detectors or trackers) are unable to interpret or predict objects from sensor data (false negatives). In some embodiments, instead of predicting unexplained sensor data with labels, the residual interpreter system predicts the opposite, i.e., known objects in the world that the vehicle should pay attention to, and uses the labels of the unexplained sensor data as a complement to the known object labels.

[0004] In this way, uninterpreted sensor data (e.g., ground, animals, debris, balls, obstacles, etc.) can be efficiently interpreted without using relatively expensive labeling techniques. In addition, the residual interpreter system can utilize trajectory data input obtained from other components of the perception system (e.g., trackers) to remove duplicate objects from the uninterpreted sensor data, thereby not only avoiding redundancy but also more accurately interpreting the uninterpreted sensor data.

[0005] In some embodiments according to the first aspect of the present disclosure, a method includes obtaining sensor data including multiple sensor returns from an environment of an autonomous vehicle. The method may include extracting a first set of features from the sensor data. The method may include processing the first set of features using a machine learning model to generate a first output and a second output for at least a subset of the multiple sensor returns, the first output classifying the corresponding sensor returns as corresponding to an object or a non-object, and the second output indicating an attribute of the object. The method may include comparing the sensor returns classified as corresponding to the object with a plurality of pre-classified objects to generate one or more general object classifications. The method may include controlling the autonomous vehicle based at least in part on the one or more general object classifications.

[0006] In some embodiments according to the second aspect of the present disclosure, a system includes one or more processors and one or more memories operably coupled to the one or more processors. The one or more memories may store instructions, which, in response to being executed by the one or more processors, cause the one or more processors to obtain sensor data including multiple sensor returns from the environment of the autonomous vehicle. The one or more processors may extract a first set of features from the sensor data. The one or more processors may process the first set of features using a machine learning model to generate a first output and a second output for at least a subset of the multiple sensor returns, the first output classifying the corresponding sensor returns as corresponding to an object or a non-object, and the second output indicating the attributes of the object. The one or more processors may compare the sensor returns classified as corresponding to the object with a plurality of pre-classified objects to generate one or more general object classifications. The one or more processors may control the autonomous vehicle based at least in part on the one or more general object classifications. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] These and other aspects and features of the present embodiment will become clear to those of ordinary skill in the art after reading the following description of the specific embodiment in conjunction with the accompanying drawings, in which:

[0008] Figure 1 is a block diagram illustrating an example of a system environment for an autonomous vehicle according to some implementations;

[0009] Figure 2 is a block diagram illustrating an example of a computing system for training a neural network model according to some embodiments;

[0010] Figure 3 is a block diagram illustrating an example of a computing system for testing a neural network model according to some embodiments;

[0011] Figure 4 is a block diagram illustrating an example of a computing system according to some implementations;

[0012] Figure 5 is a block diagram illustrating an example of a perception subsystem according to some embodiments;

[0013] Figure 6 is a block diagram illustrating an example of a residual interpreter system according to some implementations;

[0014] Fig. 7A is a graph illustrating the distribution of different types of foreign matter or debris (FOD); and Figure 7B is a graph illustrating precision / recall (PR) curves for different operating points of a residual interpreter system according to some embodiments; and

[0015] Figure 8 is a flow chart illustrating an example method for generating information about measured remainders according to some implementations. DETAILED DESCRIPTION

[0016] According to certain aspects, embodiments in the present disclosure relate to techniques for training a machine learning (ML) model and generating data using the trained ML model, and more particularly to systems and methods for training an ML model and generating object presence information and object motion information regarding measurement data corresponding to uninterpreted or general objects.

[0017] In order to appropriately control (or "plan") the movement of an autonomous vehicle to guide the vehicle through its current environment, a system (e.g., a perception system) may capture measurement data from one or more sensors of an autonomous vehicle (or a vehicle equipped with autonomous vehicle sensors) and use this data for "perception," e.g., tracking and / or identifying dynamic objects within the vehicle's surroundings. As the state of objects changes dynamically, the tracking and / or identification of objects needs to be updated, e.g., by using new measurement data. Conventional perception systems update trajectories associated with objects by scanning data measured from the entire scene around the vehicle, resulting in significant delays in updating the trajectories. For example, conventional perception systems may need to collect new measurement data from the entire scene around the vehicle and match or associate the new measurement data with the corresponding trajectories of the objects to identify the locations of all of these objects in the new scene.

[0018] Perception systems can track: (1) objects (or actors) such as vehicles, pedestrians, and cyclists, (2) “occluders”—things that motion planning should ignore (e.g., rain, snow, fog, smoke, exhaust, etc.), and / or (3) “objects”—matter that has not yet been interpreted as objects or occluders (e.g., animals, debris, balls, obstacles, etc.). However, generating trajectories of such occluders or objects can be a complex process that requires interactions among multiple perception modules (e.g., trackers, detectors, etc.). There are several challenges in implementing perception systems.

[0019] First, it can be challenging to explicitly label obstructions, train obstruction classifiers, and interpret sensor data that is considered an obstruction because it can be difficult to enumerate and label all types of obstructions. For example, ghost returns may appear from retro-reflectors on the ground (e.g., traffic signs). Ghost returns may be a "bloom" of lidar points around retro-reflectors, resulting in false lidar returns around real objects in open space. The returns are difficult to label because they appear sporadically around real objects. The returns may also be higher in altitude than the true location of the reflector, thus generating false positive general objects from it. It is difficult to discover this obstruction failure mode without being warned of the false positive first. Even if people know about this failure mode, it is difficult to explicitly label them.

[0020] Second, providing ground elevation using an atlas or map to remove ground points can be very challenging because the perception system tries or attempts to detect smaller things at a larger scale. When operating onboard, ground elevation queried from an atlas can have accuracy issues for a variety of reasons (e.g., ground surface resolution, localization errors), which can be exaggerated at larger scales due to the lever arm effect. The lever arm effect occurs when small angular errors in attitude (especially pitch) lead to large spatial errors at larger distances (e.g., distance to a point of interest). This effect is similar to looking from the vertex of a triangle toward the other side - the farther away the side is, the larger it appears, even without changing any angles.

[0021] Third, sometimes it is difficult to distinguish the ground from real objects. In this case, it is difficult to mark the ground as an occluder.

[0022] To address these issues, embodiments of the present disclosure relate to techniques for simplifying the process of classifying sensor returns as objects or non-objects. In some embodiments, the perception system of an autonomous vehicle may include a residual interpreter system that uses a model (e.g., an ML model) that divides sensor data into "objects" (e.g., vehicles, pedestrians, cyclists, and other objects that the vehicle's motion planning system should care about) and "occluders" (e.g., rain, snow, fog, smoke, exhaust, erroneous sensor returns, and other things that the motion planning system should ignore). The residual interpreter system can explain sensor measurements that have not yet been explained by other components of the perception system (e.g., detectors or trackers). In some embodiments, instead of predicting occluders with labels and using the predicted results to interpret sensor data, the residual interpreter system can predict the opposite, that is, objects in the world that the vehicle should pay attention to. For example, the residual interpreter system can predict all tangible objects in the scene and the properties of the objects. Then, by default, the remaining returns can be treated as occluders that the motion planning system can ignore. In this way, since the general object is labeled (e.g., as a known or predetermined object classification), the remainder interpreter system can supplement the general object label as the occluder label (and, subtly, the ground). For example, in a situation where it is difficult to distinguish an occluder (e.g., the ground) from a real object, the remainder interpreter system can handle this situation without having to interpret or explicitly label the ground.

[0023] In some embodiments, the remainder interpreter system can perform (1) classification of sensor data and (2) generation of generic trajectories (e.g., generating and publishing (or reporting) generic trajectories from uninterpreted sensor returns). In some embodiments, the classification of sensor data can be triggered by the arrival of new measurement data, and the generation of generic trajectories can be triggered by the arrival of new trajectory bundles.

[0024] In some embodiments, the residual object interpreter system does not track scenes (or objects) over time, but can track "general objects" as general trajectories. Here, "general objects" refer to sensor returns that have been classified as objects and should not be ignored by motion planning but have not yet been interpreted as objects of a specific type or classification. General objects can be implemented as trajectory types that are neither tracked over time (so the consistency of general trajectories over time cannot be guaranteed) nor describe object instance boundaries. General objects can include information about the predicted position of objects and their instantaneous velocity. In some embodiments, general objects can be published or reported at the same frequency as object trajectories (e.g., less than or equal to 20Hz). In some embodiments, the residual object interpreter system can predict information such as whether it is an obstruction, speed, and properties (general trajectory) for each unit in the voxelized view to publish participant information required by the motion planning system.

[0025] In some embodiments, the remainder interpreter system can implement a "no measurement missed" strategy by: (1) handling classifications not handled by the primary tracker, including animals, debris, balls, obstacles, etc.; (2) providing graceful degradation for the perception system by covering false negatives; and / or (3) identifying static parts of the world (e.g., obstacles). In some embodiments, the remainder interpreter system can interpret or predict foreign objects and debris (FOD) - e.g., objects found in inappropriate locations. Based on experiments, most FOD are characterized as (1) static, (2) "straddleable" (e.g., objects small enough that a vehicle can straddle them), and / or (3) likely to require a camera to detect or predict.

[0026] In some embodiments, the remainder interpreter system can cover false negatives by interpreting or predicting objects and performing graceful degradation (e.g., taking backup safety precautions - issuing warnings such as "we don't know what it is, but don't hit it") even if other components of the perception system (e.g., detectors or trackers) do not interpret or predict objects (false negatives). The remainder interpreter system can provide a low-fidelity velocity profile of the object (e.g., as a general trajectory) even if the object is not clearly detected or identified, thereby implementing the perception system's "no measurement missed" strategy. In some embodiments, the remainder interpreter system can process the data for each voxel so that it does not identify the shape or detailed appearance of the object.

[0027] In some embodiments, the residue interpreter system can perform low-fidelity classification into non-negligible objects and negligible non-objects ("obscurations"; e.g., rain, snow, fog, smoke, exhaust, etc.). In some embodiments, the residue interpreter system can perform classification into different classifications: e.g., vulnerable or non-vulnerable, harmful or non-harmful, etc.

[0028] In some embodiments, the residue interpreter system may output a range view as an output of the classification process, while other components of the perception system (e.g., a detector or tracker) may output a bird's-eye view. The residue interpreter system may output a rasterized view and / or a grid view. In some embodiments, after generating an output view (e.g., a rasterized view or a 3-D voxel map), the residue interpreter system may obtain or input a location (e.g., (x, y, z) coordinates) and check the results of the classification process at that location. If the location has been interpreted by the classification process (e.g., an object exists), the residue interpreter system may discard (discard or remove or delete or ignore) the output view for that location.

[0029] In some embodiments, the residue interpreter system can handle off-road general or static objects as well as on-road general or static objects. In some embodiments, the residue interpreter system can use a labeling scheme that only considers or detects general objects on the road, for example, a scheme that only labels general objects within a certain distance from the road area boundary, and the residue interpreter system can also perform reasoning only on such areas. Using this labeling scheme, for static objects, the residue interpreter system may only consider or detect static objects on the road, because the planning system can treat the road area boundary as something that the autonomous vehicle (AV) should never exceed, so reporting static objects outside the road area boundary will not change the AV behavior. Limiting labeling and reasoning to only on the road can improve data efficiency and reduce unnecessary labeling costs. For off-road objects, the residue interpreter system may still need to predict their speed in case something may move towards the AV's path.

[0030] In some embodiments, the residual interpreter system (e.g., the classification process) may report and / or visualize off-road static objects upon request using a labeling scheme similar to general objects on-road. However, the residual interpreter system may provide information about off-road static objects with less precision than on-road objects (e.g., it may report false positive static objects) because the AV behavior is not affected. In summary, off-road predictions may follow a similar labeling scheme as on-road predictions, or use a simplified scheme to predict speed, and the residual interpreter system may choose a high-recall / low-precision operating point (see Figure 7B ) for off-road use.

[0031] In some embodiments, the remainder interpreter system can perform a process of general trajectory generation (or "general generator") by predicting velocity vectors for general objects. The prediction can be performed by (1) obtaining or inputting feature vectors for the current time and previous time, (2) applying vision or machine learning (e.g., neural network) techniques, and (3) outputting prediction results. In some embodiments, the remainder interpreter system can output a velocity vector for each sensor return (e.g., LIDAR point). In some embodiments, the remainder interpreter system can further obtain inputs of the classification and velocity vectors and generate a 3-D voxel map (e.g., 25 cm voxels).

[0032] In some embodiments, the general generation process may output a general trajectory in the form of a box, while in other embodiments, the general trajectory does not describe the boundaries of the object instance. Both approaches have advantages. For example, when the boundaries are not described in the general trajectory, lower latency can be achieved. On the other hand, when the boundaries are available in the general trajectory, the boundaries can be compared with the boundaries of the objects detected by the main detector, and duplicates can be removed from the objects detected by the main detector, thereby more accurately detecting the general object. The general trajectory may have instantaneous dynamics, but no periodic consistency. In some embodiments, the residual interpreter system can only report objects that may cause or receive "damage", where "damage" is independent of the platform. The perception system can use the reported object size to determine the "straddling ability". In some embodiments, the perception system or the residual interpreter system can output information in the form of a static grid. In some embodiments, the residual interpreter system can generate output with high accuracy to meet the requirements of the perception system with low false positive tolerance at highway speeds. For example, the residual interpreter system can not only filter out "occluders", but also provide reasonable dynamic estimates. The residual interpreter system can also generate output with high recall. For example, the remainder interpreter system may report all objects that could cause "harm", and the perception system may use the reported object sizes to determine "straddle capability". The remainder interpreter system may also generate long-range outputs. For example, the range of the output may be 160m behind for a merge, 200m ahead for a stop for any obstacles, or 300m ahead to allow enough time for a lane change.

[0033] In some embodiments, the residual interpreter system can classify points in the range view as objects or obstructions. After performing clustering (e.g., some simple clustering), the classification process can send (or ship) the clustering or classification output to the general generation process. The general generation process can receive the output from the classification process and fuse or combine the received output with the latest trajectory output to create a "general" trajectory based on the interpretation that the trajectory does not explain. The general generation process can apply a range view-based model to lidar points to perform or optimize some tasks, such as multi-class detection, multi-class trajectory prediction, lidar point segmentation, and class-independent detection. In some embodiments, the general generation process can (1) take sensor full data (e.g., camera / radar / lidar) and a map as input, (2) output a general trajectory with instantaneous velocity, and (3) handle multiple objects in the z direction (e.g., objects under an overpass).

[0034] In some embodiments, the residue interpreter system can use a large number of general object labels and / or occluder labels. In some embodiments, the residue interpreter system can classify whether the return is an occluder, rather than classifying whether the return belongs to a real object. In some embodiments, the residue interpreter system can use the ground surface from an atlas or map to perform ground removal via post-processing. In some embodiments, the residue interpreter system can perform filtering based on point counts and ray tracing statistics as a post-processing step, rather than treating point counts and ray tracing statistics as features. One advantage of this post-processing step is to reduce false positives. Post-processing can be outside the ML model (e.g., a pytorch model), or inside the ML model as an unweighted component (e.g., an unlearned component).

[0035] In some embodiments, the residue interpreter system may directly predict real objects or general trajectories as part of the learning / inference process by performing ground subtraction, which uses local sensor measurements and optional map information as a prior. In some embodiments, post-processing may be performed to improve general quality, for example, using ray tracing statistics to reject false positives of reflections. In some embodiments, the residue interpreter system may perform such post-processing as part of the learning / inference process. By providing similar features, post-processing can be incorporated or integrated into the learning / inference process. In some embodiments, the residue interpreter system can directly convert and package the output of the model into a general trajectory, thereby easily adjusting the learning objectives to reflect what the planning system needs. In some embodiments, assuming that the output view includes multiple monomers, the residue interpreter system can output a general trajectory by performing predictions based on the model of each monomer, post-processing of grouping, clustering and / or classifying the predicted monomers, and generating a general trajectory by providing an explanation. While explanations via trajectories, roads, and other external sources can be part of post-processing after the model learning / inference process, in some embodiments, a residual interpreter system can potentially provide such explanations as model inputs so that predictions can be closer to what the system publishes or reports (e.g., to a planning system). In some embodiments, a residual interpreter system can generate or produce a set of general trajectories that explain uninterpreted sensor data believed to be from real objects. In some embodiments, the output of a model (e.g., a neural network) can be a point-by-point prediction of objectness (e.g., object or non-object), velocity, and / or properties.

[0036] The embodiments of the present disclosure have at least the following advantages and benefits.

[0037] First, embodiments of the present disclosure may provide useful techniques for efficiently interpreting ground / obstructions without predicting obstructions and using the predicted results to interpret sensor data. The Residual Interpreter system instead tags and predicts all tangible objects in the scene, so that the remaining returns may fall into a "don't care" classification by default, in which the system may optionally explicitly tag ground / obstructions as needed. This approach has the following advantages. In this way, the tagging process can be simplified, as identifying all obstructions may be very difficult, while identifying all real objects is relatively easy and more intuitive.

[0038] Second, embodiments of the present disclosure may provide useful techniques for using learned models to process ground in a more robust manner than using an atlas or map. Ground elevation from an atlas or map may be inaccurate due to various sources of error, such as map resolution limitations, map inaccuracies, positioning inaccuracies (just to name a few). Therefore, relying on ground elevation to exclude false positives from ground points is not a robust solution, and the inaccuracy may become worse as range increases. To address this issue, in accordance with some embodiments, the residue interpreter system may perform ground subtraction as part of the learning / inference process, which utilizes local sensor measurements and, optionally, map information as a prior.

[0039] Third, post-processing can be simplified by performing it as part of the learning / inference process. Post-processing can be performed to, for example, use ray tracing statistics to reject false positives of reflections to improve general quality. By providing similar features, post-processing can be incorporated or integrated into the learning / inference process.

[0040] Fourth, the output of a learning model (e.g., an ML model) can be easily predicted and adjusted for use by a planning system. For example, the output of a model can be almost directly converted and packaged into a general trajectory, making it easy to adjust the learning objective to reflect what the planning system needs.

[0041] I. Autonomous Vehicle System Environment

[0042] Figure 1 is a block diagram illustrating an example of a system environment for an autonomous vehicle according to some implementations.

[0043] See also Figure 1, an example autonomous vehicle 100 in which various techniques disclosed herein may be implemented. For example, vehicle 100 may include a powertrain 102 including a prime mover 104 powered by an energy source 106 and capable of providing power to a drivetrain 108; and a control system 110 including a directional control 112, a powertrain control 114, and a brake control 116. Vehicle 100 may be implemented as any number of different types of vehicles, including vehicles capable of transporting people and / or cargo and capable of traveling in a variety of environments, and it should be understood that the above-described components 102-116 may vary widely based on the type of vehicle in which they are used.

[0044] For simplicity, the embodiments discussed below will focus on wheeled land vehicles, such as cars, vans, trucks, buses, etc. In such embodiments, the prime mover 104 may include one or more electric motors and / or internal combustion engines (etc.). The energy source may include, for example, a fuel system (e.g., providing gasoline, diesel, hydrogen, etc.), a battery system, solar panels or other renewable energy sources, and / or a fuel cell system. The transmission system 108 includes: wheels and / or tires, as well as a transmission and / or any other mechanical drive assembly suitable for converting the output of the prime mover 104 into vehicle motion; and one or more brakes configured to controllably stop or slow the speed of the vehicle 100, and a direction or steering assembly suitable for controlling the trajectory of the vehicle 100 (e.g., a rack and pinion steering linkage that enables one or more wheels of the vehicle 100 to pivot about a substantially vertical axis to change the angle of the wheel rotation plane relative to the longitudinal axis of the vehicle). In some embodiments, a combination of power systems and energy sources may be used (e.g., in the case of an electric / gas hybrid vehicle), and in some cases, multiple electric motors (e.g., dedicated to a single wheel or axle) may be used as prime movers.

[0045] The directional control 112 may include one or more actuators and / or sensors for controlling and receiving feedback from a directional or steering assembly to enable the vehicle 100 to follow a desired trajectory. The powertrain control 114 may be configured to control the output of the powertrain 102, such as to control the output power of the prime mover 104, control the gear of a transmission in the transmission 108, etc., thereby controlling the speed and / or direction of the vehicle 100. The brake control 116 may be configured to control one or more brakes for slowing or stopping the vehicle 100, such as disc brakes or drum brakes coupled to the wheels of the vehicle.

[0046] Other vehicle types (including but not limited to off-road vehicles, all-terrain or tracked vehicles, ships, submarines, construction equipment, etc.) will necessarily use different powertrains, transmissions, energy sources, directional controls, powertrain controls, and brake controls. In addition, in some embodiments, some components may be combined, for example, where directional control of the vehicle is handled primarily by changing the output of one or more prime movers. Therefore, the embodiments disclosed herein are not limited to the specific application of the technology described herein to autonomous wheeled land vehicles.

[0047] In the illustrated embodiment, fully or semi-autonomous driving control of the vehicle 100 is implemented in a vehicle control system 120, which may include one or more processors 122 and one or more memories 124, wherein each processor 122 is configured to execute program code instructions 126 stored in the memory 124. The processor may include, for example, a graphics processing unit (“GPU”) and / or a central processing unit (“CPU”).

[0048] The sensor 130 may include various sensors suitable for collecting information from the surrounding environment of the vehicle for controlling the operation of the vehicle. For example, the sensor 130 may include a radar sensor 134, a lidar (light detection and ranging) sensor 136, a 3D positioning sensor 138 (e.g., a satellite navigation system such as GPS (Global Positioning System), GLONASS (Globalnaya Navigazionnaya Sputnikovaya Sistema, or Global Navigation Satellite System), Beidou Navigation Satellite System (BDS), Galileo, compass, etc.). The 3D positioning sensor 138 can be used to determine the position of the vehicle on the earth using satellite signals. The sensor 130 may optionally include a camera 140 and / or an IMU (Inertial Measurement Unit) 142. The camera 140 may be a single or stereo camera and may record static and / or video images. The IMU 142 may include multiple gyroscopes and accelerometers that are capable of detecting linear and rotational motion of the vehicle in three directions. One or more encoders (not shown), such as wheel encoders, may be used to monitor the rotation of one or more wheels of the vehicle 100.

[0049] The output of the sensor 130 may be provided to a set of control subsystems 150, including a positioning subsystem 152, a planning subsystem 156, a perception subsystem 154, and a control subsystem 158. The positioning subsystem 152 is primarily responsible for accurately determining the position and orientation (sometimes also referred to as "pose") of the vehicle 100 within its surroundings and typically within a reference frame. As part of generating labeled autonomous vehicle data, the position of the autonomous vehicle may be compared to the position of another vehicle in the same environment. The perception subsystem 154 is primarily responsible for detecting, tracking, and / or identifying objects in the environment surrounding the vehicle 100. According to some embodiments, a machine learning model may be used to track objects. The planning subsystem 156 is primarily responsible for planning a trajectory for the vehicle 100 within a certain time frame given a desired destination and static and moving objects within the environment. According to some embodiments, a machine learning model may be used to plan a vehicle trajectory. The control subsystem 158 is primarily responsible for generating appropriate control signals to control various controls in the vehicle control system 120 to achieve the planned trajectory of the vehicle 100. Similarly, a machine learning model may be used to generate one or more signals to control the autonomous vehicle to achieve the planned trajectory.

[0050] It should be understood that Figure 1 The assembly of components of the vehicle control system 120 shown in FIG. 1 is merely an example. In some embodiments, individual sensors may be omitted. Additionally or alternatively, in some embodiments, a Figure 1 Multiple sensors of the type shown in can be used to achieve redundancy and / or cover different areas around the vehicle, and other types of sensors can be used. Similarly, different types and / or combinations of control subsystems can be used in other embodiments. In addition, although subsystems 152-158 are shown as being separated from processor 122 and memory 124, it is understood that in some embodiments, some or all of the functions of subsystems 152-158 can be implemented by program code instructions 126 residing in one or more memories 124 and executed by one or more processors 122, and these subsystems 152-158 can be implemented using the same processor and / or memory in some cases. The subsystem can be implemented at least in part using various dedicated circuit logic, various processors, various field programmable gate arrays ("FPGA"), various application-specific integrated circuits ("ASIC"), various real-time controllers, etc. As described above, multiple subsystems can utilize circuits, processors, sensors and / or other components. In addition, the various components in the vehicle control system 120 can be networked in various ways.

[0051] In some embodiments, the vehicle 100 may also include an auxiliary vehicle control system (not shown) that may serve as a redundant or backup control system for the vehicle 100. In some embodiments, the auxiliary vehicle control system is capable of fully operating the autonomous vehicle 100 in the event of an adverse event in the vehicle control system 120, while in other embodiments, the auxiliary vehicle control system may have only limited functionality, such as to perform a controlled stop of the vehicle 100 in response to an adverse event detected in the primary vehicle control system 120. In other embodiments, the auxiliary vehicle control system may be omitted.

[0052] In general, a myriad of different architectures (including various combinations of software, hardware, circuit logic, sensors, networks, etc.) can be used to implement Figure 1 . Each processor may be implemented, for example, as a microprocessor, and each memory may represent: a random access memory ("RAM") device, which includes main memory; and any supplementary levels of memory, such as cache memory, non-volatile or backup memory (such as programmable memory or flash memory), read-only memory, etc. In addition, each memory may be considered to include: memory physically located elsewhere in the vehicle 100, such as any cache memory in a processor; and any storage capacity used as virtual memory, such as stored on a mass storage device or another computer controller. Figure 1 One or more of the processors shown in or a completely separate processor may be used to implement additional functionality in the vehicle 100 beyond autonomous driving control purposes, such as controlling entertainment systems, operating doors, lights, convenience features, etc.

[0053] In addition, to enable additional storage, the vehicle 100 may include one or more mass storage devices, such as a removable disk drive, a hard disk drive, a direct access storage device (“DASD”), an optical disk drive (e.g., a CD drive, a DVD drive, etc.), a solid-state storage drive (“SSD”), a network attached storage, a storage area network, and / or a tape drive, etc.

[0054] In addition, the vehicle 100 may include a user interface 164 to enable the vehicle 100 to receive a plurality of inputs from and generate outputs for a passenger or operator (e.g., one or more displays, touch screens, voice and / or gesture interfaces, buttons and other tactile controls, etc.). Otherwise, inputs from a passenger or operator may be received via another computer or electronic device, such as via an app on a mobile device or via a web interface.

[0055] In addition, the vehicle 100 may include one or more network interfaces, such as a network interface 162, which is suitable for communicating with one or more networks 170 (e.g., a local area network ("LAN"), a wide area network ("WAN"), a wireless network, and / or the Internet, etc.) to allow communication of information with other computers and electronic devices (including, for example, central services such as cloud services), from which the vehicle 100 receives environmental and other data for its autonomous driving control. In many embodiments, data collected by one or more sensors 130 can be uploaded to a computing system 172 via the network 170 for additional processing. In some embodiments, a timestamp can be added to each instance of vehicle data prior to upload. According to many embodiments, reference is made to Figure 2 , Figure 3 and Figure 4 Additional processing of autonomous vehicle data by computing system 172 is described.

[0056] Figure 1 Each of the processors shown, as well as the various other controllers and subsystems disclosed herein, typically operate under the control of an operating system and execute or otherwise rely on various computer software applications, components, programs, objects, modules, data structures, etc., as described in more detail below. In addition, the various applications, components, programs, objects, modules, etc. may also be executed on one or more processors in another computer coupled to the vehicle 100 via the network 170, such as in a distributed, cloud-based, or client-server computing environment, so that the processing required to implement the computer program functions can be distributed to multiple computers and / or services on the network.

[0057] In general, routines executed to implement the various embodiments described herein, whether implemented as part of an operating system or a specific application, component, program, object, module or sequence of instructions, or even a subset thereof, are referred to herein as "program code". Program code typically includes one or more instructions that reside at various times in various memories and storage devices, and when read and executed by one or more processors, perform the steps necessary to implement the steps or elements embodying various aspects of the present disclosure. In addition, while the embodiments have been and will be described below in the context of fully functional computers and systems, it will be appreciated that the various embodiments described herein can be distributed as program products in various forms, and that the embodiments can be implemented regardless of the specific type of computer-readable medium used to actually perform the distribution.

[0058] Examples of computer-readable media include tangible, non-transitory media such as volatile and nonvolatile memory devices, floppy disks and other removable disks, solid-state drives, hard drives, magnetic tapes, and optical disks (e.g., CD-ROMs, DVDs, etc.), and the like.

[0059] In addition, the various program codes described below can be identified based on the application in which the program code is implemented in a specific embodiment. However, it should be understood that any subsequent specific program nomenclature is used for convenience only, and therefore the present disclosure should not be limited to use in any specific application identified and / or implied only by such nomenclature. In addition, in view of the generally endless ways in which computer programs can be organized into routines, processes, methods, modules, objects, etc., and the various ways in which program functions can be distributed between various software layers (e.g., operating systems, libraries, APIs, applications, applets, etc.) residing in a typical computer, it should be understood that the present disclosure is not limited to the specific organization and distribution of program functions described herein.

[0060] Figure 1 The environment shown in is not intended to limit the embodiments disclosed herein. In fact, other alternative hardware and / or software environments can be used without departing from the scope of the embodiments disclosed herein.

[0061] II. Computing Systems for Processing Autonomous Vehicle Data

[0062] Figure 2 is a block diagram illustrating an example of a computing system for training a machine learning model, such as a neural network model, in accordance with some implementations.

[0063] The computing system 172 can communicate with the network 170 (see Figure 1 ) receives time-stamped vehicle observation data (i.e., a collection of vehicle data and / or environmental data collected by one or more autonomous vehicles and one or more non-autonomous vehicles). In some implementations, computing system 172 may include temporal correlation engine 202, location engine 204, tagging engine 206, neural network engine 210, training engine 214, and training instance engine 218. Temporal correlation engine 202, location engine 204, tagging engine 206, neural network engine 210, training engine 214, and training instance engine 218 are example components in which the techniques described herein may be implemented and / or with which the systems, components, and techniques described herein may interface. Figure 2The operations performed by one or more engines 202, 204, 206, 210, 214, 218 in the embodiment can be distributed across multiple computing systems. In some embodiments, one or more aspects of the engines 202, 204, 206, 210, 214, 218 can be combined into a single system and / or one or more aspects can be implemented by the computing system 172. For example, in some of these embodiments, aspects of the time correlation engine 202 can be combined with aspects of the tagging engine 206. The engines according to many embodiments can each be implemented in one or more computing devices, which communicate, for example, through a communication network. The communication network can include a wide area network (such as the Internet), one or more local area networks ("LANs") (such as Wi-Fi LANs, mesh networks, etc.), and one or more bus subsystems. The communication network can optionally utilize one or more standard communication technologies, protocols, and / or inter-process communication technologies.

[0064] The computing system 172 may perform various processing on the vehicle data 208. In some embodiments, the vehicle data 208 includes time-stamped autonomous vehicle data (such as described herein with respect to Figure 1 The time correlation engine 202 may (if necessary) synchronize timestamps between data sets collected by separate vehicles collecting data in the same environment. For example, when two vehicles are simultaneously collecting data in an environment, the timestamps attached to the data from one vehicle may not correspond to the timestamps on the data collected from the other vehicle. In some embodiments, the timestamps in the data collected by one vehicle may be shifted to correspond to the timestamps in the data collected by the other vehicle.

[0065] The location engine 204 can determine the proximity of vehicles within the environment (typically at each timestamp) based on the vehicle data 208. In some embodiments, one or more proximity sensors within the vehicle can be used to determine the co-presence of vehicles. In some embodiments, the signal from the proximity sensor can indicate a variety of ranges, including: not within range, within one meter, within five meters, within ten meters, within fifty meters, within one hundred meters, within two hundred meters, etc. In some embodiments, only vehicle data for vehicles within a proximity threshold level can be further processed (e.g., only data from vehicles within a 250-meter range can be additionally processed).

[0066] Additionally or alternatively, a vehicle may enter and exit a proximity threshold range while maneuvering in an environment. For example, only data at timestamps where the vehicle is within the proximity range may be additionally processed. In some embodiments, portions of vehicle data where the vehicle is not within the proximity range may be discarded.

[0067] For example, the location engine 204 may additionally or alternatively use the vehicle data 208 and a high-definition map constructed with sensor data to determine the vehicle location. In some embodiments, 3D positioning sensor data (such as a location provided by a GPS system) can locate the vehicle within the environment. In other embodiments, common landmarks can be used to locate the location of the vehicle in the environment. Common landmarks can include various objects, including fixed objects such as buildings, street signs, stop signs, traffic lights, mailboxes, trees, bushes, fence sections, etc. The distance of the autonomous vehicle from the common landmarks can be determined based on the autonomous vehicle data (e.g., using lidar data). Similarly, the distance of the additional vehicle from the common landmarks can be determined based on the additional vehicle. The distance between the autonomous vehicle and the additional vehicle can be calculated at a specific timestamp using the distance of each vehicle from the common landmark. For example, a common landmark such as a stop sign can be captured in the autonomous vehicle data as well as the non-autonomous vehicle data (e.g., which can be collected using autonomous vehicle sensors installed on the non-autonomous vehicle). The data collected by the corresponding vehicle lidar unit can determine the distance of each vehicle to the stop sign at the same timestamp. The distance between the autonomous vehicle and the non-autonomous vehicle can be calculated using the distance of each vehicle to the stop sign. Additionally or alternatively, the additional vehicle may determine its position in the map using a 3D reference frame (e.g., an earth-fixed reference frame centered on the earth). In some embodiments, the autonomous vehicle may determine its position on the same map relative to the same reference frame, and / or one or more additional methods of determining its position relative to the same map as the additional vehicle.

[0068] The tagging engine 206 may use the vehicle data collected from one or more additional vehicles to generate tags for autonomous vehicle data (in some embodiments, tags are automatically generated). In some embodiments, the computing system 172 may use the location engine 204 to determine whether two vehicles coexist in the environment. In some embodiments, the tagging engine 206 may determine an autonomous vehicle data instance that captures only a single additional vehicle that coexists in the environment (i.e., when the autonomous vehicle is known to be within the proximity of the additional vehicle and only one vehicle is captured in the autonomous vehicle data, the additional vehicle will usually be the vehicle captured in the autonomous vehicle data). In some embodiments, the tagging engine 206 may determine an autonomous vehicle data instance that captures additional non-vehicle objects coexisting with the current vehicle in the environment. The data collected from the additional vehicles may be mapped to the location of the additional vehicle in the autonomous vehicle data instance at a common timestamp. For example, the brake light signal of a non-autonomous vehicle (equipped with autonomous vehicle sensors) may be collected via a controller area network (CAN) bus and time-stamped by a computing device of the non-autonomous vehicle. A tag indicating the brake light state of the non-autonomous vehicle may be mapped to the location of the non-autonomous vehicle captured in the autonomous vehicle data to automatically generate a brake light tag for the non-autonomous vehicle at the corresponding timestamp. Additionally or alternatively, additional vehicle data identifying the non-autonomous vehicle (such as vehicle dimensions) may be used to determine an accurate bounding box around the non-autonomous vehicle in the autonomous vehicle's observation. In other embodiments, the labeling engine 206 may utilize the locations of the two vehicles determined by the location engine 204 (e.g., locations determined using GPS data collected from each vehicle and / or locations determined by locating each vehicle using common landmarks in the environment).

[0069] The neural network engine 210 can train the neural network model 212. According to some embodiments, the neural network model 212 may include one and / or more layers of storage units, wherein each storage unit has a corresponding weight. Various neural network models may be utilized, including feedforward neural networks, convolutional neural networks, recursive neural networks, radial basis functions, other neural network models, and combinations of several neural networks. In addition or alternatively, the neural network model 212 may represent various machine learning techniques other than neural networks, such as support vector machines, decision trees, Bayesian networks, other machine learning techniques, and / or combinations of machine learning techniques. The training neural network model 212 according to some embodiments described herein may utilize the neural network engine 210, the training engine 214, and the training instance engine 218. The neural network model may be trained for various autonomous vehicle tasks, including determining the position of a target autonomous vehicle, generating one or more signals to control an autonomous vehicle, tracking or identifying objects in an autonomous vehicle environment, and the like. For example, the neural network model may be trained to identify traffic lights in an autonomous vehicle environment. As a further example, the neural network model may be trained to predict the brand and model of other vehicles in an autonomous vehicle environment. In many embodiments, the neural network model may be trained to perform a single task. In other implementations, a neural network model can be trained to perform multiple tasks.

[0070] The training instance engine 218 may generate training instances to train the neural network model. The training instances may include, for example, instances of autonomous vehicle data, where the autonomous vehicle may detect additional vehicles using one or more sensors and labels corresponding to data collected from the additional vehicles. The training engine 214 may apply the training instances as inputs to the neural network model 212. In some embodiments, the neural network model 212 may be trained using at least one of supervised learning, unsupervised learning, or semi-supervised learning. In addition or alternatively, according to some embodiments, the neural network model may be a deep learning network including a recurrent neural network, a convolutional neural network (CNN), a network that is a combination of multiple networks, and the like. For example, the training engine 214 may generate a predicted neural network model output by applying the training input to the neural network model 212. In addition or alternatively, the training engine 214 may compare the predicted neural network model output with the known output of the neural network model from the training instance, and use the comparison to update one or more weights in the neural network model 212. In some embodiments, one or more weights may be updated by back-propagating the difference across the neural network model 212.

[0071] Figure 3 is a block diagram illustrating an example of a computing system for testing a trained neural network model according to some embodiments. Figure 2 The computing system 172, the time correlation engine 202, the location engine 204, the labeling engine 206, and the vehicle data 208 are described in section 1. According to some embodiments, the neural network engine 212, the test engine 314, and the test instance engine 318 can be used to generate test instances of autonomous vehicle data (including labels corresponding to additional vehicles or non-vehicle objects present in the autonomous vehicle data) and test the trained (or learned) neural network model 312. In some embodiments, the trained neural network model 312 can generate predicted outputs for a single autonomous vehicle task. In other embodiments, the trained neural network model 312 can generate predicted outputs for multiple autonomous vehicle tasks. The test instance engine 314 can generate test instances 316 using labeled autonomous vehicle data collected from the autonomous vehicle and additional vehicles (or non-vehicle objects), performing the specific autonomous vehicle task for which the neural network model 312 was trained.

[0072] The test instance may include, for example, an instance of autonomous vehicle data in which an additional vehicle (or non-vehicle object) is detected by one or more sensors of the autonomous vehicle and a label corresponding to the data collected by the additional vehicle. The test engine 314 may apply the test instance as an input to the neural network model 312. The predicted output generated by applying the test instance to the neural network model 312 may be compared to the known output of the test instance (i.e., the label generated by the labeling engine 206) to update the accuracy value (e.g., accuracy percentage) of the neural network model.

[0073] Figure 4 is a block diagram illustrating an example of a computing system according to some implementations.

[0074] refer to Figure 4 , the illustrated example computing system 172 includes: one or more processors 410, which communicate with memory 460, at least one network interface controller 430 having a network interface port for connecting to a network (not shown) and other components (e.g., input / output ("I / O") component interface 450 connected to a display (not shown) and input devices (not shown) via a communication system 440 (e.g., a bus). Typically, the processor 410 will execute instructions (or computer programs) received from the memory. The illustrated processor 410 contains or is directly connected to a cache 420. In some cases, instructions are read from the memory 460 into the cache 420 and executed by the processor 410 from the cache 420.

[0075] In more detail, processor 410 can be any logic circuit that processes instructions (e.g., instructions obtained from memory 460 or cache 420). In some embodiments, processor 410 is a microprocessor unit or a dedicated processor. Computing device 400 can be based on any processor or processor group capable of operating as described herein. Processor 410 can be a single-core or multi-core processor. Processor 410 can be a plurality of different processors.

[0076] Memory 460 may be any device suitable for storing computer-readable data. Memory 460 may be a device with fixed storage or a device for reading removable storage media. Examples include all forms of non-volatile memory, media, and memory devices, semiconductor memory devices (e.g., EPROM, EEPROM, SDRAM, and flash memory devices), magnetic disks, magneto-optical disks, and optical disks (e.g., CDROM, DVD-ROM, or Blu-Ray® optical disks). Computing system 172 may have any number of memory devices as memory 460.

[0077] Cache memory 420 is a form of computer memory that is typically placed close to processor 410 to achieve fast read times. In some implementations, cache memory 420 is part of processor 410 or is located on the same chip as processor 410. In some implementations, cache memory 420 has multiple levels, such as L2 and L3 cache layers.

[0078] The network interface controller 430 manages data exchange via a network interface (sometimes referred to as a network interface port). The network interface controller 430 handles the physical layer and data link layer of the OSI model for network communication. In some embodiments, some of the network interface controller tasks are handled by one or more processors 410. In some embodiments, the network interface controller 430 is part of the processor 410. In some embodiments, the computing system 172 has multiple network interfaces controlled by a single controller 430. In some embodiments, the computing system 172 has multiple network interface controllers 430. In some embodiments, each network interface is a connection point for a physical network link (e.g., a cat-5 Ethernet link). In some embodiments, the network interface controller 430 supports wireless network connections, and the interface port is a wireless (e.g., radio) receiver / transmitter (e.g., for any of the IEEE 802.11 protocol, near field communication "NFC", Bluetooth, ANT, or any other wireless protocol). In some embodiments, the network interface controller 430 implements one or more network protocols, such as Ethernet. Typically, the computing device 172 exchanges data with other computing devices via a network interface via a physical or wireless link. The network interface may be linked to another device directly or via an intermediate device (eg, a network device such as a hub, bridge, switch, or router) that connects the computing device 172 to a data network such as the Internet.

[0079] Computing system 172 may include, or provide interfaces for, one or more input or output ("I / O") devices. Input devices include, but are not limited to, keyboards, microphones, touch screens, foot pedals, sensors, MIDI devices, and pointing devices such as mice or trackballs. Output devices include, but are not limited to, video displays, speakers, refreshable Braille terminals, lights, MIDI devices, and 2-D or 3-D printers.

[0080] Other components may include I / O interfaces, external serial device ports, and any additional coprocessors. For example, computing system 172 may include interfaces (e.g., Universal Serial Bus (USB) interfaces) for connecting input devices, output devices, or additional storage devices (e.g., portable flash drives or external media drives). In some embodiments, computing device 172 includes additional devices such as coprocessors, such as math coprocessors that can assist processor 410 in performing high-precision or complex calculations.

[0081] III. Systems and methods for generating information about measurement remainder using sensor data

[0082] Figure 5 is a block diagram illustrating an example of a perception subsystem according to an embodiment of the present disclosure.

[0083] The perception subsystem 154 is responsible for detecting, classifying, and tracking participants and objects in the environment surrounding the vehicle 100 (see Figure 1 ). In some embodiments, the perception subsystem 154 may include at least one of a residue interpreter 519, a detector 550 (or multiple detectors 550), and a tracker 510. In some embodiments, the perception subsystem 154 may include multiple detectors that receive sensor data from corresponding sensors.

[0084] The detector 550 is primarily responsible for determining and / or detecting objects within the vehicle's surroundings based on sensor data received from sensors. In some embodiments, the detector 550 may receive data from sensors (e.g., LIDAR data from a lidar sensor 136 or multiple lidar sensors 136, radar data from a radar sensor 134 or multiple radar sensors 134) and process the received sensor data to determine and / or detect objects. In some embodiments, the detector 550 may be one or more detectors corresponding to corresponding different types of sensors, or a single detector that receives data from different types of sensors, or multiple detectors that each receive data from different types of sensors. In some embodiments, the detector 550 may be configured to classify an object into a specific classification based on sensor data received from one or more sensors (e.g., using one or more machine learning models to classify an object into a specific classification). In some embodiments, the detector 550 may determine features associated with the object based on the sensor data. For example, the detector 550 may determine the size, shape, speed, or direction of movement of the object. In some embodiments, the detector 550 may output the determined classification or determined features of the object to input into the tracker 510.

[0085] In some embodiments, the output of the detector 550 may include detection-related data and / or prediction-related data. The detection data may include a three-dimensional (3D) orientation box for the detected object (e.g., a pedestrian, a cyclist, a vehicle, etc.). In some embodiments, a motorcyclist may be labeled as a vehicle. In some embodiments, the detector 550 may perform detection within a spatial (or grid) grid of a predetermined size, which may contain multiple cells. The prediction data may include a two-dimensional (2D) spatial grid containing information about objects for which the detector does not have a sufficient or complete shape representation or motion model. The grid may include at least one of an obstruction grid, a velocity grid, or a vegetation grid. The obstruction grid may represent the probability that each cell is an obstruction (e.g., rain, snow, fog, smoke, exhaust, etc.). The velocity grid may represent a ballistic two-dimensional velocity vector indicating how the object contained in each cell moves. The vegetation grid may represent the probability that each cell is vegetation. In some embodiments, the velocity grid and the vegetation grid may not have been predicted by the detector. In some embodiments, all grids can contain information related to the sensor data contained in each cell (e.g., min / max z values ​​measured from the lidar sensor, timestamps). In some embodiments, the sensor buffer can use the contents of the grid to remove spurious lidar data (e.g., obstruction grids), mark common objects (e.g., moving objects away from the tracked object) with velocity values ​​(e.g., velocity grids), or inform static grids based on the contents of the grid (e.g., vegetation grids).

[0086] Tracker 510 is primarily responsible for tracking objects (eg, object 530) within the environment surrounding vehicle 100. In some embodiments, tracker 510 may be configured to correlate continuous sensor observations of the object (eg, radar points in a cloud of points measured by a radar sensor) and generate a track based on the sensor observations.

[0087] In some embodiments, the tracker 510 may include a track manager 512 that initiates and / or updates tracks associated with the object based on changes in the target object. In some embodiments, the tracker 510 may include a radar tracker 517, and in some embodiments, the radar tracker 517 may be separate from the tracker 510. In some embodiments, when the radar measurement data is encoded in a certain format, the radar tracker 517 may decode and convert the radar measurement data (e.g., radar points received from the radar sensor 134 or multiple radar sensors 134) into a different format. For example, the radar tracker 517 may add certain channels to the format of the radar measurement data so that additional data may be associated with the additional channels included in the different format. In some embodiments, the radar tracker 517 determines tracks of different objects (e.g., the current position and velocity of different objects) and outputs radar points associated with the tracks of the different objects for input to the track manager 512. In some embodiments, in response to a request for the latest radar measurement data from the track manager 512, the radar tracker 517 may be configured to output track data based on the radar points of the latest radar measurement for input to the track manager 512. In some embodiments, radar tracker 517 may be configured to send updates 517 to track manager 512 to notify the new position and range rate of the track, which may be integrated into track manager 512 or tracker 510 .

[0088] In some embodiments, the tracker 510 may include a detector tracker 516 that receives detection data from the detector 550 and outputs trajectory data (e.g., position and velocity data) of newly detected objects for input to the trajectory manager 512. In some embodiments, the detector tracker 516 may be separate from the tracker 510. In some embodiments, the detector tracker 516 may associate detections with existing trajectories to update, for example, the position, velocity, and range integrated in the trajectory manager 512 or the tracker 510.

[0089] In some embodiments, the tracker 510 may include a lidar tracker 515 that converts lidar measurement data (e.g., reflected laser light measured by the lidar sensor 136 or multiple lidar sensors 136) into points or surfaces in the 3D environment and outputs the points or surfaces to be input to the trajectory manager 512. In some embodiments, the lidar tracker 515 may be separate from the tracker 510. In some embodiments, the lidar tracker 515 may send updates 525 to the trajectory manager 512 to inform visibility information about whether the trajectory actually exists in the world (e.g., via ray tracing). For example, using visibility information (such as ray hits / misses), the tracker 510 can remove trajectories that are not supported by lidar data (e.g., removing trajectories generated by radar returns on an overpass). The tracker 510 may include a camera tracker 518 that converts images captured by a camera (e.g., camera 140 or multiple cameras 140) into points or surfaces in the 3D environment and outputs the points or surfaces to be input to the trajectory manager 512. In some implementations, camera tracker 518 may be separate from tracker 510 .

[0090] In some embodiments, the remainder interpreter system 519 receives (1) trajectory data 529 of an object from the trajectory manager 512, (2) new measurement data from at least one sensor (e.g., sensors 134, 136, 140), and / or (3) map data 135 from a map or atlas. The remainder interpreter system 519 may use a machine learning model (e.g., by Figure 2 The training engine 214 in the training model 212 trains the model 212, which divides the sensor data into "objects" (e.g., vehicles, pedestrians, and cyclists) and "occluders" (things that the motion plan should ignore). The residual interpreter system 519 can explain sensor measurements that have not yet been explained by other components of the perception system (e.g., detectors 550 or trackers 512, 515, 516, 517, 518). In some embodiments, the residual interpreter system 519 can predict the opposite, such as what to pay attention to in the world, instead of predicting occluders with labels and using the predicted results to interpret the sensor data. For example, the residual interpreter system 519 can mark all tangible objects in the scene and label these objects with some properties. Then, the remaining returns may default to the "don't care" classification, where the system can optionally explicitly mark the ground / occluders as needed. In some embodiments, the residual interpreter system 519 can track "general objects" as general trajectories and output the general trajectories 529 to the trajectory manager 512 for controlling the vehicle based on the general trajectories. In some embodiments, general objects may include classes of FOD (foreign objects or debris) that are not handled by the tracker: for example, animals, debris, balls, obstacles, etc.

[0091] Figure 6 is a block diagram illustrating an example of a residue interpreter system according to an embodiment of the present disclosure.

[0092] In some embodiments, the residue interpreter system 600 may include (1) a classifier 650 configured to perform a classification process of the measurement data 620; and (2) a general generator 670 configured to perform a process of generating a general trajectory as a general output 690. The residue interpreter system 600 may receive the measurement data 620, the trajectory data 630, and / or the map data 640 as input. The measurement data 620 may be obtained from a sensor (e.g., Figure 5 In some embodiments, the measurement data 620 may represent data received by the autonomous vehicle (e.g., lidar 136, radar 134, camera 140). Figure 1 The map data may be data from a map or atlas (e.g., Figure 5 135). The general output 690 (and / or potential general object 674) can be (1) an output view including multiple monomers or (2) a 3-D voxel map.

[0093] In some embodiments, the classifier 650 may include: (1) a feature generator 652 configured to generate a set of features, and (2) one or more ML models (e.g., neural networks) 654 configured to classify the measurement data as an “object” or “obstruction” based on the set of features. In some embodiments, the set of features may include range image features (e.g., features associated with a threshold range of proximity), point-by-point features (e.g., features associated with objects such as vehicles, cyclists, pedestrians, etc., or known obstructions such as rain, snow, fog, smoke, exhaust, etc.), and / or filter features (e.g., features associated with ground information, point counts, and ray tracing statistics). In some embodiments, the classifier 650 may output classification or categorized data 656 and a velocity vector 658.

[0094] In some embodiments, instead of predicting obstructions using labels and using the predictions to interpret sensor data, the classifier 650 can label and predict the opposite, e.g., what to watch out for in the world. For example, the classifier 650 can label all tangible objects in the scene (e.g., vehicles, passengers, cyclists, etc.) and label these objects with some properties. The remaining returns may fall into the "don't care" or "unexplained" classification by default, in which the classifier 650 can selectively explicitly label the ground / obstructions as needed. For example, the classifier 650 can also label known obstructions such as rain, snow, fog, smoke, exhaust fumes, so that the remaining returns will fall into the "don't care" or "unexplained" classification as the output of the classifier.

[0095] In some embodiments, the classifier 650 may use a labeling scheme that only considers or detects general objects on the road, for example, a scheme that only labels general objects within a certain distance from the road region boundary, and the classifier 650 may also perform reasoning only on such areas. Using the labeling scheme, for static objects, the classifier 650 may only consider or detect static objects on the road, because the planning system may consider the road region boundary as something that the autonomous vehicle (AV) should never go beyond, so reporting static objects beyond the road region boundary will not change the AV behavior. In some embodiments, the classifier 650 may provide information about off-road static objects with lower accuracy than on-road objects (e.g., it may report false positive static objects) because the AV behavior will not be affected.

[0096] In some embodiments, the classification or classified data 656 output from the classifier 650 may include uninterpreted sensor data (as potential general objects) that is not classified as a known object (e.g., vehicle, cyclist, pedestrian) or a known obstruction (e.g., rain, snow, fog, smoke, exhaust). In some embodiments, the classification or classified data 656 output from the classifier 650 may include sensor data classified as a known object (e.g., vehicle, cyclist, pedestrian) and uninterpreted sensor data.

[0097] In some embodiments, the velocity vector 658 output from the classifier 650 can be a velocity vector of a potential general object. In some embodiments, the classifier 650 can be configured to predict a potential general velocity or position. In some embodiments, the classifier 650 can predict a potential general velocity vector by (1) acquiring or inputting feature vectors at a current time and a previous time (e.g., feature vectors extracted from an "uninterpreted" output of the classifier 650), (2) applying vision or machine learning techniques (e.g., neural networks), and (3) outputting the prediction as a velocity vector.

[0098] The general generator 670 may take inputs of the classification 656 and velocity vector 658 from the classifier 650 and generate (1) an output view including a plurality of monomers or (2) a 3-D voxel map (e.g., 25 cm voxels). In some embodiments, the general generator 670 may output a rasterized view and / or a grid view. In some embodiments, the general generator 670 may (1) generate an output view (e.g., a rasterized view or a 3-D voxel map), (2) take or input a location of the output view, such as (x, y, z) coordinates, and (3) check the results of the classification process at the location. If the location has been explained by the classification process (e.g., there is an object or a known occluder), the general generator 670 may discard (discard or remove or delete or ignore) the output view for the location. If the location is not explained, the general generator 670 may aggregate the classification data 656 (e.g., point-by-point sensor data) and velocity vector 658 at the corresponding location into the output view. In some implementations, the general generator 670 may perform aggregation by averaging subsets of the classification data 656 and velocity vectors 658 (eg, subsets corresponding to particular locations).

[0099] General trajectories may have instantaneous dynamics but no cycle-to-cycle consistency.

[0100] In some embodiments, the classifier 650 can contain or store a range view model that classifies points as objects or obstructions. In some embodiments, after performing classification or clustering (e.g., some simple clustering), the classifier 650 can send (or ship) the clustering or classification output to the general generator 670. The general generator 670 can receive the output from the classification process (e.g., uninterpreted sensor data) and fuse or combine the received output with the latest trajectory output to create a "general" trajectory from the interpretation that is not interpreted by the trajectory. The general generator 670 can apply the range view based model to the lidar points to perform or optimize some tasks, such as multi-class detection, multi-class trajectory prediction, lidar point segmentation, and class-independent detection. In some embodiments, the general generator 670 can (1) take sensor full data (e.g., camera / radar / lidar) and a map as input, (2) output a general trajectory with instantaneous velocity, and (3) handle multiple objects in the z direction (e.g., objects under an overpass). For example, the output rate of the general generator 670 can be 20 Hz, which matches the trajectory output frequency. If the output range is less than 150m, the size of the smallest object detected may be 20cm, and if the output range is less than 300m, the size of the smallest object detected may be 50cm. The generic generator 670 may cover the desired range (e.g., 150m or 300m), report objects coming from the road (including jersey guardrails) as generic objects, and / or report moving objects that may enter the road in the form of generic tracks.

[0101] Generic generator 670 may include track filter 677 and / or map filter 678 to perform post-processing to reduce false positives. Track filter 677 may interpret or filter or remove duplicate potential generic tracks based on track data 630. For example, track filter 677 may compare potential generic tracks (e.g., tracks based on uninterpreted sensor data and / or interpreted sensor data classified as known objects) with track data (e.g., from Figure 5 In some embodiments, the trajectory filter 677 can compare the attributes of the sensor data of the potential general trajectory (e.g., position information or velocity vector) with the corresponding attributes of the trajectory data 630 (e.g., from Figure 5 If the trajectory data 630 can explain the potential general trajectory or a part thereof based on the comparison result, the trajectory filter 677 can filter out or remove or remove the duplicated explained potential general trajectory or the explained part.

[0102] The map filter 678 can interpret or filter out or remove duplicate potential general trajectories based on the map data 640 (e.g., road boundary information). For example, the map filter 678 can compare the potential general trajectories with the map data (e.g., Figure 5 135 in the map data 640). If the map data 640 is able to interpret the potential general trajectory or a portion thereof as a ground surface based on the comparison result, the map filter 678 can filter out or remove or remove duplicate interpreted potential general trajectories or interpreted portions. In some embodiments, the general generator 670 may include other filters, including (1) a ground filter (e.g., a filter for inclined ground planes with different inclination angles, and types), (2) a ray tracing classification filter (e.g., classification by modeling light transport), (3) a multipath classification filter (e.g., classification by detecting multipath ghost target images), (4) a lidar point erosion filter (e.g., removing pixels on the object boundary), and / or (5) a threshold number of points. In some embodiments, post-processing or filtering can be performed as a weightless component (e.g., an unlearned component) within the ML model (e.g., the classifier 650). In some embodiments, post-processing or filtering can be performed as a weightless component (e.g., an unlearned component) within the ML model. In some embodiments, filtering by the map filter 678 can be performed before filtering by the trajectory filter 677.

[0103] In some embodiments, the remainder interpreter system 600 may only report objects that may cause or receive "harm," where "harm" is platform-independent. In some embodiments, the remainder interpreter system 600 may include sensor data (e.g., Figure 6The residual interpreter system 600 may generate outputs with high accuracy to meet the requirements of a perception system with low false positive tolerance at highway speeds. For example, the residual interpreter system 600 may not only filter out "obscures" but also provide reasonable dynamic estimates. The residual interpreter system 600 may also generate high-recall outputs. For example, the residual interpreter system 600 may report all objects that may cause "harm" and the perception system may use the reported object size to determine "straddle capability". The residual interpreter system 600 may also generate outputs at long distances. For example, the output range may be 160 m behind for a merge, 200 m ahead for a stop for any obstacle, or 300 m ahead to allow enough time for a lane change.

[0104] In some embodiments, the residue interpreter system can have an engineered filter that is used in parallel with the machine learning model to determine object / non-object classification. For example, the engineered filter can include an erosion filter, a multipath classification filter, a ground plane filter, and / or a point count filter.

[0105] Fig. 7A is a graph illustrating the distribution of different types of foreign matter or debris (FOD). Figure 7B is a graph illustrating precision / recall (PR) curves for different operating points of a residual interpreter system according to some embodiments.

[0106] Residue interpreter systems that perform filtering in a post-processing step (e.g. Figure 6 One concern in the trajectory filter 677 or map filter 678) is how the system maintains a high call on general objects. Fig. 7A A distribution 710 of different kinds of FOD that the AV needs to watch out for on the road is shown. Fig. 7A The FODs shown in have a long tail, which are rare events. The residual interpreter system may need a large number of labels to cover these rare events.

[0107] Figure 7BPrecision / Recall (PR) curves are shown, which show the trade-off between precision and recall for different operating points of the residual interpreter system. There are two existing metrics related to residual interpreter systems: (1) fill-in metrics and (2) general metrics. The fill-in metrics can measure the performance of occluder prediction and velocity prediction, while the general metrics measure the overall recall of general objects based on heuristically automatically generated labels. In residual interpreter systems, the output of the model and the published general objects can be measured separately. Figure 7B The PR curves shown in are obtained based on the filling index.

[0108] After training a learning / inference model (e.g., a neural network), an operating point on the PR curve may be selected or chosen. For example, an operating point with high recall and “acceptable” accuracy may be chosen (e.g., Figure 7B The increment from the operating point to 100% call (e.g., Figure 7B The increment from the operating point to 100% accuracy (e.g., Figure 7B 740 in ) corresponds to a certain amount of false positives. When the operating point is close to 100% recall, the residual interpreter system will report too many false positive general objects, which may prevent the AV from running. On the other hand, when the operating point is close to 100% precision, the residual interpreter system will report too many false negative general objects, which may prevent the AV from explaining or predicting "real" general objects. This is a trade-off that exists in all classification schemes of residual interpreter systems.

[0109] In some embodiments, the remainder interpreter system can continue to push the PR curve to the upper right corner (e.g., Figure 7B 760). However, one challenge with using learning / inference models (e.g., artificial neural networks (ANNs)) is to prevent the worst-case scenario from happening. For example, the AV should “never” hit a wall. To prevent such false negatives from happening, an operating point with high recall and “acceptable” precision can be chosen (e.g., Figure 7B 730 in FIG. 1 ). This high recall and “acceptable” accuracy can be achieved by leveraging a set of engineering algorithms coupled with occluder prediction. The engineering algorithms include ground plane filtering (e.g., filters for tilted ground planes with different tilt angles, and types), ray tracing classification (e.g., classification by modeling light transport), multipath classification (e.g., classification by detecting multipath ghost target images), lidar point erosion filters (e.g., removing pixels on object boundaries), and threshold point counts, to name a few.

[0110] Figure 8 is a flow chart illustrating an example method for generating information about measured remainders according to some implementations.

[0111] In some embodiments, the method may be performed by a vehicle computing system (e.g., perception system 154, Figure 5 Residue interpreter in 519, Figure 6 The vehicle computing system includes one or more processors (e.g., Figure 4 410) and one or more memories operably coupled to the one or more processors (e.g., Figure 4 460 in the memory).

[0112] In this example method, the process begins at operation 802, where the remainder interpreter system or one or more processors thereof obtains sensor data (e.g., Figure 5 The lidar data 136, the radar data 134, and the camera data 140; Figure 6 620), which includes measurements from autonomous vehicles (e.g., Figure 1 environment of the vehicle 100) (see Figure 5 In some implementations, the plurality of sensor returns may represent one of a range view or a bird's eye view relative to the autonomous vehicle.

[0113] In operation 804, the remainder interpreter system or one or more processors thereof (e.g., Figure 6 The feature generator 652 of the classifier 650 in the sensor data may extract a first set of features from the sensor data. In some embodiments, the first set of features may include range image features (e.g., features related to a threshold range of proximity), point-by-point features (e.g., features related to objects such as vehicles, cyclists, pedestrians, or known obstructions (e.g., rain, snow, fog, smoke, exhaust, etc.)), and / or filter features (e.g., features related to ground information, point counts, and ray tracing statistics).

[0114] In operation 806, the remainder interpreter system or one or more processors thereof (e.g., Figure 6 The classifier 650 in ) can utilize a machine learning model (e.g., Figure 6 The machine learning model 654 in the embodiment processes the first set of features to generate a first output (e.g., Figure 6 656) and a second output (e.g., Figure 6 The first output classifies the corresponding sensor return as corresponding to an object or a non-object, and the second output indicates a property of the object. In some embodiments, the property of the object may include predicted speed information associated with the corresponding sensor return (e.g., Figure 6at least one of the velocity vector 658 in the image or the predicted position information.

[0115] In some embodiments, while processing the first set of features, the remainder interpreter system or one or more processors (e.g., Figure 6 The classifier 650 in ) can be configured to generate a third output (e.g., Figure 6 In the classification 656 ), the third output classifies the corresponding sensor return as corresponding to a vulnerable participant or a non-vulnerable participant.

[0116] In operation 808, the remainder interpreter system or one or more processors thereof (e.g., Figure 6 The general generator 670 or the track filter 677 in FIG. 6 can classify sensor returns (e.g., Figure 6 656) with multiple pre-classified objects (e.g., Figure 6 The pre-classified object-based trajectory data 630 in the example embodiment is compared to generate one or more general object classifications (eg, generate a general trajectory based on the general object classifications).

[0117] In some embodiments, when comparing sensor returns, the remainder interpreter system or one or more processors (e.g., Figure 6 The track filter 677 in the example may be configured to determine that the corresponding sensor return does not correspond to any of the plurality of pre-classified objects. In response to the determination, the one or more processors may be configured to classify the corresponding sensor return as corresponding to a general object. The general object may be debris. In some embodiments, the general object may be the ground, an animal, debris, a ball, an obstacle, etc. Reference Figure 6 If the trajectory data 630 can explain a potential general trajectory or a portion thereof based on the comparison result, the trajectory filter 677 can filter out or remove or de-duplicate the explained potential general trajectory or the explained portion.

[0118] In some embodiments, when comparing sensor returns, the remainder interpreter system or one or more processors (e.g., Figure 6 The track filter 677 in can be configured to compare the attributes of the object with corresponding attributes of a plurality of pre-classified objects. Figure 6 , the trajectory filter 677 can compare the attributes of the sensor data of the potential general trajectory (e.g., position information or velocity vector) with the corresponding attributes of the trajectory data 630 (e.g., from Figure 5 The trajectory data of the trajectory manager 512 in the example is compared.

[0119] In some embodiments, a residue interpreter system or one or more processors (e.g., Figure 6 The general generator 670 in ) can be further configured to aggregate the first and second outputs returned by the plurality of sensors into a plurality of three-dimensional (3-D) voxels before comparing the sensor returns. During the aggregation, the one or more processors can be configured to average a subset of the first and second outputs returned by the plurality of sensors. For example, referring to Figure 6 If the sensor data at a particular location is not interpreted, the general generator 670 can aggregate the classification data 656 (e.g., point-by-point sensor data) and velocity vectors 658 at the particular location into an output view including a plurality of 3-D voxels. In some implementations, the general generator 670 can perform the aggregation by averaging a subset of the classification data 656 and velocity vectors 658 (e.g., a subset corresponding to a particular location).

[0120] In some embodiments, a residue interpreter system or one or more processors (e.g., Figure 6 The track filter 677 in the example may be further configured to filter the plurality of sensor returns based at least in part on the map data before comparison. In some embodiments, the filtering of the map filter 678 may be performed before the filtering of the track filter 677. The map data may include road boundary information. During filtering, one or more processors may be configured to filter the ground surface information. For example, referring to Figure 6 , the map filter 678 can compare potential generic trajectories with map data (e.g., Figure 5 If the map data 640 is able to interpret the potential general track or a portion thereof as a ground surface based on the comparison result, the map filter 678 may filter out or remove or de-duplicate the interpreted potential general track or the interpreted portion.

[0121] In operation 810, the remainder interpreter system or one or more processors thereof (e.g., Figure 6 The general generator 670 in FIG. 6 may control the autonomous vehicle based at least in part on one or more general object classifications (e.g., general trajectories based on the general object classifications). Figure 6 , the general generator 670 can provide a trajectory manager (e.g., trajectory manager 512) and an autonomous vehicle control system (e.g., Figure 1 The vehicle control system 120 in the embodiment provides a general trajectory (generated based on the general object classification) so that the vehicle is controlled based on the general trajectory.

[0122] IV. Performance issues related to the residual interpreter system

[0123] In some embodiments, the classification process and the general generation process of the residue interpreter system can be two independent processes, but can share a model (e.g., an ML model). One advantage of independent processes is that if one process fails, the other process can run, thereby improving the reliability and availability of the perception system. Another advantage of independent processes is that such an embodiment can provide flexibility for either module in terms of processing speed or frequency, and each module does not have to be synchronized with another module. For example, the general generation process can run as fast as the tracker, while the classification process can run slower. In this way, the residue interpreter system can immediately cover false negatives from the tracker without having to wait for a new run of the classification process.

[0124] One challenge of perception systems is to meet publishing (or reporting) frequency and latency requirements within the constraints of computing resources. In some embodiments, the main classification system and the residual interpreter system can be two independent processes, one of which provides input to the other. For example, the main classification system can provide input to the residual interpreter system and vice versa. In some embodiments, there are two callbacks (or post-call functions) corresponding to the main interpreter system and the residual interpreter system. In some embodiments, the callback of the main interpreter system performs the classification of lidar points triggered by the entry of new data, while the callback of the residual interpreter system performs the interpretation of lidar points and the generation of general objects, which is triggered by the entry of new trajectory bundles. In some embodiments, the two callbacks can run asynchronously as two independent threads. However, due to the prediction density, the publish / subscribe overhead can be expensive.

[0125] In some embodiments, the remainder interpreter callback (or process) can meet several requirements: (1) publication frequency, (2) latency, and / or (3) GPU availability. For example, a specific publication frequency value may be required because it is constrained by the perception provider protocol. In other words, explanations may need to be provided according to a specific publication frequency value. In some cases, the data window for providing explanations should be as low latency as possible, but stale data up to a predetermined latency should be acceptable. To meet this publication frequency requirement, the remainder interpreter callback can maintain an explanation buffer and maintain the explanations by updating / clearing / etc. when new inferences occur, when explanations become stale, etc.

[0126] Under the assumption that each scan batch is performed, the latency can be lower bounded by the operating frequency of the lidar's rolling shutter plus the inference time of the model (e.g., neural network). In some embodiments, the residual interpreter callback can reduce the latency of stream processing, at the expense of more overall computation. With stream processing, the latency can be lower bounded by the lidar packet latency (which is relatively very small) plus the packet processing time (which can also be very small). In the nominal perception case, since most general objects are static, latency may not be an issue in the short term. Ultimately, since there will be moving general objects and nominal tracking false negatives, improving latency will become an inherent part of improving performance.

[0127] A predetermined percentage of GPU availability may be required because if there are other models or processes, the GPU will be available for the remainder interpreter callback a predetermined percentage of the time. Scheduling may be challenging depending on the I / O patterns of the other models sharing the GPU. For example, the remainder interpreter callback may be eager to infer a portion of the lidar points as they come in, but the tracker may have just started and been blocked for a while. In order for scheduling to be efficient, models sharing a GPU may need to run at a comparable cadence. In some embodiments, two models sharing a GPU can run at a comparable cadence by having both models perform batch processing at a predetermined frequency, or both models perform stream processing using a shared process queue.

[0128] In some embodiments, GPU availability can be met by setting a certain number of GPU threads (e.g., gpu_thread_limit) for models or processes located on the same device to facilitate concurrent processing of the residual interpreter callback and other models on the GPU. This can ensure that any particular model or process does not occupy more SMs (streaming multiprocessors) than the configured gpu_thread_limit, and the remaining available SMs can be used for other processes. In some embodiments, the gpu_thread_limit of the tracker can be set to a predetermined percentage (e.g., α%) so that (100- α)% is available for the residual interpreter callback. In this way, the residual interpreter inference model will not be blocked by the tracker for its entire inference cycle, so the residual interpreter inference model and the tracker can run more efficiently at different rhythms, thereby improving GPU utilization and achieving the target inference frequency with more computational margin. In some embodiments, the scheduling of the process cannot be controlled, so the behavior of the process cannot be guaranteed. For example, another inference process (by radar or camera) and the residual interpreter callback can be interleaved so that they both run at a predetermined frequency and each takes a predetermined amount of inference time. However, there is no guarantee that the inference of processes will occur in an interleaved manner.

[0129] In some embodiments, the remainder interpreter callback (or process) can use a single instruction multiple data (SIMD) library to perform inference on the CPU, especially for heterogeneous network designs where GPUs do not have a strong advantage.

[0130] Taking all factors into consideration, in some embodiments, the main interpreter callback and the remainder interpreter callback can run in two independent threads. In some embodiments, the entire interpreter module can be split into two modules that can run in two independent threads. In some embodiments, a ring buffer can be used instead of passing protocol buffers to perform inter-process communication (IPC) between the two threads, especially where large message sizes are of concern. In some embodiments, since the ring buffer is not recorded, the producer of the ring buffer can be run with the consumer in order to run offline. In some embodiments, gpu_thread_limit can be used to divide the availability of the GPU so that no single model or process will block other models or processes from processing. Batch processing of lidar points can be performed at a predetermined frequency. The computational margin can be confirmed by experimenting with the configuration of gpu_thread_limit. The remainder interpreter callback (or process) can be run when the trajectory of the latest inference result comes in.

[0131] The previous description is provided to enable those skilled in the art to practice the various aspects described herein. Various modifications to these aspects will be apparent to those skilled in the art, and the general principles defined herein may be applied to other aspects. Therefore, the claims are not intended to be limited to the aspects shown herein, but should be given the full scope consistent with the language claims, wherein, unless expressly stated, references to singular elements are not intended to mean "one and only one", but "one or more". Unless expressly stated otherwise, the term "some" refers to one or more. All structural and functional equivalents of the elements of the various aspects described throughout the preceding description that are known or later known to those of ordinary skill in the art are expressly incorporated herein by reference and are intended to be included in the claims. In addition, regardless of whether such disclosure is expressly recorded in the claims, nothing disclosed herein is intended to be dedicated to the public. Unless any claim element is expressly recorded using the phrase "means for...", the element should not be interpreted as a means plus function.

[0132] It should be understood that the specific order or hierarchy of blocks in the disclosed process is an example of an illustrative method. Based on design preferences, it is understood that the specific order or hierarchy of blocks in the process can be rearranged while remaining within the scope of the previous description. The accompanying method claims present elements of the various blocks in an example order and are not meant to be limited to the specific order or hierarchy presented.

[0133] The description of the foregoing disclosed embodiments is provided to enable those skilled in the art to make or use the disclosed subject matter. Those skilled in the art will readily appreciate various modifications to these embodiments, and the general principles defined herein may be applied to other embodiments without departing from the spirit or scope of the foregoing description. Therefore, the foregoing description is not intended to be limited to the embodiments shown herein, but should be given the widest scope consistent with the principles and novel features disclosed herein.

[0134] The various examples shown and described are provided as examples only to illustrate the various features of the claims. However, the features shown and described with respect to any given example are not necessarily limited to the relevant example and can be used or combined with other examples shown and described. In addition, the claims are not intended to be limited by any one example.

[0135] The foregoing method descriptions and process flow charts are provided as illustrative examples only and are not intended to require or imply that the blocks of the various examples must be executed in the order presented. Those skilled in the art will understand that the order of the blocks in the foregoing examples can be executed in any order. Words such as "thereafter," "then," and "next" are not intended to limit the order of the blocks; these words are only used to guide the reader to read the description of the method. In addition, any reference to a claim element in the singular (e.g., using the articles "a," "an," or "the") should not be interpreted as limiting the element to the singular.

[0136] The various illustrative logic blocks, modules, circuits, and algorithm blocks described in connection with the examples disclosed herein may be implemented as electronic hardware, computer software, or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and blocks have been generally described above with respect to their functionality. Whether these functions are implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. A skilled person may implement the described functions in different ways for each specific application, but such implementation decisions should not be interpreted as causing a deviation from the scope of the present disclosure.

[0137] The hardware for implementing the various illustrative logics, logic blocks, modules, and circuits described in connection with the examples disclosed herein may be implemented or performed with a general purpose processor, a DSP, an ASIC, an FPGA, or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to perform the functions described herein. A general purpose processor may be a microprocessor, but, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, for example, a combination of a DSP and a microprocessor, a plurality of microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Alternatively, some blocks or methods may be performed by circuits specific to a given function.

[0138] In some examples, the functions described may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, the functions may be stored as one or more instructions or codes on a non-temporary computer-readable storage medium or a non-temporary processor-readable storage medium. The blocks of the methods or algorithms disclosed herein may be embodied in a processor-executable software module, which may reside on a non-temporary computer-readable or processor-readable storage medium. A non-temporary computer-readable or processor-readable storage medium may be any storage medium that can be accessed by a computer or processor. As an example but not limited to, such a non-temporary computer-readable or processor-readable storage medium may include RAM, ROM, EEPROM, FLASH memory, CD-ROM or other optical disk storage, disk storage or other magnetic storage device, or any other medium that can be used to store the required program code in the form of instructions or data structures and can be accessed by a computer. The disks and optical disks used herein include compact disks (CDs), laser disks, optical disks, digital versatile disks (DVDs), floppy disks, and blue-ray disks, wherein disks typically reproduce data magnetically, while optical disks use lasers to reproduce data optically. The above combinations are also included within the scope of non-temporary computer-readable and processor-readable media. Furthermore, the operations of a method or algorithm may reside as any combination or set of codes and / or instructions on a non-transitory processor-readable storage medium and / or a computer-readable storage medium, which may be incorporated into a computer program product.

[0139] The foregoing description of the disclosed examples is provided to enable those skilled in the art to make or use the present disclosure. Those skilled in the art will readily appreciate various modifications to these examples, and the general principles defined herein may be applied to some examples without departing from the spirit or scope of the present disclosure. Therefore, the present disclosure is not intended to be limited to the examples shown herein, but is to be given the widest scope consistent with the appended claims and the principles and novel features disclosed herein.

Claims

1. A method comprising: (a) obtaining sensor data including a plurality of sensor returns from an environment of the autonomous vehicle, wherein the plurality of sensor returns include at least one of lidar data, radar data, or camera data; (b) extracting a first set of features from the sensor data; (c) processing the first set of features using a machine learning model to generate a first output and a second output for at least a subset of the plurality of sensor returns, the first output classifying the corresponding sensor return as corresponding to an object or a non-object, and the second output indicating a property of the object, wherein the non-object includes at least one of rain, snow, fog, smoke, or exhaust gas; (d) comparing the sensor returns classified as corresponding to an object to a plurality of pre-classified objects to generate one or more general object classifications; and (e) controlling the autonomous vehicle based at least in part on the one or more general object classifications, Among them, (d) includes: (f) determining that the corresponding sensor return does not correspond to any of the plurality of pre-classified objects; and In response to (f), the corresponding sensor return is classified as corresponding to a general object.

2. The method according to claim 1, further comprising: Prior to (d), the plurality of sensor returns are filtered based at least in part on the map data.

3. The method according to claim 2, wherein: The map data includes road boundary information.

4. The method according to claim 2, wherein: The filtering includes filtering ground surface information.

5. The method according to claim 1, wherein: The general object is a fragment.

6. The method according to claim 1, wherein: (d) comprising comparing the attribute of the object with corresponding attributes of the plurality of pre-classified objects.

7. The method according to claim 1, further comprising: Prior to (d), the first outputs and the second outputs returned by the plurality of sensors are aggregated into a plurality of three-dimensional (3-D) voxels.

8. The method according to claim 7, wherein: The aggregating includes averaging a subset of the first and second outputs returned by the plurality of sensors.

9. The method according to claim 1, wherein: The plurality of sensors return information representing one of a range view or a bird's eye view relative to the autonomous vehicle.

10. The method according to claim 1, wherein: The properties of the object include at least one of predicted speed information or predicted position information associated with the corresponding sensor return.

11. The method according to claim 1, wherein: (c) comprising: generating, for at least the subset of the plurality of sensor returns, a third output that classifies the respective sensor returns as corresponding to a vulnerable actor or a non-vulnerable actor.

12. A system comprising one or more processors and one or more memories, the one or more memories being operably coupled to the one or more processors, wherein: The one or more memories store instructions, which, in response to execution of the instructions by one or more processors, cause the one or more processors to perform the following operations: (a) obtaining sensor data including a plurality of sensor returns from an environment of the autonomous vehicle, wherein the plurality of sensor returns include at least one of lidar data, radar data, or camera data; (b) extracting a first set of features from the sensor data; (c) processing the first set of features using a machine learning model to generate, for at least a subset of the plurality of sensor returns, a first output that classifies the corresponding sensor return as corresponding to an object or a non-object and a second output that indicates an attribute of the object; (d) comparing the sensor returns classified as corresponding to an object to a plurality of pre-classified objects to generate one or more general object classifications; and (e) controlling the autonomous vehicle based at least in part on the one or more general object classifications, Wherein, when executing (d), the one or more processors are configured to: (f) determining that the corresponding sensor return does not correspond to any of the plurality of pre-classified objects; and In response to (f), classifying the corresponding sensor return as corresponding to a general object, in, The autonomous vehicle is controlled based on sensor returns classified as corresponding to an object, and The autonomous vehicle is not controlled based on sensor returns classified as corresponding to non-objects.

13. The system according to claim 12, wherein: In implementation (d), the one or more processors are configured to compare the attribute of the object with corresponding attributes of the plurality of pre-classified objects.

14. The system according to claim 12, wherein: The properties of the object include at least one of predicted speed information or predicted position information associated with the corresponding sensor return.

15. The system of claim 12, wherein: The map data includes road boundary information.

Images