Camera-radar data fusion for efficient target detection
By using end-to-end sensing models in autonomous vehicles for multi-scale BEV grid fusion of camera and radar data, the problem of high accuracy and cost of object detection in autonomous vehicles is solved, and fast and accurate object detection and tracking is achieved, which improves the Safety and resource efficiency of driving automation.
Patent Information
- Application Number
- CN202380033221.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Priority Date
- 2023-02-13
- Filing Date
- 2023-02-14
- Publication Date
- 2024-11-08
AI Technical Summary
When existing autonomous vehicles detect and classify objects, especially based on camera and radar data fusion, there are problems of high accuracy and cost, and it is difficult to achieve fast and resource-effective object detection.
Using end-to-end perception model (EEPM) to fusion through multi-scale bird's-eye view (BEV) grids of camera and radar data, a fused BEV grid is generated for object detection, using multiple sets of feature transformation and voxelization technology, combined with machine Learn the model for object recognition and tracking.
It realizes fast and accurate object detection and tracking, reduces calculation overhead, is suitable for various levels of driving automation, and improves the safety and resource utilization efficiency of autonomous vehicles.
Smart Images

Figure CN118922742A_ABST
Abstract
Description
Technical Field
[0001] The present specification generally relates to systems and applications for detecting and classifying objects, and in particular to autonomous vehicles and vehicles deploying driver assistance technologies. More specifically, the present specification relates to camera-radar data fusion for faster and more resource-efficient detection of objects, including but not limited to vehicles, pedestrians, cyclists, animals, etc. Background Art
[0002] Autonomous (fully or partially autonomous) vehicles (AVs) operate by sensing the external environment with various electromagnetic (e.g., radar and optical) and non-electromagnetic (e.g., audio and humidity) sensors. Some autonomous vehicles plot a driving path through the environment based on the sensed data. The driving path can be determined based on Global Navigation Satellite System (GNSS) data and road map data. While GNSS and road map data can provide information about static aspects of the environment (buildings, street layout, road closures, etc.), dynamic information (such as information about other vehicles, pedestrians, street lights, etc.) is obtained from simultaneously collected sensory data. The accuracy and safety of the driving path and rate scheme selected by the autonomous vehicle depends on the timely and accurate recognition of the various objects present in the driving environment, and on the ability of the driving algorithm to process information about the environment and provide correct instructions to the vehicle control and powertrain. Summary of the invention
[0003] In one embodiment, a method is disclosed, the method comprising: obtaining, by a processing device, input data derived from a set of sensors associated with an autonomous vehicle (AV), extracting, by the processing device, a plurality of sets of features from the input data, generating, by the processing device, a fused bird's eye view (BEV) mesh using the plurality of sets of features. The fused BEV mesh is generated based on a first BEV mesh having a first scale and a second BEV mesh having a second scale different from the first scale. The method further comprises providing, by the processing device, the fused BEV mesh for object detection.
[0004] In another embodiment, a system is disclosed, the system comprising a memory and a processing device operably coupled to the memory, the processing device configured to obtain input data derived from a set of sensors associated with an autonomous vehicle (AV), extract multiple sets of features from the input data, and generate a fused bird's eye view (BEV) mesh using the multiple sets of features. The fused BEV mesh is generated based on a first BEV mesh having a first scale and a second BEV mesh having a second scale different from the first scale. The processing device is also configured to provide the fused BEV mesh for object detection.
[0005] In yet another embodiment, a non-transitory computer-readable storage medium having stored thereon instructions is disclosed, the instructions, when executed by a processing device, causing the processing device to perform operations, the operations including obtaining input data derived from a set of sensors associated with an autonomous vehicle (AV). The set of sensors includes at least one camera and at least one radar, and the input data includes a set of camera data obtained from the at least one camera and a set of radar data obtained from the at least one radar. The operations also include extracting multiple sets of features from the input data. The multiple sets of features include a set of camera data features generated from the set of camera data and a set of radar data features generated from the set of radar data. The operations also include generating a fused bird's eye view (BEV) grid using the multiple sets of features. The fused BEV grid is generated based on a first BEV grid having a first scale and a second BEV grid having a second scale different from the first scale. The operations further include providing the fused BEV grid for object detection. BRIEF DESCRIPTION OF THE DRAWINGS
[0006] The present disclosure is illustrated by way of example and not limitation and may be more fully understood with reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:
[0007] 1 is a diagram illustrating components of an example autonomous vehicle (AV), according to some implementations of the present disclosure.
[0008] 2A-2B are diagrams illustrating an example architecture of a portion of a perception system capable of efficiently detecting and classifying objects according to some embodiments of the present disclosure.
[0009] 3 is a diagram illustrating an example method of generating a fused bird's eye view (BEV) mesh from a multi-scale BEV mesh, according to some embodiments of the present disclosure.
[0010] 4A-4B illustrate example methods of implementing camera-radar data fusion to generate a fused bird's eye view (BEV) grid for efficient object detection, according to some embodiments of the present disclosure.
[0011] 5A-5B illustrate example methods for implementing a bird's eye view (BEV) grid generated using camera-radar data fusion for efficient object detection, according to some embodiments of the present disclosure.
[0012] 6 depicts a block diagram of an example computer device capable of implementing camera-radar data fusion to produce a fused bird's eye view (BEV) grid for efficient object detection, according to some embodiments of the present disclosure. DETAILED DESCRIPTION
[0013] Although various embodiments may be described below using autonomous driving systems and driver assistance systems as examples for purposes of illustration, it should be understood that the techniques and systems described herein may be used to track objects in a wide range of applications, including aviation, marine applications, traffic control, animal control, industrial and academic research, public and personal safety, or in any other application where automatic detection of objects is advantageous.
[0014] In one example, for the safety of autonomous driving operations, it may be desirable to develop and deploy technologies for rapid and accurate detection, classification, and tracking of various road users and other objects encountered on or near the road (such as road obstacles, construction equipment, roadside structures, etc.). Autonomous vehicles (as well as various driver assistance systems) can utilize multiple sensors to facilitate the detection of objects in the driving environment and determine the movement of these objects. Sensors typically include radio detection and ranging sensors (radars), light detection and ranging sensors (lidars), various types of digital cameras, sonars, position sensors, etc. Different types of sensors provide different and often complementary benefits. For example, radars and lidars transmit electromagnetic signals (radio signals or light signals) that reflect from objects and carry information that allows the distance to the object (e.g., based on the flight time of the signal) and the speed of the object (e.g., based on the Doppler shift of the signal frequency) to be determined. Radars and lidars can cover a full 360-degree view, for example, by using a scanning transmitter of a sensing beam. The sensing beam can cause many reflections covering the driving environment in a dense grid of return points. Each return point can be associated with the distance to the corresponding reflecting object and the radial velocity (the component of the velocity along the line of sight) of the reflecting object.
[0015] Some systems and methods for object recognition and tracking use various sensing modalities (such as lidar, radar, cameras, etc.) to obtain images of the environment. The images can then be processed by a trained machine learning model to identify the locations of various objects in the image (e.g., in the form of bounding boxes), the motion state of the objects (e.g., velocity as detected by a lidar or radar Doppler effect based sensor), the type of object (e.g., vehicle or pedestrian), etc. The motion of the objects (or any other evolution, such as splitting a single object into multiple objects) can be performed by creating and maintaining tracks associated with specific objects.
[0016] The use of multiple sensing modalities (e.g., lidar, radar, camera) to obtain generally complementary data improves the accuracy of object detection, recognition, and tracking, but at a high cost in terms of sensing hardware and processing software. For example, lidar sensors can provide valuable information about the distance to various reflective surfaces in the external environment. However, lidar sensors are expensive optical and electronic devices that operate by actively detecting the external environment using optical signals and require considerable maintenance and periodic calibration. The lidar returns (point clouds) must be processed, segmented into groups associated with separate hypothetical objects, and matched with objects detected using other sensing modalities (e.g., cameras), which requires additional processing and memory resources. On the other hand, cameras operate by passively collecting light (and / or infrared electromagnetic waves) emitted (or reflected) by objects of the environment, and are significantly simpler and cheaper in design, installation, and operation. Therefore, various driver assistance systems that do not deploy lidar (for cost and maintenance reasons) are typically equipped with one or more cameras. Cameras can also be more easily installed at various fixed locations and are used for traffic monitoring and control, public and private safety applications, etc. Cameras have certain advantages over radar, which, while allowing the distance to an object (and the object's velocity) to be detected, operates in a wavelength range that has an inherently lower resolution than cameras. Therefore, the ability to detect and identify objects based on camera images alone is beneficial.
[0017] However, cameras produce a projection of the three-dimensional (3D) external environment onto a two-dimensional imaging surface (e.g., the camera's array of photodetectors), which can be a flat or curved surface. This creates two related challenges. On the one hand, the distance to an object (often referred to as the depth of an object in an image) is not immediately known (although it can usually be determined from the context of the imaged object). On the other hand, camera images have perspective distortion, resulting in the same number of pixels of an image of separated objects corresponding to different distances between objects, depending on the depth of the object. In addition, objects that are depicted close to each other can still be separated by a considerable distance (e.g., a car and a pedestrian visible behind the car). Machine learning techniques for object detection sometimes attempt to map objects from a perspective view to a top-down view, also known as a bird's-eye view (BEV), where objects are represented on a convenient manifold, e.g., a plane viewed from above, and are characterized by a simple set of Cartesian coordinates. Object recognition and tracking can then be performed directly within the BEV representation. The success of such techniques depends on an accurate mapping of objects to the BEV representation. This in turn can rely on accurate estimates of the distances to various objects, since misalignment of objects within the BEV representation can not only lead to errors in determining the distance to a road user, but can also lead to the loss of important contextual information.
[0018] Aspects and embodiments of the present disclosure address these and other challenges by implementing methods and systems that can implement camera-radar data fusion and multi-scale BEV grids. In particular, the disclosed technology provides an end-to-end perception model (EEPM) that may include a set of neural networks (NNs) trained to process input data from a set of sensors of an AV. For example, the input data may include camera data and radar data. The input data may be used to generate a fused BEV grid formed by fusing multiple BEV grids together. More specifically, at least two of the BEV grids may have different scales (e.g., resolution and / or size). For example, a first BEV grid may have a first scale, and a second BEV grid may have a second scale that is different from the first scale. Each BEV grid may be generated by transforming a corresponding feature group extracted from the input data into a corresponding point group and voxelizing the point groups to generate multiple BEV grids. Therefore, the BEV grid may also be referred to as a BEV voxel grid.
[0019] For example, a set of camera data features may be extracted from the camera data, and the set of camera data features may be transformed into a set of pixel points. Transforming the set of camera data features into a set of pixel points may include projecting three-dimensional (3D) camera data onto a two-dimensional (2D) space. As another example, a set of radar data features may be extracted from radar data, and the set of radar data features may be transformed into a set of radar points. Transforming the set of radar data features into a set of radar points may include transforming a coordinate representation of the radar data. For example, if the radar data has a polar coordinate representation, the coordinate representation of the radar data may be transformed into a Cartesian coordinate representation.
[0020] The EEPM can be trained using sensor dropout scenarios, where some sensors are removed or inoperable (e.g., at least one camera and / or at least one radar). For example, the right-facing camera can be removed, and information about objects in the portion of the space covered by the right-facing camera can be provided by other sensing modalities (e.g., lidar and / or radar sensors). Training scenarios can also include a complete dropout of a particular sensing modality, such as the dropout of a lidar data feed so that all information about the environment is provided by the camera and radar. This trains the output of the EEPM to be robust to failures of individual sensors and entire sensing modalities. Depending on the computational complexity and the complexity of the training, the EEPM can be used for various levels of driving automation, including Level 2 driver assistance systems, Level 3 contextual autonomous driving, Level 4 dominant autonomous driving, Level 5 fully autonomous driving, and other implementations.
[0021] Advantages of the described embodiments include (but are not limited to) quickly and accurately detecting, identifying, and tracking objects in a manner that avoids the large computational overhead of processing data from multiple sensing modalities. Because the trained and deployed machine learning models as disclosed herein are capable of effective object detection based on input data (e.g., camera data and radar data), the EEPM models described herein can be deployed on a variety of platforms (e.g., AVs), including systems with modest computing resources.
[0022] 1 is a diagram showing components of an example autonomous vehicle (AV) 100 according to some embodiments of the present disclosure. Autonomous vehicles may include motor vehicles (cars, trucks, buses, motorcycles, all-terrain vehicles, recreational vehicles, any specialized agricultural or construction vehicles, etc.), aircraft (aircraft, helicopters, drones, etc.), naval vehicles (ships, boats, yachts, submarines, etc.), spacecraft (controllable objects operating outside the Earth's atmosphere), or any other self-propelled vehicle capable of operating in an autonomous mode (without or with reduced human input) (e.g., a robot, a factory or warehouse robotic vehicle, a sidewalk delivery robotic vehicle, etc.).
[0023] Vehicles (such as those described herein) can be configured to operate in one or more different driving modes. For example, in a manual driving mode, the driver can directly control acceleration, deceleration, and steering via inputs such as an accelerator pedal, a brake pedal, a steering wheel, etc. The vehicle can also operate in one or more autonomous driving modes, including, for example, a semi-autonomous or partially autonomous driving mode in which a person performs a certain amount of direct or remote control of the driving operation, or a fully autonomous driving mode in which the vehicle handles the driving operation without direct or remote control of a person. These vehicles may be referred to by different names, including, for example, autonomous driving vehicles, autonomous vehicles, etc.
[0024] As described herein, in semi-autonomous or partially autonomous driving modes, even if the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), or emergency braking), the human driver is expected to be situationally aware of the vehicle's surroundings and supervise the assisted driving operations. Here, even if the vehicle can perform all driving tasks in certain situations, the human driver is expected to be responsible for taking control as needed.
[0025] Although various systems and methods may be described below in conjunction with autonomous vehicles for the sake of brevity and simplicity, similar techniques may be used in various driver assistance systems that do not rise to the level of fully autonomous driving systems. In the United States, the Society of Automotive Engineers (SAE) has defined different levels of autonomous driving operation to indicate the extent to which the vehicle controls driving, although different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used to implement SAE Level 2 driver assistance systems for steering, braking, acceleration, lane centering, adaptive cruise control, etc., as well as other driver support. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems that are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 autonomous driving systems, which operate autonomously in most regular driving situations and require only occasional attention from a human operator. In all such driver assistance systems, accurate lane estimation can be performed automatically without driver input or control (e.g., when the vehicle is in motion), and leads to improvements in the reliability of vehicle positioning and navigation and the overall safety of autonomous, semi-autonomous, and other driver assistance systems. As previously mentioned, in addition to the way that the SAE classifies the levels of automated driving operation, other organizations in the United States or other countries may classify the levels of automated driving operation differently. Without limitation, the systems and methods disclosed herein may be used in driver assistance systems defined by the automated driving operation levels of these other organizations.
[0026] The driving environment 101 may include any object (animate or inanimate) located outside the AV, such as roads, buildings, trees, bushes, sidewalks, bridges, mountains, other vehicles, pedestrians, docks, embankments, landing strips, animals, birds, etc. The driving environment 101 may be urban, suburban, rural, etc. In some embodiments, the driving environment 101 may be an off-road environment (e.g., a farm or other agricultural land). In some embodiments, the driving environment 101 may be an indoor environment, such as an industrial plant, a shipping warehouse, a hazardous area of a building, etc. In some embodiments, the driving environment 101 may be substantially flat, with various objects moving parallel to the surface (e.g., parallel to the surface of the earth). In other embodiments, the driving environment 101 may be three-dimensional and may include objects that can move in all three directions (e.g., balloons, fallen leaves, etc.). Hereinafter, the term "driving environment" should be understood to include all environments in which autonomous motion (e.g., SAE Level 5 and SAE Level 4 systems), conditional autonomous motion (e.g., SAE Level 3 systems), and / or motion of a vehicle equipped with driver assistance technology (e.g., SAE Level 2 systems) may occur. Additionally, the "pilot environment" may include any possible flight environment for an aircraft (or spacecraft) or a marine environment for a naval vessel. Objects of the pilot environment 101 may be located at any distance from the AV, from a close distance of a few feet (or less) to a few miles (or more).
[0027] The exemplary AV 100 may include a sensing system 110. The sensing system 110 may include various electromagnetic (e.g., optical, infrared, radio waves, etc.) and non-electromagnetic (e.g., acoustic) sensing subsystems and / or devices. The sensing system 110 may include one or more lidars 112, which may be laser-based units capable of determining the distance to an object and the velocity of an object in the driving environment 101. The sensing system 110 may include one or more radars 114, which may be any system that utilizes radio or microwave frequency signals to sense objects within the driving environment 101 of the AV 100. The lidar 112 and / or radar 114 may be configured to sense the spatial position of an object (including its spatial dimensions) and the velocity of the object (e.g., using Doppler shift techniques). Hereinafter, "velocity" refers to how fast an object is moving (the velocity of the object) and the direction of the object's motion. Each of the lidar(s) 112 and radar(s) 114 may include a coherent sensor, such as a frequency modulated continuous wave (FMCW) lidar or radar sensor. For example, the lidar(s) 112 and / or radar(s) 114 may use heterodyne detection for velocity determination. In some embodiments, the functionality of ToF and coherent lidar (or radar) are combined into a lidar (or radar) unit that is capable of simultaneously determining the distance to a reflecting object and the radial velocity of the reflecting object. Such a unit may be configured to operate in an incoherent sensing mode (ToF mode) and / or a coherent sensing mode (e.g., a mode using heterodyne detection) or in both modes simultaneously. In some embodiments, multiple lidars 112 and / or radars 114 may be mounted on the AV 100.
[0028] The lidar 112 (and / or radar 114) may include one or more light sources (and / or radio / microwave sources) that generate and transmit signals and one or more detectors of signals reflected from an object. In some embodiments, the lidar 112 and / or radar 114 may perform a 360-degree scan in the horizontal direction. In some embodiments, the lidar 112 and / or radar 114 may be capable of spatial scanning in both the horizontal and vertical directions. In some embodiments, the field of view may be up to 60 degrees in the vertical direction (e.g., scanning at least a portion of the area above the horizon with a lidar or radar signal). In some embodiments (e.g., aerospace applications), the field of view may be a complete sphere (consisting of two hemispheres).
[0029] The sensing system 110 may also include one or more cameras 118 to capture images of the driving environment 101. The cameras 118 may operate in the visible portion of the electromagnetic spectrum, such as the 300-800 nm wavelength range (also referred to herein as the optical range for simplicity). Some of the optical range cameras 118 may use a global shutter, while other cameras 118 may use a rolling shutter. The image may be a two-dimensional projection of the driving environment 101 (or a portion of the driving environment 101) onto a projection surface (flat or non-flat) of the camera. Some cameras 118 of the sensing system 110 may be video cameras configured to capture a continuous (or quasi-continuous) stream of images of the driving environment 101. The sensing system 110 may also include one or more sonars 116 (e.g., ultrasonic sonars) for active sound detection of the driving environment 101 and one or more microphones for passively listening to sounds of the driving environment 101. The sensing system 110 may also include one or more infrared range (IR) sensors 119. For example, the IR sensor(s) 119 may include an IR camera. The IR sensor(s) 119 may use focusing optics (e.g., made of germanium-based materials, silicon-based materials, etc.) configured to operate in a wavelength range from micrometers to tens of micrometers or more. The IR sensor(s) 119 may include a phased array of IR detector elements. A pixel of an IR image generated by the IR sensor(s) 119 may represent the total amount of IR radiation collected by the corresponding detector element (associated with the pixel), the temperature of a physical object whose IR radiation is collected by the corresponding detector element, or any other suitable physical quantity.
[0030] The sensory data obtained by the sensing system 110 may be processed by the data processing system 120 of the AV 100. For example, the data processing system 120 may include a perception system 130. The perception system 130 may be configured to detect and track objects in the driving environment 101 and to identify detected objects. For example, the perception system 130 may analyze images captured by the camera 118 and be able to detect traffic light signals, road signs, road layout (e.g., boundaries of traffic lanes, topology of intersections, designation of parking lots, etc.), the presence of obstacles, etc. The perception system 130 may also receive radar sensing data (Doppler data and ToF data) to determine the distance to various objects in the environment 101 and the speed of such objects (radial, and in some embodiments, lateral, as described below). In some embodiments, the perception system 130 may use the radar data in conjunction with the data captured by the camera 118, as described in more detail below.
[0031] The perception system 130 may include one or more components to facilitate detection, classification, and tracking of objects, including an end-to-end perception model (EEPM) 132 that may be used to process data provided by the sensing system 110. More specifically, in some embodiments, the EEPM 132 may receive data from sensors of different sensing modalities. For example, the EEPM 132 may receive images from at least some of the (one or more) lidars 112, (one or more) radars 114, and (one or more) (optical range) cameras 118, (one or more) IR sensors 119, (one or more) sonars 116, etc. In particular, the EEPM 132 may include one or more trained machine learning models (MLMs) that process some or all of the above data to detect, classify, and track the movement of various objects in the driving environment 101. The EEPM 132 may use multiple classifier heads to determine various attributes of the external environment, including, but not limited to, the occupancy of space by various objects, the type of objects, the movement of objects, the identification of objects that may be occluded, the relationship of objects to the road, the relationship of objects to other objects, and the relationship of objects to traffic flow. Various models of the EEPM 132 may be trained using sets of images / data that are annotated to identify specific features in the corresponding sensed data. In some implementations, the perception system 130 may include a behavior prediction module (BPM) 134 that predicts future motion of detected objects.
[0032] The perception system 130 may also receive information from a global navigation satellite system (GNSS) positioning subsystem (not shown in FIG. 1 ), which may include a GNSS transceiver (not shown) configured to obtain information about the position of the AV relative to the earth and its surroundings. The positioning subsystem may use positioning data (e.g., GNNS and inertial measurement unit (IMU) data) in conjunction with the sensory data to help accurately determine the position of the AV relative to fixed objects (e.g., roads, lane boundaries, intersections, sidewalks, crosswalks, road signs, curbs, surrounding buildings, etc.) of the driving environment 101, the positions of which may be provided by the map information 124. In some embodiments, the data processing system 120 may receive non-electromagnetic data, such as audio data (e.g., ultrasonic sensor data from sonar 116 or data from a microphone picking up an emergency vehicle sirens), temperature sensor data, humidity sensor data, pressure sensor data, meteorological data (e.g., wind speed and direction, precipitation data), etc.
[0033] The data processing system 120 may also include an environment monitoring and prediction component 126 that can monitor how the driving environment 101 evolves over time, for example, by tracking the position and velocity of animated objects (e.g., relative to the earth). In some embodiments, the environment monitoring and prediction component 126 can track changes in the appearance of the environment due to the movement of the AV relative to the environment. In some embodiments, the environment monitoring and prediction component 126 can predict how various animated objects of the driving environment 101 will be positioned within the predicted time range. The prediction can be based on the current state of the animated object, including the current position (coordinates) and velocity of the animated object. In addition, the prediction can be based on the motion history (tracked dynamics) of the animated object during a certain time period before the current moment. For example, based on the stored data of the first object indicating the accelerated motion of the first object during the previous 3-second time period, the environment monitoring and prediction component 126 can infer from the stop sign or red traffic light signal that the first object is resuming its motion. Therefore, given the layout of the road and the presence of other vehicles, the environment monitoring and prediction component 126 can predict the position of the first object that may be within the next 3 seconds or 5 seconds of motion. As another example, based on stored data of a second object indicating decelerating motion of the second object during a previous 2 second time period, the environment monitoring and prediction component 126 may conclude that the second object is stopping at a stop sign or a red traffic light. Thus, the environment monitoring and prediction component 126 may predict where the second object is likely to be in the next 1 second or 3 seconds. The environment monitoring and prediction component 126 may perform periodic checks on the accuracy of its predictions and modify the predictions based on new data obtained from the sensing system 110. The environment monitoring and prediction component 126 may operate in conjunction with the EEPM 132. For example, the environment monitoring and prediction component 126 may track the relative motion of the AV and various objects (e.g., reference objects that are stationary or moving relative to the earth).
[0034] The data generated by the perception system 130, the GNSS processing module 122, and the environment monitoring and prediction component 126 can be used by an autonomous driving system such as an AV control system (AVCS) 140. The AVCS 140 may include one or more algorithms that control how the AV behaves in various driving situations and environments. For example, the AVCS 140 may include a navigation system for determining a global driving route to a destination point. The AVCS 140 may also include a driving path selection system for selecting a specific path through the driving environment 101, which may include selecting a lane, negotiating traffic jams, selecting a place to make a U-turn, selecting a trajectory for a parking maneuver, etc. The AVCS 140 may also include an obstacle avoidance system for safely avoiding various obstacles (rocks, stalled vehicles, etc.) within the driving environment of the AV. The obstacle avoidance system may be configured to evaluate the size of the obstacle and the trajectory of the obstacle (if the obstacle is animated) and select the best driving strategy (e.g., braking, steering, acceleration, etc.) for avoiding the obstacle.
[0035] The algorithms and modules of the AVCS 140 may generate instructions for various systems and components of the vehicle, such as the powertrain, brakes and steering 150, vehicle electronics 160, signaling 170, and other systems and components not explicitly shown in FIG. 1 . The powertrain, brakes and steering 150 may include an engine (internal combustion engine, electric engine, etc.), a transmission, a differential, axles, wheels, a steering mechanism, and other systems. The vehicle electronics 160 may include an onboard computer, engine management, ignition, communication systems, onboard computers, telematics, in-vehicle entertainment systems, and other systems and components. The signaling 170 may include high and low headlights, parking lights, turn and reverse lights, horns and sirens, interior lighting systems, dashboard notification systems, passenger notification systems, radio and wireless network transmission systems, and the like. Some commands output by the AVCS 140 may be delivered directly to the powertrain, brakes, and steering 150 (or signaling devices 170 ), while other commands output by the AVCS 140 are first delivered to the vehicle electronics 160 , which generate commands to the powertrain, brakes, and steering 150 and / or signaling devices 170 .
[0036] In one example, the EEPM 132 may determine that an image obtained by the camera(s) 118 includes a depiction of an object, and may further classify the object as a cyclist. The environmental monitoring and prediction component 126 may track the cyclist and determine that the cyclist is traveling at a rate of 15 mph along an intersecting road perpendicular to the direction of vehicle motion. In response to such a determination, the BPM 134 may determine that the vehicle needs to slow down to allow the cyclist to exit the intersection. The AVCS 140 may output instructions to the powertrain, brakes, and steering 150 (directly or via the vehicle electronics 160) to: (1) reduce fuel flow to the engine by modifying the throttle setting to reduce engine rpm; (2) downshift the driveline to a lower gear via the automatic transmission; and (3) engage the brake unit to reduce (while acting in conjunction with the engine and transmission) the speed of the vehicle. After the EEPM 132 and / or the environmental monitoring and prediction component 126 determine that the cyclist has crossed the intersection, the AVCS 140 may output commands to the powertrain, brakes, and steering 150 to restore the vehicle's previous speed settings.
[0037] The output of the EEPM 132 can be used to track the detected object. In some embodiments, the tracking can be reactive and can include a history of the posture (position and orientation) and velocity of the tracked object. In some embodiments, the tracking can be active and can include a prediction of the future posture and velocity of the tracked object. In some embodiments, the BPM 134 can generate future predictions, for example, based at least in part on the output of the EEPM 132. In some embodiments, detection tracking or instance segmentation can be used instead of building an explicit tracker. For example, for each object, the interface of the BPM 134 can include a history of the recent object position, range, heading, and velocity. In some embodiments, flow information can be defined with reference to units (voxels) of three-dimensional space. For additional prediction accuracy, the flow information associated with a single voxel can include not only velocity, but also kinematic properties such as curvature, yaw rate, etc. Based on this data, the BPM 134 can predict future trajectories in a manner superior to more traditional tracking methods. In some embodiments, an alternative method of deploying a recurrent neural network (RNN) to smooth and interpolate position and velocity over time can be used, which can be performed similar to the operation of a Kalman filter.
[0038] The output of EEPM 132 can be used for vehicle localization. In some embodiments, BPM 134 can use a lidar-based global map that maps the entire area of the 3D environment around the vehicle. In some embodiments, BPM 134 can deploy a simpler system that uses accelerometers, odometers, GNNS data, and camera-based lane mapping to identify the vehicle's current position relative to the map data.
[0039] In different embodiments, the BPM 134 may have different levels of sophistication depending on the driving environment 101 (e.g., highway driving, city driving, suburban driving, etc.). In L2 driver assistance embodiments ("hands on the wheel"), where the driver is expected to take over control of the vehicle at any time, the BPM 134 may have minimal functionality and be able to predict the behavior of other road users within a short time frame (e.g., a few seconds). For example, such predictions may include upcoming lane changes by other vehicles ("agents"). The BPM 134 may use various cues, such as turn signals, front wheel turns, the driver turning his head in the direction of the turn, etc. The BPM 134 may determine whether such an upcoming lane change requires the driver's attention. In the event that the lane change agent is far enough away from the vehicle, the AVCS 140 acting on the BPM 134 prediction may change the trajectory of the vehicle (e.g., slow the vehicle) without driver involvement. In the event that the change requires the driver's immediate attention, the BPM 134 may output a signal to the driver indicating that the driver should take over control of the vehicle.
[0040] In L3 driver assistance implementations ("hands off the wheel"), the goal may be to provide autonomous driving functionality within at least a certain time frame (e.g., X seconds) so that if a condition arises that requires driver control, the condition will be predicted at least X seconds before it occurs. The map data may also include camera and / or radar images that highlight landmarks (bridges, signs, roadside structures, etc.). In some implementations, the BPM 134 of the L3 system may output two trajectories, option A and backup option B, for X seconds at any given time. For example, when driving on a city street in the rightmost lane of the street, the BPM 134 may calculate option A for the vehicle to remain in the rightmost lane, and may further calculate option B for the vehicle to move to the left lane if a parked vehicle turns into the leftmost lane. The BPM 134 may predict that the left lane will remain available and continue vehicle operation within the next X seconds. At some point, the BPM 134 may predict that the left lane has a fast-moving agent that will move close enough to the vehicle to make the left lane (and therefore option B) unavailable to the vehicle. After determining that Option B may become unavailable, the BPM 134 may call the driver to take control of the vehicle. In even more complex systems, where driver input is not expected (e.g., autonomous L4 driving systems), if Option B disappears, the AVCS 140 may stop the vehicle on the side of the road until driving conditions change favorably.
[0041] In order to achieve reliable prediction, BPM 134 can simulate multiple possible scenarios of how different road users can behave in different ways, and estimate the probabilities and corresponding results of various such scenarios. In some embodiments, BPM 134 can use a closed-loop approach and determine the following probability distribution: if the vehicle makes a certain driving path change (or maintains the current driving path), other vehicles will respond in some way (e.g., yield to the vehicle or accelerate or otherwise block the driving path of the vehicle). BPM 134 can evaluate multiple such scenarios and output the probability of each scenario or at least some scenarios. In some embodiments, BPM 134 can use an open-loop approach, in which predictions are made based on the current motion state of the agent, and changes in vehicle motion do not affect the behavior of other agents. In some embodiments, the predicted positions of various agents can be represented via future occupancy heat maps. Further details about EEPM 132 will now be described below with reference to Figures 2A-2B.
[0042] 2A is a diagram illustrating an example network architecture of an end-to-end perception model (EEPM) 132 that may be deployed as part of a perception system of a vehicle according to some embodiments of the present disclosure. Input data 201 may include data obtained by various components (e.g., (one or more) lidars 112, (one or more) radars 114, (one or more) optical (e.g., visible) ranging cameras 118, (one or more) IR sensors 119) of a sensing system 110 (as shown in FIG. 1). For example, as shown, input data 201 may include camera data 210 and radar data 220. Although not shown, input data 201 may also include, for example, lidar data.
[0043] The input data 201 may include images and / or any other data, such as voxel intensities, velocity data associated with the voxels, and metadata such as timestamps. The input data 201 may include, for example, direction data (e.g., angular coordinates of a return point), distance data, and radial velocity data, as may be obtained by (one or more) lidars 112 and / or (one or more) radars 114. In addition, the input data 201 may also include road map data stored (or accessible) by the perception system 130, for example, as part of the map information 124. The road map data may include any two-dimensional map, three-dimensional map of the road and its surroundings (including any suitable mapping of stationary objects, such as identification of bounding boxes of these objects). It should be understood that this list of input data 201 is not exhaustive, and any suitable additional data may be used as part of the input data 201, such as IMU data, GNNS data, etc. Each modality of the input data 201 may be associated with a specific moment when the data is acquired. A set of available data associated with a specific moment (e.g., a lidar image, a radar image, a camera image, and / or an IR camera image, etc.) may be referred to as a sensing frame. In some embodiments, images acquired by different sensors may be synchronized so that all images in a given sensing frame have the same (to synchronization accuracy) timestamp. In some embodiments, some images in a given sensing frame may have a (controlled) time offset.
[0044] An image acquired by any sensor may include a corresponding intensity map ),in Can be any set of coordinates, including three-dimensional (spherical, cylindrical, Cartesian, etc.) coordinates (e.g., in the case of lidar and / or radar images) or two-dimensional coordinates (in the case of camera data). The coordinates of various objects (or surfaces of objects) that reflect lidar and / or radar signals can be determined from direction data (e.g., angle θ and azimuth ϕ in the direction of lidar / radar transmission) and distance data (e.g., radial distance R determined based on the ToF of the lidar / radar signal). The intensity map can identify the intensity of the sensed signal detected by the corresponding sensor. Similarly, the lidar and / or radar sensor can produce a Doppler (frequency shift) map , which is based on the reflected radar signal The detected Doppler shift of the frequency to identify the radial velocity of the reflecting object, where λ is the lidar / radar wavelength, with positive values associated with objects moving towards the lidar / radar (and therefore towards the vehicle). and negative values associated with objects moving away from the LiDAR / Radar In some embodiments, for example, in a driving environment where objects move substantially within a particular plane (e.g., the ground), two-dimensional coordinates (such as radial distance and azimuth angle) may be used: To define the radar intensity map and the Doppler map.
[0045] The camera feature network 212 is capable of receiving the camera data 210 and extracting a set of camera data features from the camera data 210. For example, the set of camera data features may include a set of camera data feature vectors. More specifically, the camera data features may be two-dimensional (2D) camera data features. The camera data feature network 212 may use any suitable (one or more) perspective view backbones to obtain the set of camera data features. Examples of suitable perspective backbones include Resnet, EfficientNet, etc. In some embodiments, each camera sensor (e.g., front camera, rear camera, etc.) may use the same perspective backbone (e.g., the same shared weights) in training to avoid learning viewpoint-specific priors, thereby avoiding the performance of the EEPM 132 from being affected by vehicle yaw. Each camera data feature may be associated with a specific pixel or pixel cluster. Each pixel (or pixel cluster) may be associated with a corresponding depth distribution and a corresponding depth feature. In some embodiments, the processed camera data may be downsampled for computational efficiency. In some embodiments, a pseudo camera may be used. A pseudo camera represents an image crop from a full-resolution image to provide finer details for remote tasks. The pseudo camera can have a fixed crop or a crop driven from the output of the coarse resolution backbone. In some embodiments, the crop can be trained directly. In some embodiments, differentiable cropping can be used to train the attention mechanism end-to-end.
[0046] The camera data features may be provided to a camera data feature projection component 214. The camera data feature projection component 214 may utilize the camera data feature projection to transform the set of camera data features into a set of pixels. For example, the pixel group may be a pixel point cloud. In some embodiments, utilizing the camera data feature projection includes performing a lifting transformation relative to the 2D camera data (e.g., from a 2D backbone, sensor intrinsics and extrinsics (or derived intrinsics and extrinsics for a pseudo camera)). To this end, the camera data feature transformation component 214 may project the 2D camera data into a three-dimensional (3D) space. Various depth distribution techniques may be used to accomplish the projection. During training, the depth ground truth may be obtained from other sensor data (e.g., lidar data) and may be used as a structured loss. The same (or similar) architecture may be used to process the output of other sensors (e.g., IR cameras) that may provide 2D images. Therefore, the camera data feature projection component 214 may provide a lifted camera "context" for a camera combination across AVs. Further details on generating pixel groups will be described below with reference to FIG. 3.
[0047] More specifically, the lifting transform may combine the depth distribution and the set of camera features (eg, feature vectors). As an illustrative example, the lifting transform may supplement the feature vectors with depth information from the depth distribution. Each pixel described For example, the lifting transform can calculate the same pixel (size Each feature vector and (size The corresponding depth distribution The output of the lifting transform may be a feature that can be represented, for example, by:
[0048] The feature tensor computed for a single pixel can then be used To obtain a combined feature tensor for the entire image, for example, by concatenating feature tensors for different pixels: . Combined feature tensor Has dimensions The combined feature tensor can then undergo a 2D mapping. More specifically, the 2D mapping can produce a projected feature tensor using a convenient set of planar coordinates (e.g., Cartesian coordinates x and y or polar coordinates r and θ in the ground plane).
[0049] 2D mapping can be a two-part transformation. During the first part, the perspective coordinates Can be converted to 3D Cartesian coordinates (or 3D cylindrical coordinates, ),in is the vertical coordinate (in the direction perpendicular to the ground). This transformation It can be a projective transformation, parameterized by the focal length of the camera, the direction of the camera's optical axis, and other similar parameters. In the case of images acquired by multiple cameras (or cameras with rotating optical axes), the transformation Multiple projection transformations may be included, for example, with Individual transformations.
[0050] During the second part, the 2D mapping can transform the combined feature tensor represented by the new coordinates Project onto a horizontal surface to obtain the feature tensor of projection (BEV). For example, to obtain Projected feature tensor , the combined feature tensor can be summed (or averaged) over the elements associated with each vertical column of pixels, e.g., In some embodiments, the coordinates The upper sum can be assigned to different coordinates The different weights implement: For example, the larger weight A feature tensor may be assigned to pixels that image objects within a certain height from the ground (e.g., up to a few meters) and a lower weight may be assigned to pixels that image objects within other heights from the ground (e.g., to eliminate spurious objects that do not hinder vehicle motion, such as tree branches, cables, etc.). The projected feature tensor may characterize objects and their locations in the BEV for which perspective distortion has been reduced (e.g., eliminated).
[0051] The radar data feature network 222 can receive the radar data 220 and extract a set of radar data features from the radar data 220. For example, radar data features can be generated for each radar. The radar data feature network 222 can use any suitable radar backbone. Examples of suitable radar backbones include PointPillar, Range SparseNet, etc. Each radar modality (e.g., intensity, second echo, Doppler shift, radar cross section) can have a different radar backbone and feature generation layer. In some embodiments, a complete cycle (spin) of the lidar / radar sensor can be used to obtain radar data features. In some embodiments, parts of the radar cycle can be used to obtain radar data features. Processing of parts of these time periods can allow the EEPM 132 to react faster to new agents (e.g., vehicles, pedestrians, etc.) or sudden movements of existing agents in some cases, and operate at the rate of the fastest sensor.
[0052] The set of radar data features may be provided to a radar data feature transformation component 224. The radar data feature transformation component 214 may utilize a radar data feature transformation to transform the set of radar data features into a set of radar points. For example, the set of radar points may be a radar point cloud. Further details on generating the set of radar points will be described below with reference to FIG. 3.
[0053] The set of pixel points generated by the camera data feature projection component 214 and the set of radar points generated by the radar data feature transformation component 224 can be provided to the BEV mesh processing component 230. The BEV mesh processing component 230 can combine the set of pixel points and the set of radar points to generate a set of BEV meshes. It is possible that the set of radar data features has a coordinate representation that is not computationally efficient for composition into a BEV mesh. Therefore, in some embodiments, performing a radar data feature transformation may include transforming the coordinate representation of the set of radar data features into a suitable coordinate representation for composition into a BEV mesh. For example, a computationally efficient representation may be a Cartesian coordinate representation. Illustratively, the radar data feature network 222 may process the radar data 220 in a polar coordinate representation, and transforming the coordinate representation includes transforming from a polar coordinate representation to a Cartesian coordinate representation.
[0054] Generating the set of BEV grids using the set of pixel points and the set of radar points may include voxelizing the set of pixel points and the set of radar points to generate one or more BEV grids. In some embodiments, the set of BEV grids includes a plurality of BEV grids.
[0055] In some embodiments, the set of BEV grids defines a multi-scale BEV space, where each grid in the set of BEV grids is defined by a corresponding scale (e.g., resolution and / or size). The multi-scale BEV space is a shared feature space that can accumulate various available feature vector modalities. In some cases, a particular set of feature vectors (e.g., lidar features or road map features) may not be available temporarily or by design. In this case, there may not be a corresponding contribution to the multi-scale BEV space that relies on other available features (e.g., camera and / or radar data features) processed by the EEPM 132. The set of BEV grids defining the multi-scale BEV space can be cyclic, for example, in time The features obtained at some scale can be warped (using a differentiable warp such as a spatial transformer) and multiplied in time For example, using smoothed pose increments (i.e., time and time The posture change between The new features are aggregated into a new grid. The multi-scale BEV space can be in a smooth pose consistent frame. The multi-scale BEV space can be spatially consistent over a time period used for aggregation in detection. In some embodiments, a process for clearing distant portions of a grid and transferring values as the AV moves in the world. Various priors in a global frame (e.g., elevation tiles, road maps) can undergo an accurate global to smooth transformation. Dynamic objects can be represented using flow fields in conjunction with occupancy maps to perform additional recurrent aggregation. The multi-scale BEV space can be four-dimensional, having three spatial dimensions (e.g., 3D voxel space) and one time dimension. Each element of the multi-scale BEV space can include a voxel, a time associated with the voxel, and a combined feature vector obtained by combining (e.g., cascading) feature vectors output by various feature networks.
[0056] For example, the set of pixels and the set of radar points may be voxelized to generate a first BEV grid defined by a first scale, and the set of pixels and the set of radar points may be voxelized to generate a second BEV grid defined by a second scale different from the first scale. The first BEV grid may be a coarse BEV grid with a higher resolution (e.g., a smaller voxel size) that may be used to detect objects closer to the AV, and the second BEV grid may be a fine BEV grid with a lower resolution (e.g., a larger voxel size) that may be used to detect objects farther away from the AV. The size and / or resolution of the BEV grids in the set of BEV grids may depend on the available computing facilities and the particular driving task, e.g., highway driving may involve a grid with larger pixels (compared to the case of city driving) but extending to a longer distance proportional to the typical speeds involved. For example, even if a fine BEV grid is more accurate than a coarse BEV grid, a coarse BEV grid may be used to reduce computational costs.
[0057] The BEV mesh processing component 230 may also extract a corresponding set of BEV mesh features for each BEV mesh in the set of BEV meshes. For example, the BEV mesh processing component 230 may implement a set of BEV mesh feature networks. Each BEV mesh feature network in the set of BEV mesh feature networks may extract a corresponding set of BEV mesh features from a corresponding BEV mesh in the set of BEV meshes. Each BEV mesh feature network may include any suitable number of layers (e.g., layers implementing 3D convolutions in a ResNet-type architecture) for processing its corresponding BEV mesh to extract a corresponding set of BEV mesh features. For example, if the set of BEV meshes includes a first BEV mesh and a second BEV mesh, the set of BEV mesh feature networks may include a first BEV mesh feature network for extracting a first set of BEV mesh features from the first BEV mesh, and a second BEV mesh feature network for extracting a second set of BEV mesh features from the second BEV mesh. The BEV mesh processing component 230 may then resample each set of BEV mesh features to its corresponding BEV mesh to generate a resampled BEV mesh. The BEV mesh processing component 230 may then fuse each BEV mesh together to generate a fused BEV mesh. Further details regarding the set of BEV meshes and the BEV mesh processing component 230 will be described below with reference to FIG.
[0058] The output of the BEV grid processing component 230 (e.g., the fused BEV grid) can then be provided to a set of classifier heads ("heads") 240. The set of heads 240 can include one or more heads, each generating a corresponding output. The output generated by at least one of the heads in the set of heads 240 can be provided to an object detector 250. The object detector 250 can include one or more components to generate object detection predictions based on the (one or more) outputs generated by the set of heads 240. If the set of BEV grids defines a multi-scale BEV space, the scale of each BEV grid in the set of BEV grids can be processed in different ways depending on the specific implementation. One approach can include cutting out the voxel volume used by the finer scale from the coarser scale so that one scale is used for various classification tasks. This approach can deploy special processing of voxels located near the boundaries between different scales. Another option is to have each scale detected separately and then perform non-maximum suppression (NMS) on multiple scales. For example, one or more heads in the set of heads 240 can be allowed to access multiple sizes when available. Yet another option may include implementing sparsity in the feature layer, thresholding the magnitude, and performing sparse aggregation into a global voxel grid. Subsequent tasks may then use the sparse grid for inference. Further details about the group head 240 and object detector 250 will be described below with reference to FIG. 2B.
[0059] EEPM 132 may include one or more additional feature networks (not shown). For example, EEPM 132 may include a road map feature network that can process road map data and output road map features, which may include lanes and lane markings, road edges and medians, traffic lights and stop signs, pedestrian crossings and speed bumps, garage access, parking lots and curb restrictions, railroad crossings, school areas, and traffic inaccessible areas. Road map features may be voxelized into a coordinate frame. Road map data may also include an elevation map. Such previous data may be considered a separate modality. This framework may make it easier to incorporate new location-based data, such as heat maps of object appearances observed in previous runs. Road map data may be accumulated during previous driving missions for a particular route. In some cases, where previous data is unavailable, road map data may be limited by available map information 124 for a particular route. As with other modalities, road map data may be lost, and during training, EEPR 132 may be forced to learn to incorporate road map data in an additional manner rather than relying on such data.
[0060] 2B shows the architecture of the EEPM 132 including the set of headers 240 and the object detector 250 according to some embodiments of the present disclosure. As shown, the set of headers 240 may include a detection header 242 and a set of additional headers 244.
[0061] The detection head 242 can be used to perform object detection. More specifically, the detection head 242 can classify boxes of voxels, with an emphasis on detecting objects. Examples of objects include agents (e.g., other vehicles), pedestrians, etc. In some embodiments, the detection head 242 can further perform instance aggregation. Various methods of aggregating instances in space and time can be used so that a single detection or instance is a group of voxels. In some embodiments, a detection box approach may be used. More specifically, similar to the point-lens architecture, the detection head 242 may densely generate parameters for each box and then perform non-maximum suppression (NMS) or weighted aggregation to generate discrete detections. Each voxel may be predicted with a probability of presence, a center offset, and a , box range and heading (which can be ). While the detection box approach may not naturally allow the network to produce a convex hull, a Star-Poly type approach can be used as an extension to achieve this. In some embodiments, the detection head 242 can further perform instance segmentation. In this approach, the network outputs a dense per-instance occupancy. This approach can allow for a convex hull or even more general representation of object boundaries, which may be advantageous for articulated vehicles. For example, the segmentation method may include the following operations: (i) generating a "centrality" output trained using a Gaussian close to the center of mass of each object; (ii) generating a convex hull for the object boundaries; and (iii) generating a convex hull for the object boundaries. (iii) generating a flow of object centers for each voxel within; (iii) finding peaks in the centerness output using NMS; and (iv) associating a voxel with each center using the center offset output masked using the occupancy map. Additional attributes can be aggregated using additional semantic headers and voxel associations. In some embodiments, a signed distance field can be used. In this approach, the network can be trained to output a signed distance field representation. The network can then find zero crossings of the field plus containment to identify individual object instances (e.g., using a union-find algorithm). In some embodiments, one or more of the described methods can be combined.
[0062] Examples of heads in the set of additional heads 244 include a flow head, a segmentation head, an occupancy head, a semantic head, an occlusion head, a road map head, etc. The flow head may output any suitable representation of the flow (e.g., using motion vectors, etc.) corresponding to various voxels of space (e.g., the motion of an object).
[0063] The occupancy head can determine whether a voxel is occupied by an object. More specifically, the occupancy map gives the probability that a voxel location is inside an obstacle, similar to the presence probability, for example. The probability map can be used as a prior data product to perform instance segmentation and other semantic inference tasks within the network.
[0064] The semantic head can be used to output the class of the object. For example, the semantic head can generate intent / semantic signals including but not limited to attributes such as human pose, cyclist hand gestures, etc. Depending on the quality bar of the semantic task, various approaches for semantic tasks can be used. In one approach, a dense voxelized semantic layer can be deployed that uses instance masks to find and aggregate semantic signals. In another approach, a recurrent neural network can be deployed that uses instance location and range to crop relevant features using region of interest (ROI) pooling from various sensor backbones. This second approach can be advantageous for quality critical tasks.
[0065] The occlusion head can output occlusion data related to occluded objects within the environment. For example, the occlusion data can include the probability that the occluded object exists at a location, a set of properties of the occluded object (which can be conditioned on the occluded object being at the location), the probability that the occluded object will be perceived given the presence of the object, etc. In some embodiments, the loss can be weighted using a visibility probability mask to prevent the network from pre-guessing properties about objects that the network should not be able to see.
[0066] The road map head can output a reconstructed road map near the vehicle based on a set of parameters. For example, the set of parameters may include voxel occupancy, motion flow of detected and classified objects, available map data, etc. The reconstructed road map may be in a vectorized format (e.g., lanes represented as polylines) or a heat map format. In some embodiments, the reconstructed road map includes an association of various driving lanes with detected lights indicating whether traffic is allowed to move in a particular lane. For example, a set of lights at an intersection may indicate that the rightmost lane currently has a green light that allows a right turn, the two middle lines have red lights that prohibit advancing through the intersection in the forward direction, and the leftmost lane has a flashing yellow arrow indicating that a left turn is allowed in the absence of oncoming traffic. The reconstructed road map can be used to determine that one side of the street where the vehicle is located has four lanes, and it can be further determined that the set of lights has four lights. The reconstructed road map may include an identification of the current state of each of the set of street lights and an association of each street light with a corresponding driving lane. In addition, the reconstructed road map may include the location of the stop line at the intersection.
[0067] Some of the headers in the set of headers 240 may be independent of the additional headers in the set of headers 240, while some of the headers in the set of headers 240 may be interdependent with the additional headers in the set of headers. For example, the detection header may be interdependent with at least the occupancy header, and the occupancy header may be interdependent with the semantic header.
[0068] In some embodiments, it is not necessary to determine the type of object, and identifying the occupancy grid (occupied and unoccupied voxels) may be sufficient, and the flow (movement of voxels) may be sufficient. For example, in a highway driving use case, identification of the type of object may not be as important as the fact that a certain object occupies a specific spatial region (because all or most objects on a highway are vehicles). In a city driving use case, identification of the type of object may be more important because there may be a greater variety of road users (e.g., pedestrians, electric cars, cyclists, dogs, etc.), each with a specific type of motion behavior (e.g., pedestrians may move across the road).
[0069] As further shown, the object detector 250 may include a set of prediction components. Each prediction component in the set of prediction components can generate a corresponding prediction, thereby forming a set of predictions. For example, the prediction components may include a heat map prediction component 252 and an attribute prediction component 254. The heat map prediction component 252 and the attribute prediction component 254 can each receive the output of at least the detection head 242 and generate a heat map prediction and an attribute prediction, respectively. The heat map prediction value can represent the likelihood of an object appearing in the BEV grid. Examples of attribute predictions include object speed predictions, vehicle lane association predictions (e.g., for telling which lane the vehicle is running in), etc.
[0070] Each prediction in the set of predictions (e.g., heatmap predictions and attribute predictions) may be combined to obtain a combined prediction, and the combined prediction may be provided to the bounding box generator 256 to generate a set of candidate bounding boxes. Each candidate bounding box corresponds to a corresponding bounding box prediction for a corresponding object. More specifically, each candidate bounding box in the set of candidate bounding boxes describes a spatial position prediction of an object detected from the combined prediction. The set of candidate bounding boxes may include a single candidate bounding box for an object, or multiple overlapping candidate bounding boxes for an object.
[0071] The set of bounding box predictions can be provided to the bounding box filter 268 to select the best candidate bounding box from the set of candidate bounding boxes as the bounding box of the object. In some embodiments, the bounding box filter 268 utilizes NMS. Thus, the object detector 250 can determine the most likely spatial location of the object based on a set of BEV grids generated from the camera data and the radar data.
[0072] 3 is a diagram 300 illustrating an example method of generating a fused BEV mesh from a multi-scale BEV mesh according to some embodiments of the present disclosure. Diagram 300 illustrates a set of camera data features 310-1 and a set of radar data features 310-2. As described above with reference to FIG. 2A, the set of camera data features 310-1 may be extracted from camera data, and the set of radar data features 310-2 may be extracted from radar data.
[0073] At step 315-1, the camera data feature projection is used to transform the group of camera data features 310-1 into a group of pixel points 320-1. At step 315-2, the radar data feature transformation is used to transform the group of radar data features 310-2 into a group of radar points 320-2. For example, the pixel point group 320-1 may be a pixel point cloud, and the radar point group 320-2 may be a radar point cloud.
[0074] At step 325, the set of pixel points 320-1 and the set of radar points 320-2 are used to generate a BEV grid 330-1 and a BEV grid 330-2. More specifically, voxelization is performed to generate the BEV grids 330-1 and 330-2. The BEV grid 330-1 has a different size and / or resolution than the BEV grid 330-2. More specifically, the BEV grid 330-1 is a coarser grid, and the BEV grid 330-2 is a finer grid.
[0075] At step 335-1, the BEV mesh 330-1 is provided to a first BEV mesh feature network in a set of BEV mesh feature networks to extract a first set of BEV mesh features, and the first set of BEV mesh features are resampled to the BEV mesh 330-1 to generate a resampled BEV mesh 340-1. At step 335-2, the BEV mesh 330-2 is provided to a second BEV mesh feature network in the set of BEV mesh feature networks to extract a second set of BEV mesh features, and the second set of BEV mesh features are resampled to the BEV mesh 330-2 to generate a resampled BEV mesh 340-2. Each BEV mesh feature network may include any suitable number of layers (e.g., layers implementing 3D convolutions in a ResNet-type architecture) for processing its respective BEV mesh to extract a respective set of BEV mesh features.
[0076] At step 345, the resampled BEV meshes 340-1 and 340-2 are fused together to generate a fused BEV mesh 350. The fused BEV mesh 350 can then be provided to a set of heads (e.g., a set of heads 240 of FIGS. 2A-2B ) for further processing (e.g., object detection using the object detector 250 of FIGS. 2A-2B ). Further details regarding the diagram 300 are described above with reference to FIGS. 2A-2B .
[0077] 4A is a flow chart illustrating an example method for implementing camera-radar data fusion to generate a fused BEV grid for effective object detection according to some embodiments of the present disclosure. At least one processing device operably coupled to a memory may execute method 400 and / or each of its individual functions, routines, subroutines, or operations. For example, one or more processors may be communicatively coupled to one or more memory devices. Examples of processors include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The processing device executing method 400 may execute instructions issued by various components (e.g., EEPM 132) of the sensing system 110 or the data processing system 120 of FIG. 1. In some embodiments, method 400 may involve systems and components of an autonomous driving vehicle (such as autonomous vehicle 100 of FIG. 1). In some implementations, method 400 may be performed by EEPM 132 or any other similar model that may be part of a perception system for an autonomous vehicle, a vehicle deploying driver assistance technology, or any other application platform that uses object detection and classification.
[0078] Method 400 can be used to improve the performance of processing system 120 and / or autonomous vehicle control system 140. In some embodiments, a single processing thread can perform method 400. Alternatively, two or more processing threads can perform method 400, each thread performing one or more separate functions, routines, subroutines or operations of method 400. In an illustrative example, the processing threads implementing method 400 can be synchronized (e.g., using semaphores, critical sections and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 400 can be executed asynchronously relative to each other. The various operations of method 400 can be performed in a different order compared to the order shown in Figure 4A. Some operations of method 400 can be performed simultaneously with other operations. Some operations can be optional.
[0079] At operation 410, processing logic obtains input data derived from a set of sensors. The set of sensors may be associated with an autonomous vehicle (AV). Input data may be obtained within the driving environment of the AV. For example, the input data may include camera data derived from at least one camera of the AV and radar data derived from at least one radar of the AV. For example, the camera data may be based on real-time images obtained by one or more cameras of the AV or by cameras installed on any other suitable application platform. The camera may be an optical range camera and / or an IR camera, including a panoramic (surround view) camera, a partial panoramic camera, a high definition (high resolution) camera, a close-range camera, a camera with a fixed field of view (relative to the vehicle), a camera with a dynamic (adjustable) field of view, a camera with a fixed or adjustable focal length, a camera with a fixed or adjustable numerical aperture, and any other suitable camera. The optical rangefinder camera may also include a night vision camera. The image acquired by the camera may include various metadata that provide a geometric association between image pixels and the spatial position of an object, a correspondence between pixels of different images, and the like.
[0080] At operation 420, processing logic extracts multiple sets of features from the input data. For example, the multiple sets of features may include a set of camera data features extracted from the camera data and a set of radar features extracted from the radar data.
[0081] At operation 430, processing logic generates a fused BEV mesh using the plurality of sets of features. More specifically, the fused BEV mesh may be generated by fusing a plurality of BEV meshes, wherein each BEV mesh in the plurality of BEV meshes is generated from the plurality of sets of feature groups. Each BEV mesh in the plurality of BEV meshes may have a corresponding scale (e.g., size and / or resolution). For example, a first BEV mesh may have a coarser resolution than a second BEV mesh. Further details regarding generating the fused BEV mesh are described above with reference to FIGS. 2A-3 and will now be described below with reference to FIG. 4B.
[0082] FIG. 4B illustrates an example method 430 for generating a fused BEV grid according to some embodiments of the present disclosure. At least one processing device operably coupled to a memory may execute the method 430 and / or each of its individual functions, routines, subroutines, or operations. For example, one or more processors may be communicatively coupled to one or more memory devices. Examples of processors include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The processing device executing the method 430 may execute instructions issued by various components (e.g., EEPM 132) of the sensing system 110 or the data processing system 120 of FIG. 1. In some embodiments, the method 430 may involve systems and components of an autonomous driving vehicle (such as the autonomous vehicle 100 of FIG. 1). In some embodiments, the method 430 may be performed by the EEPM 132 or any other similar model, which may be part of a perception system of an autonomous vehicle, a vehicle deploying driver assistance technology, or part of any other application platform using object detection and classification.
[0083] Method 430 can be used to improve the performance of processing system 120 and / or autonomous vehicle control system 140. In some embodiments, a single processing thread can perform method 430. Alternatively, two or more processing threads can perform method 430, each thread performing one or more separate functions, routines, subroutines, or operations of method 430. In an illustrative example, the processing threads implementing method 430 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 430 can be executed asynchronously relative to each other. The various operations of method 430 can be performed in a different order compared to the order shown in FIG. 4B. Some operations of method 430 can be performed simultaneously with other operations. Some operations can be optional.
[0084] At operation 432, processing logic transforms each of the plurality of sets of features into a corresponding set of points. In some embodiments, each set of points is a point cloud. For example, the set of camera data features may be transformed into a set of pixel points, and the set of radar data features may be transformed into a set of radar points.
[0085] For example, transforming the set of camera data features into a set of pixel points may include utilizing a camera data feature projection. The camera data feature projection may include projecting the 2D camera data features into a 3D space. In some embodiments, utilizing the camera data feature projection includes performing a lifting transformation with respect to the 2D camera data features (e.g., from a 2D backbone, sensor intrinsics and extrinsics (or derived intrinsics and extrinsics for a pseudo camera)).
[0086] At operation 434, processing logic generates a set of BEV grids using each set of points. For example, the set of pixel points and the set of radar points may be used to generate the set of BEV grids. More specifically, generating the set of BEV grids may include performing voxelization using each set of points. The set of BEV grids may include a first BEV grid and a second BEV grid. The first BEV grid may have a first scale (e.g., size and / or resolution), and the second BEV grid may have a second scale different from the first scale. Illustratively, the first BEV grid may be a coarser grid than the second BEV grid.
[0087] It may be the case that the set of radar data features has a coordinate representation that is not suitable for integration into the BEV grid. Therefore, in some embodiments, transforming the set of radar data features may include transforming the coordinate representation of the set of radar data features into a suitable coordinate representation for integration into the BEV grid. Transforming the set of radar features may include transforming from a polar coordinate representation to a Cartesian coordinate representation.
[0088] At operation 436, processing logic extracts a corresponding BEV grid feature set for each BEV grid in the set of BEV grids. For example, each BEV grid in the set of BEV grids may be provided to a corresponding BEV grid feature network to generate a corresponding BEV grid feature set.
[0089] At operation 438, processing logic generates a resampled BEV mesh for each BEV mesh using the corresponding BEV mesh feature set. At operation 439, processing logic fuses each resampled BEV mesh to generate a fused BEV mesh. Further details about operations 432-439 are described above with reference to FIGS. 2A-3.
[0090] Referring back to FIG. 4A , at operation 440 , the processing logic may provide the fused BEV mesh for object detection. For example, the fused BEV mesh may be provided to an object detector. Object detection may be performed to identify at least one object within the driving environment of the AV. Further details regarding operations 410-440 are described above with reference to FIGS. 1-3 . Further details regarding performing object detection will now be described below with reference to FIGS. 5A to 5B .
[0091] FIG. 5A illustrates an example method 500 for implementing a BEV mesh generated using camera-radar data fusion for effective object detection according to some embodiments of the present disclosure. At least one processing device operably coupled to a memory may perform the method 500 and / or each of its individual functions, routines, subroutines, or operations. For example, one or more processors may be communicatively coupled to one or more memory devices. Examples of processors include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and the like. The processing device executing the method 500 may execute instructions issued by various components (e.g., EEPM 132) of the sensing system 110 or the data processing system 120 of FIG. 1. In some embodiments, the method 500 may involve systems and components of an autonomous driving vehicle (such as the autonomous vehicle 100 of FIG. 1). In some implementations, method 500 may be performed by EEPM 132 or any other similar model that may be part of a perception system for an autonomous vehicle, a vehicle deploying driver assistance technology, or any other application platform that uses object detection and classification.
[0092] Method 500 can be used to improve the performance of processing system 120 and / or autonomous vehicle control system 140. In some embodiments, a single processing thread can perform method 500. Alternatively, two or more processing threads can perform method 500, each thread performing one or more separate functions, routines, subroutines or operations of method 500. In an illustrative example, the processing threads implementing method 500 can be synchronized (e.g., using semaphores, critical sections and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 500 can be executed asynchronously relative to each other. The various operations of method 500 can be performed in a different order compared to the order shown in Figure 5A. Some operations of method 500 can be performed simultaneously with other operations. Some operations can be optional.
[0093] At operation 510, processing logic obtains a fused BEV mesh. For example, the fused BEV mesh may be generated by fusing a plurality of BEV meshes, each of which is obtained from a feature group (eg, a camera feature group and a radar feature group) according to the method 400 described above with reference to FIGS. 4A-4B.
[0094] At operation 520, processing logic performs object detection using the fused BEV mesh to identify at least one object within the driving environment associated with the AV. Further details regarding performing object detection are described above with reference to Figures 2A-2B, and will now be described below with reference to Figure 5B.
[0095] FIG. 5B illustrates an example method 520 for performing object detection according to some embodiments of the present disclosure. At least one processing device operably coupled to a memory may perform the method 520 and / or each of its individual functions, routines, subroutines, or operations. For example, one or more processors may be communicatively coupled to one or more memory devices. Examples of processors include a central processing unit (CPU), a graphics processing unit (GPU), a tensor processing unit (TPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), etc. The processing device executing the method 520 may execute instructions issued by various components (e.g., EEPM 132) of the sensing system 110 or the data processing system 120 of FIG. 1. In some embodiments, the method 520 may involve systems and components of an autonomous driving vehicle (such as the autonomous vehicle 100 of FIG. 1). In some embodiments, the method 520 may be performed by the EEPM 132 or any other similar model, which may be part of a perception system of an autonomous vehicle, a vehicle deploying driver assistance technology, or part of any other application platform using object detection and classification.
[0096] Method 520 can be used to improve the performance of processing system 120 and / or autonomous vehicle control system 140. In some embodiments, a single processing thread can perform method 520. Alternatively, two or more processing threads can perform method 520, each thread performing one or more separate functions, routines, subroutines, or operations of method 520. In an illustrative example, the processing threads implementing method 520 can be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing method 520 can be executed asynchronously relative to each other. The various operations of method 520 can be performed in a different order compared to the order shown in FIG. 5B. Some operations of method 520 can be performed simultaneously with other operations. Some operations can be optional.
[0097] At operation 522, processing logic obtains a set of predictions generated using the fused BEV mesh. Each prediction in the set of predictions can be generated based on the output generated by a set of heads. For example, the set of heads can include a detection head and a set of other heads. Examples of heads that can be included in the set of other heads include flow heads, segmentation heads, occupancy heads, semantic heads, occlusion heads, road map heads, etc. In some embodiments, the set of predictions includes heat map predictions and attribute predictions. Heat map predictions and attribute predictions can be generated from the output of at least the detection head.
[0098] At operation 524, processing logic generates a set of candidate bounding boxes from the set of predictions. Generating the set of candidate bounding boxes may include combining each prediction in the set of predictions to obtain a combined prediction, and generating the set of candidate bounding boxes using the combined predictions. Each candidate bounding box corresponds to a respective bounding box prediction for a corresponding object. More specifically, each candidate bounding box in the set of candidate bounding boxes describes a spatial position prediction of an object detected from the combined prediction. The set of candidate bounding boxes may include a single candidate bounding box for an object, or multiple overlapping candidate bounding boxes for an object.
[0099] At operation 526, processing logic selects a bounding box from the set of candidate bounding boxes. For example, the bounding box may be an optimal bounding box for the object. More specifically, the bounding box may correspond to the most likely spatial location of the object. Selecting a bounding box from the set of candidate bounding boxes may include applying a filter to the set of candidate bounding boxes. In some embodiments, applying a filter to the set of candidate bounding boxes includes applying NMS. Further details about operations 522-526 are described above with reference to FIG. 2B.
[0100] Referring back to Figure 5A, at operation 530, processing logic may cause the driving path of the AV to be modified in view of the at least one object. Further details regarding operations 510-530 are described above with reference to Figures 1-3.
[0101] FIG6 depicts a block diagram of an example computer device 600 capable of implementing camera-radar data fusion to generate a fused BEV grid for effective object detection according to some embodiments of the present disclosure. The example computer device 600 can be connected to other computer devices in a LAN, an intranet, an extranet, and / or the Internet. The computer device 600 can operate in the capacity of a server in a client-server network environment. The computer device 600 can be a personal computer (PC), a set-top box (STB), a server, a network router, a switch or a bridge, or any device capable of executing a set of instructions (sequentially or otherwise) specifying the actions to be taken by the device. In addition, although only a single example computer device is shown, the term "computer" should also be deemed to include any collection of computers that execute a set (or multiple sets) of instructions individually or jointly to perform any one or more of the methods discussed herein.
[0102] The example computer device 600 may include a processing device 602 (also referred to as a processor or CPU), a main memory 604 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 606 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 618), which may communicate with each other via a bus 630.
[0103] The processing device 602 (which may include the processing logic 603) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device 602 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processing device 602 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. According to one or more aspects of the present disclosure, the processing device 602 may be configured to execute instructions for executing the method 400 and / or the method 430 of Figures 4A-4B and / or the method 500 and / or the method 520 of Figures 5A-5B.
[0104] The example computer device 600 may also include a network interface device 608, which may be communicatively coupled to a network 620. The example computer device 600 may also include a video display 610 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 612 (e.g., a keyboard), a cursor control device 614 (e.g., a mouse), and an acoustic signal generating device 616 (e.g., a speaker).
[0105] The data storage device 618 may include a computer-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 628 having stored thereon one or more sets of executable instructions 622. According to one or more aspects of the present disclosure, the executable instructions 622 may include executable instructions to perform the method 400 and / or the method 430 of Figures 4A-4B and / or the method 500 and / or the method 520 of Figures 5A-5B.
[0106] The executable instructions 622 may also reside completely or at least partially within the main memory 604 and / or within the processing device 602 during execution thereof by the example computer device 600, the main memory 604 and the processing device 602 also constituting computer-readable storage media. The executable instructions 622 may also be sent or received over a network via the network interface device 608.
[0107] Although the computer-readable storage medium 628 is shown as a single medium in FIG6 , the term "computer-readable storage medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store one or more sets of operating instructions. The term "computer-readable storage medium" should also be taken to include any medium that is capable of storing or encoding a set of instructions for execution by a machine, which set of instructions causes the machine to perform any one or more of the methods described herein. Thus, the term "computer-readable storage medium" should be taken to include, but is not limited to, solid-state memories and optical and magnetic media.
[0108] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means by which those skilled in the art of data processing most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps leading to a desired result. These steps are steps that require physical manipulation of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals that can be stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient, primarily for general reasons, to sometimes refer to these signals as bits, values, elements, symbols, characters, terms, numbers, etc.
[0109] It should be remembered, however, that all of these and similar terms are associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically noted otherwise, as will be apparent from the following discussion, it should be understood that throughout the specification, discussions utilizing terms such as "obtain," "generate," "provide," "cause," "transform," "merge," "select," "execute," etc. refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.
[0110] Examples of the present disclosure also relate to an apparatus for performing the methods described herein. The apparatus may be specially constructed for the desired purpose, or it may be a general-purpose computer system selectively programmed by a computer program stored in a computer system. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk (including optical disks, CD-ROMs, and magneto-optical disks), read-only memory (ROM), random access memory (RAM), EPROM, EEPROM, magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.
[0111] The methods and displays presented herein are not inherently related to any particular computer or other device. Various general purpose systems can be used together with the programs taught herein, or it may prove convenient to construct more specialized devices to perform the required method steps. The required structures of various these systems will appear as described in the following description. In addition, the scope of the present disclosure is not limited to any particular programming language. It should be understood that various programming languages can be used to implement the teachings of the present disclosure.
[0112] It should be understood that the above description is intended to be illustrative rather than restrictive. After reading and understanding the above description, many other implementation examples will be apparent to those skilled in the art. Although the present disclosure describes specific examples, it will be appreciated that the systems and methods of the present disclosure are not limited to the examples described herein, but can be modified to practice within the scope of the appended claims. Therefore, the description and the drawings should be considered illustrative rather than restrictive. Therefore, the scope of the present disclosure should be determined with reference to the appended claims and the full scope of equivalents given by these claims.
Claims
1. A method comprising: obtaining, by a processing device, input data derived from a set of sensors associated with an autonomous vehicle (AV); extracting, by the processing device, a plurality of sets of features from the input data; generating, by the processing device, a fused bird's eye view (BEV) mesh using the plurality of sets of features, wherein the fused BEV mesh is generated based on a first BEV mesh having a first scale and a second BEV mesh having a second scale different from the first scale; as well as The fused BEV mesh is provided by the processing device for object detection.
2. The method according to claim 1, wherein: The set of sensors includes at least one camera and at least one radar; The input data includes a set of camera data obtained from the at least one camera and a set of radar data obtained from the at least one radar; as well as The plurality of sets of features include a set of camera data features generated from the set of camera data and a set of radar data features generated from the set of radar data.
3. The method according to claim 1, wherein: Generating the fused BEV grid further includes: associating each set of features in the plurality of sets of features with a corresponding set of points; generating a set of BEV grids using each set of points, the set of BEV grids including the first BEV grid and the second BEV grid; extracting a corresponding set of BEV grid features for each BEV grid in the set of BEV grids; generating a resampled BEV grid for each BEV grid in the set of BEV grids using the corresponding set of BEV grid characteristics, wherein the first BEV grid is associated with a first resampled BEV grid, and wherein the second BEV grid is associated with a second resampled BEV grid; and Each resampled BEV grid is fused to generate the fused BEV grid.
4. The method according to claim 3, wherein: Associating each set of features in the plurality of sets of features with a corresponding set of points further comprises: Transforming a set of camera features from the plurality of sets of features into a set of pixel points; and A set of radar features in the plurality of sets of features is transformed into a set of radar points, including transforming from a polar coordinate representation to a Cartesian coordinate representation. 5 . The method of claim 1 , further comprising performing, by the processing device, the object detection using the fused BEV mesh to identify at least one object using a set of neural networks.
6. The method of claim 5, wherein performing object detection further comprises: obtaining a set of predictions generated using the fused BEV grid, wherein the set of predictions includes a heat map prediction and an attribute prediction; generating a set of candidate bounding boxes from the set of predictions, each candidate bounding box in the set of candidate bounding boxes corresponding to the at least one object; as well as At least one bounding box corresponding to the at least one object is selected from the set of candidate bounding boxes.
7. The method according to claim 5, further comprising: The processing device causes a driving path of the AV to be modified in view of the at least one object.
8. A system comprising: Memory; as well as a processing device communicatively coupled to the memory, the processing device being configured to: obtaining input data derived from a set of sensors associated with an autonomous vehicle (AV); extracting a plurality of sets of features from the input data; generating a fused bird's eye view (BEV) mesh using the plurality of sets of features, wherein the fused BEV mesh is generated based on a first BEV mesh having a first scale and a second BEV mesh having a second scale different from the first scale; as well as The fused BEV mesh is provided for object detection.
9. The system of claim 8, wherein: The set of sensors includes at least one camera and at least one radar; The input data includes a set of camera data obtained from the at least one camera and a set of radar data obtained from the at least one radar; as well as The plurality of sets of features include a set of camera data features generated from the set of camera data and a set of radar data features generated from the set of radar data.
10. The system according to claim 8, wherein: To generate the fused BEV grid, the processing device is further configured to: associating each set of features in the plurality of sets of features with a corresponding set of points; generating a set of BEV grids using each set of points, the set of BEV grids including the first BEV grid and the second BEV grid; extracting a corresponding set of BEV grid features for each BEV grid in the set of BEV grids; generating, for each BEV grid in the set of BEV grids, a resampled BEV grid using the corresponding set of BEV grid characteristics, wherein the first BEV grid is associated with a first resampled BEV grid, and wherein the second BEV grid is associated with a second resampled BEV grid; and Each resampled BEV grid is fused to generate the fused BEV grid.
11. The system according to claim 10, wherein: In order to associate each set of features in the plurality of sets of features with a corresponding set of points, the processing device is further configured to: Transforming a set of camera features from the plurality of sets of features into a set of pixel points; and A set of radar features in the plurality of sets of features is transformed into a set of radar points by transforming from a polar coordinate representation to a Cartesian coordinate representation.
12. The system according to claim 8, wherein: The processing device is further configured to perform the object detection using the fused BEV mesh to identify at least one object using a set of neural networks.
13. The system of claim 12, wherein to perform object detection, the processing device is further configured to: obtaining a set of predictions generated using the fused BEV grid, wherein the set of predictions includes a heat map prediction and an attribute prediction; generating a set of candidate bounding boxes from the set of predictions, each candidate bounding box in the set of candidate bounding boxes corresponding to the at least one object; and At least one bounding box corresponding to the at least one object is selected from the set of candidate bounding boxes.
14. The system according to claim 12, wherein: The processing device is also configured to cause a travel path of the AV to be modified in view of the at least one object.
15. A non-transitory computer-readable storage medium having stored thereon instructions which, when executed by a processing device, cause the processing device to perform operations comprising: obtaining input data derived from a set of sensors associated with an autonomous vehicle (AV), wherein the set of sensors includes at least one camera and at least one radar, and wherein the input data includes a set of camera data obtained from the at least one camera and a set of radar data obtained from the at least one radar; extracting a plurality of sets of features from the input data, wherein the plurality of sets of features include a set of camera data features generated from the set of camera data and a set of radar data features generated from the set of radar data; generating a fused bird's eye view (BEV) mesh using the plurality of sets of features, wherein the fused BEV mesh is generated based on a first BEV mesh having a first scale and a second BEV mesh having a second scale different from the first scale; and The fused BEV mesh is provided for object detection.
16. The non-transitory computer-readable storage medium of claim 15, wherein: Generating the fused BEV grid further includes: associating each set of features in the plurality of sets of features with a corresponding set of points; generating a set of BEV grids using each set of points, the set of BEV grids including the first BEV grid and the second BEV grid; extracting a corresponding set of BEV grid features for each BEV grid in the set of BEV grids; generating a resampled BEV grid for each BEV grid in the set of BEV grids using the corresponding set of BEV grid characteristics, wherein the first BEV grid is associated with a first resampled BEV grid, and wherein the second BEV grid is associated with a second resampled BEV grid; and Each resampled BEV grid is fused to generate the fused BEV grid.
17. The non-transitory computer-readable storage medium of claim 16, wherein associating each set of features in the plurality of sets of features with a corresponding set of points further comprises: Transforming the set of camera features into a set of pixel points; as well as The set of radar signatures is transformed into a set of radar points, including transforming from a polar coordinate representation to a Cartesian coordinate representation.
18. The non-transitory computer-readable storage medium of claim 16, wherein the operations further comprise performing the object detection using the fused BEV mesh to identify at least one object using a set of neural networks.
19. The non-transitory computer-readable storage medium of claim 18, wherein performing object detection further comprises: obtaining a set of predictions generated using the fused BEV grid, wherein the set of predictions includes a heat map prediction and an attribute prediction; generating a set of candidate bounding boxes from the set of predictions, each candidate bounding box in the set of candidate bounding boxes corresponding to the at least one object; as well as At least one bounding box corresponding to the at least one object is selected from the set of candidate bounding boxes.
20. The non-transitory computer-readable storage medium of claim 18, wherein: The operations also include causing a travel path of the AV to be modified in view of the at least one object.
Citation Information
Cited By
Vehicle control method and device, electronic equipment and storage medium
CN121469550A
Methods of generating a map of anomalies in agricultural fields, and related agricultural vehicles
US20260068802A1
