Object identification in bird's-eye view reference frame with explicit depth estimation co-training

Abstract

The described aspects and embodiments deploy a bird's-eye view representation to enable efficient object detection and classification using a machine learning model trained using depth ground truth data. In one embodiment, a system and technique are disclosed that includes acquiring an image and generating a feature vector (FV) and depth distribution pixels of the image using a first neural network (NN), where the first NN is trained using training images and depth ground truth data for the training images. The technique further includes acquiring a feature tensor (FT) taking into account the FV and depth distribution, and processing the acquired FT using a second NN to identify one or more objects depicted in the image.

Description

[Technical Field]

[0001] This specification relates generally to systems and applications for detecting and classifying objects, and particularly to autonomous vehicles and vehicles incorporating driver assistance technologies. More specifically, this specification relates to processing perspective camera images using machine learning techniques for faster and more resource-efficient detection and classification of objects, including, but not limited to, vehicles, pedestrians, cyclists, animals, etc. [Background technology]

[0002] Autonomous (fully or partially autonomous) vehicles (AVs) operate by sensing the external environment with a variety of electromagnetic (e.g., radar and optical) and non-electromagnetic (e.g., sound and humidity) sensors. Some autonomous vehicles chart a driving path through the environment based on the sensed data. The driving path may be determined based on Global Navigation Satellite System (GNSS) data and roadmap data. The GNSS and roadmap data may provide information about static aspects of the environment (such as buildings, street layouts, road closures, etc.), while dynamic information (such as information about other vehicles, pedestrians, street lights, etc.) is obtained from simultaneously collected sensory data. The accuracy and safety of the driving path, as well as the accuracy and safety of the speed regime selected by the autonomous vehicle, depend on the timely and accurate identification of various objects present in the driving environment and the ability of the driving algorithms to process information about the environment and provide correct instructions to the vehicle controls and drivetrain. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure is presented by way of example, and not by way of limitation, and may be more fully understood by reference to the following detailed description when considered in conjunction with the figures in which: [Brief explanation of the drawings]

[0004] [Figure 1]FIG. 1 illustrates components of an example autonomous vehicle (AV) using a bird's-eye view and deploying a model trained using depth ground truth data for efficient object detection and classification, in accordance with some embodiments of the present disclosure. [Figure 2] FIG. 2 illustrates an example architecture of a portion of a perception system capable of efficient object detection and classification, according to some embodiments of the present disclosure. [Figure 3A] FIG. 3A is a schematic diagram illustrating an example operation of a model trained using depth ground truth data for efficient object detection and classification using a bird's-eye view, according to some embodiments of the present disclosure. [Figure 3B] FIG. 3B is a schematic diagram of one embodiment of a model that uses a bird's-eye view and is trained using depth ground truth data for efficient object detection and classification. [Figure 3C] FIG. 3C illustrates a model in which a depth estimation network is pre-trained using depth ground truth data, according to some embodiments of the present disclosure. [Figure 4A] FIG. 4A is a schematic diagram illustrating a distillation framework for training a bird's-eye view model using depth ground truth data, according to some embodiments of the present disclosure. [Figure 4B] FIG. 4B is a schematic diagram illustrating another embodiment of a distillation framework for training a bird's-eye view model using depth ground truth data, according to some embodiments of the present disclosure. [Figure 5] FIG. 5 is a schematic diagram illustrating the operation of a bird's-eye view model to provide instance and semantic segmentation, according to some embodiments of the present disclosure. [Figure 6] FIG. 6 is a schematic diagram illustrating the operation of a bird's-eye view model using temporal aggregation, according to some embodiments of the present disclosure. [Figure 7]FIG. 7 illustrates an example method for developing a model that uses a bird's-eye view representation and is trained using depth ground truth data for efficient object detection and classification, according to some embodiments of the present disclosure. [Figure 8] FIG. 8 illustrates an example method for using depth ground truth data to train a model that develops bird's-eye view representations for efficient object detection and classification, according to some embodiments of the present disclosure. [Figure 9] FIG. 9 illustrates a block diagram of an example computing device capable of operating and / or training a model that uses a bird's-eye view and is trained using depth ground truth data for efficient object detection and classification, according to some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0005] <Summary> In one embodiment, a method is disclosed that includes acquiring one or more perspective camera images of an environment and, for each pixel in a set of pixels in the one or more perspective camera images, generating, using a first neural network (NN), a feature vector (FV) and a depth distribution of a portion of the environment imaged by the corresponding pixel. The first NN is trained using a plurality of training images and depth ground truth data for the plurality of training images. The method further includes, for each pixel in the set of pixels, acquiring a feature tensor (FT) taking into account (i) the FV for the respective pixel and (ii) the depth distribution for the respective pixel. The method further includes processing the acquired FT using a second NN to identify one or more objects in the environment.

[0006] In another embodiment, a method for training a student model is disclosed, the method including obtaining training images and processing the training images using a first neural network of the student model to generate a plurality of FVs and a plurality of depth distributions. Each FV of the plurality of FVs and each depth distribution of the plurality of depth distributions is associated with a respective pixel of the plurality of pixels of the training image. The method further includes obtaining a plurality of ground truth FVs generated by the first neural network of the teaching model, each ground truth FV of the plurality of ground truth FVs being associated with a respective pixel of the plurality of pixels of the training image. The method further includes obtaining a plurality of ground truth depth indicators, each ground truth depth indicator of the plurality of ground truth depth indicators being associated with a respective pixel of at least a subset of the plurality of pixels of the training image. The method further includes adjusting parameters of the first neural network of the student model. The adjustment is based on a comparison of the plurality of FVs to the plurality of ground truth FVs and further based on a comparison of the plurality of depth distributions to the plurality of ground truth depth indicators.

[0007] In another embodiment, a system is disclosed that includes a memory and a processing unit. The processing unit is configured to acquire one or more perspective camera images of an environment and, for each pixel of a set of pixels in the one or more perspective camera images, generate, using a first neural network, a FV and a depth distribution of a portion of the environment imaged by the corresponding pixel. The first neural network is trained using a plurality of training images and depth ground truth data for the plurality of training images. The processing unit is further configured to, for each pixel of the set of pixels, acquire a FT by considering (i) the FV for the respective pixel and (ii) the depth distribution for the respective pixel. The processing unit is further configured to process the acquired FT using a second neural network to identify one or more objects in the environment.

[0008] <Detailed explanation> While various embodiments may be described below, using autonomous driving systems and driver assistance systems as examples for illustration, it should be understood that the techniques and systems described herein may be used to track objects in a wide range of applications, including aerology, marine applications, traffic control, animal control, industrial and academic research, public or personal safety, or in any other application where automatic detection of objects is advantageous.

[0009] In one example, for the safety of automated driving operations, it may be desirable to develop and deploy technology for rapid and accurate detection, classification, and tracking of various road users and other objects encountered on or near roadways, such as roadway obstacles, construction equipment, roadside structures, etc. Autonomous vehicles (as well as various driver assistance systems) can utilize several sensors to facilitate the detection of objects in the driving environment and determine the movement of such objects. Sensors typically include radio detection and ranging sensors (radar), light detection and ranging sensors (lidar), various types of digital cameras, sonar, position sensors, etc. Different types of sensors offer different, and often complementary, benefits. For example, radar and lidar emit electromagnetic signals (radio or optical signals) that reflect off objects and convey information that allows the distance to the object (e.g., from the signal's time of flight) and the object's velocity (e.g., from the Doppler shift in the signal's frequency) to be determined. Radar and lidar can cover a full 360-degree view, for example, by using a scanning transmitter of the sensing beam. The sensing beam may make numerous reflections that cover the driving environment in a dense grid of return points, each of which may be associated with the distance to the corresponding reflecting object and the radial velocity (the component of velocity along the line of sight) of the reflecting object.

[0010] Existing systems and methods for object identification and tracking use various sensing modalities, e.g., lidar, radar, camera, etc., to acquire images of an environment. The images are then processed by a trained machine learning model to identify the location of various objects in the image (e.g., in the form of bounding boxes), the state of object motion (e.g., speed as detected by lidar or radar Doppler effect-based sensors), the type of object (e.g., vehicle or pedestrian), etc. Object motion (or any other evolution, such as splitting a single object into multiple objects) can be implemented by creating and maintaining tracks associated with specific objects.

[0011] Using multiple sensing modalities (e.g., lidar, radar, cameras, etc.) to acquire complementary data often improves the accuracy of object detection, identification, and tracking, but at the expense of significant costs for sensing hardware and processing software. For example, lidar sensors can provide valuable information about the distance to various reflective surfaces in the external environment. However, lidar sensors operate by actively probing the external environment using optical signals and are expensive optical and electronic devices that require significant maintenance and periodic calibration. Lidar returns (point clouds) must be processed, segmented into groups associated with distinct hypothesized objects, and matched with objects detected using other sensing modalities (e.g., cameras), which require additional processing and memory resources. On the other hand, cameras operate by passively collecting light (and / or infrared electromagnetic waves) emitted (or reflected) by objects in the environment and are significantly simpler and cheaper to design, install, and operate. As a result, various driver assistance systems that do not deploy lidar (for cost and maintenance reasons) typically include one or more cameras. Cameras can also be more easily installed in a variety of fixed locations and used for traffic monitoring and control, public and private safety applications, etc. Cameras based on optical or infrared imaging technology have certain advantages over radar in that, while they allow for the detection of the distance to an object (and the object's speed), they operate in a range of wavelengths that inherently have lower resolution compared to cameras. Therefore, the ability to detect and identify objects based on camera images alone is beneficial.

[0012] However, cameras project the three-dimensional (3D) external environment onto a two-dimensional imaging surface (e.g., the camera's photodetector array), which can be flat or curved. This creates two related challenges. On the one hand, the distance to an object (often referred to as the object's depth in the image) is not immediately known (but can often be determined from the context of the imaged object). On the other hand, because camera images are subject to perspective distortion, the distance between objects varies depending on their depth, even if the number of pixels separating their images is the same. Furthermore, objects whose representations are close to each other may nevertheless be separated by a significant distance (e.g., a car and a pedestrian visible behind it). Existing machine learning techniques for object detection sometimes attempt to map objects to a top-down view, also known as bird's-eye view (BEV), in which objects are represented on a convenient manifold, such as a plane seen from above, and characterized by a simple set of Cartesian coordinates. Object identification and tracking can then be performed directly within the BEV representation. The success of such techniques depends on accurately mapping objects to the BEV. This in turn requires accurate estimation of the distance to various objects, as misplacement of objects inside the BEV can not only result in errors in ascertaining the distance to road users, but can also lead to the loss of important contextual information.

[0013] Aspects and embodiments of the present disclosure address these and other challenges of existing technologies by enabling methods and systems that use depth information for training machine-learned object detection, classification, and tracking models. In particular, the disclosed technology includes a system of neural networks (NNs) trained to process perspective camera images. The first NN can be configured to associate context with various pixels (e.g., in the form of feature vectors) and further estimate the probability that the pixels have several discretized depth values. As described in more detail below, the first neural network can be trained using training camera images annotated with ground truth, including depth data for at least some pixels of the images. The depth data can be acquired by an appropriate sensor capable of detecting distance to an object, e.g., lidar, radar, sonar, or any other distance-aware sensor. During inference, the trained first NN can process a new set of perspective camera images and estimate depths for various pixels, e.g., as the probability P(d) that a given pixel indicates an object (or part of an object) located at a distance d from the camera. The distance distribution can be combined with the pixel's feature vector FV(c), where c is the context space index, to obtain a feature tensor FT(c,d) that characterizes both the likely location (depth or distance d relative to the object) and the object's context. The feature tensors for various pixels with coordinates w, h in the image are used to generate a combined feature tensor for the entire image {FT(c,d)} w,h}→CFT(c,d,w,h). Next, a mapping transformation d,w,h→x,y,z can be performed from the perspective coordinates w,h,d (width, height, depth) to a set of Cartesian coordinates x,y,z. The combined feature tensor, CFT(c,x,y,z), in the new coordinates is then projected onto the horizontal plane, for example by averaging or adding the elements of the combined feature tensor along a vertical column of pixels, to obtain the BEV projection context tensor, PCT(c,x,y) = Σi CFT(c,x,y,z i) can be obtained. The projected context tensor PCT(c,x,y) can then be processed by a second trained NN to identify objects, extract semantic information (e.g., object type), precisely localize objects, etc.

[0014] Many variations of these techniques are described herein. In some embodiments, the first NN can include a first sub-network trained to output the depth distribution P(d) and a separate second NN trained to output the feature vector FV(c). In some embodiments, the first NN and the second NN can be trained simultaneously. In such embodiments, the first NN and the second NN are sub-networks of an end-to-end model architecture that are trained together. In some embodiments, the first NN can be first trained using both training images and depth ground truth, and the second NN is then trained using training images but not depth ground truth. Any number of images taken simultaneously (or nearly simultaneously) can be processed simultaneously, for example, multiple images taken by a peripheral vision camera (SVC) taken during a single camera cycle. In some implementations, images taken at different times can be processed simultaneously. For example, at a particular time t j Using the images taken at j) can be generated. Multiple BEV context tensors can then be processed at once by a second NN. Multiple depictions of the same object at the same or different locations at multiple times can increase the likelihood of correct segmentation and identification of objects in an image. In some embodiments, the second NN can have a common backbone and multiple classification heads. For example, one classification head can be trained to output a semantic segmentation of the input image. A second classification head can be trained to output the geometric centers of various objects. A third classification head can be trained to output the distances of various pixels in the BEV representation to the geometric center of the object. The combined output of the classification heads can be used to provide object classification (type, class) along with identification of the object's boundary. In some embodiments, training of the first NN and / or the second NN can be performed using a teacher-student distillation framework. More specifically, the output of a teaching model trained using data acquired using multiple sensing modalities (camera, lidar, radar, sonar, etc.) can be used as ground truth for training the student model's NN. A Student model may be a distillation of a teaching model, e.g., a model with a reduced number of neuron layers and / or neurons within a particular layer. As a result, the Student model may be more easily deployable on vehicles with less powerful processing and memory resources while retaining substantial functionality of the teaching model.

[0015] The described embodiments depart from traditional object detection and classification paradigms by using depth ground truth data to complement BEV segmentation techniques with efficient co-training of models. Advantages of the described embodiments include (but are not limited to) rapid and accurate detection, identification, and tracking of objects in a manner that avoids the significant computational overhead of processing data from multiple sensing modalities. Because machine learning models trained and deployed as disclosed herein are capable of efficient object detection based on camera images, the models can be deployed on a variety of platforms (e.g., autonomous vehicles and vehicles equipped with driver assistance technologies), including systems with modest computational resources.

[0016] 1 illustrates components of an example autonomous vehicle (AV) 100 using a bird's-eye view and deploying a model trained using depth ground truth data for efficient object detection and classification, according to some embodiments of the present disclosure. An autonomous vehicle may include a motor vehicle (such as a car, truck, bus, motorcycle, atv, recreational vehicle, or any specialized agricultural or construction vehicle), an aircraft (such as an airplane, helicopter, or drone), a marine vessel (such as a ship, boat, yacht, or submarine), a spacecraft (a controllable object operating outside the Earth's atmosphere), or any other self-propelled vehicle (e.g., a robot, a factory or warehouse robotic vehicle, a sidewalk delivery robotic vehicle, or the like) capable of operating in an autonomous driving mode (without or reduced human input).

[0017] Vehicles such as those described herein may be configured to operate in one or more different driving modes. For example, in a manual driving mode, a driver may directly control acceleration, deceleration, and steering via inputs such as an accelerator pedal, brake pedal, steering wheel, etc. Vehicles may also operate in one or more autonomous driving modes, including, for example, a semi-autonomous or partially autonomous driving mode in which a human exercises some amount of direct or remote control over driving operations, or a fully autonomous driving mode in which the vehicle handles driving operations without direct or remote control by a human. These vehicles may be known by different names, including, for example, autonomous vehicles, automated vehicles, etc.

[0018] As described herein, in a semi-autonomous or partially autonomous driving mode, the vehicle assists with one or more driving maneuvers (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), or emergency braking), but the human driver is expected to maintain situational awareness of the vehicle's surroundings and supervise the assisted driving maneuvers. Here, the vehicle may perform all driving tasks in certain situations, but the human driver is expected to be responsible for taking control as needed.

[0019] For simplicity and brevity, various systems and methods are described below in conjunction with autonomous vehicles, although similar technologies may be used in various driver assistance systems that fall short of fully autonomous driving systems. In the United States, the Society of Automotive Engineers (SAE) defines different levels of automated driving operation to indicate how much or how little control a vehicle has over the driving experience, but different organizations in the United States or other countries may categorize the levels differently. More specifically, the disclosed systems and methods may be used in SAE Level 2 driver assistance systems, which implement steering, braking, acceleration, lane centering, adaptive cruise control, and other driver assistance functions. The disclosed systems and methods may be used in SAE Level 3 driver assistance systems, which are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 automated driving systems, which operate autonomously under most normal driving conditions and require only occasional attention from a human operator. In all such driver assistance systems, accurate lane estimation can be performed automatically without driver input or control (e.g., while the vehicle is moving), resulting in improved reliability of vehicle positioning and navigation, and overall safety of autonomous, semi-autonomous, and other driver assistance systems. As noted above, in addition to the way SAE classifies levels of autonomous driving operation, other organizations in the United States or other countries may classify levels of autonomous driving operation differently. Without limitation, the systems and methods disclosed herein may be used in driver assistance systems defined by the levels of autonomous driving operation of these other organizations.

[0020] The driving environment 101 may include any objects (moving or non-moving) located outside the AV, such as roads, buildings, trees, bushes, sidewalks, bridges, mountains, other vehicles, pedestrians, bridge piers, embankments, landing strips, animals, birds, etc. The driving environment 101 may be an urban, suburban, rural, etc. In some embodiments, the driving environment 101 may be an off-road environment (e.g., agricultural or other farmland). In some embodiments, the driving environment may be an indoor environment, such as an industrial plant environment, a shipping warehouse, a hazardous area of ​​a building, etc. In some embodiments, the driving environment 101 may be substantially flat, with various objects moving parallel to the surface (e.g., parallel to the surface of the Earth). In other embodiments, the driving environment may be three-dimensional and may include objects capable of moving along all three directions (e.g., balloons, falling leaves, etc.). Hereinafter, the term "driving environment" should be understood to include all environments in which autonomous movement (e.g., SAE Level 5 and SAE Level 4 systems), conditional autonomous movement (e.g., SAE Level 3 systems), and / or movement of a vehicle equipped with driver assistance technology (e.g., SAE Level 2 systems) may occur. Furthermore, "driving environment" may include any possible flight environment of an aircraft (or spacecraft) or marine environment of a naval vessel. Objects in the driving environment 101 may be located at any distance from the AV, from a close distance of a few feet (or less) to several miles (or more).

[0021] Example AV 100 may include sensing system 110. Sensing system 110 may include various electromagnetic (e.g., optical, infrared, radio, etc.) and non-electromagnetic (e.g., acoustic) sensing subsystems and / or devices. Sensing system 110 may include one or more lidars 112, which may be laser-based units capable of determining distance to and velocity of objects within driving environment 101. Sensing system 110 may include one or more radars 114, which may be any system that utilizes radio or microwave frequency signals to detect objects within AV 100's driving environment 101. LIDAR(s) 112 and / or RADAR(s) 114 may be configured to sense both the spatial location of objects (including their spatial dimensions) and their velocity (e.g., using Doppler shift techniques). Hereinafter, "velocity" refers to both how fast an object is moving (object speed) as well as the direction of the object's motion. Each of the lidar(s) 112 and radar(s) 114 may include a coherent sensor, such as a frequency-modulated continuous wave (FMCW) lidar or radar sensor. For example, the lidar(s) 112 and / or radar(s) 114 may use heterodyne detection for velocity determination. In some embodiments, ToF and coherent lidar (or radar) functionality is combined into a lidar (or radar) unit that can simultaneously determine both the distance to a reflecting object and its radial velocity. Such a unit may be configured to operate in a non-coherent sensing mode (ToF mode) and / or a coherent sensing mode (e.g., a mode using heterodyne detection), or both modes simultaneously. In some embodiments, multiple lidars 112 and / or radars 114 may be mounted on the AV 100.

[0022] The LIDAR 112 (and / or RADAR 114) may include one or more light sources (and / or radio / microwave sources) that generate and emit signals and one or more detectors of signals reflected from objects. In some embodiments, the LIDAR 112 and / or RADAR 114 may perform 360-degree scanning in the horizontal direction. In some embodiments, the LIDAR 112 and / or RADAR 114 may be capable of spatial scanning along both the horizontal and vertical directions. In some embodiments, the field of view may be up to 90 degrees vertically (e.g., at least a portion of the area above the horizon is scanned by the LIDAR or RADAR signal). In some embodiments (e.g., aerospace applications), the field of view may be spherical (consisting of two hemispheres).

[0023] The sensing system 110 may further include one or more cameras 118 to capture images of the driving environment 101. The cameras 118 may operate in the visible portion of the electromagnetic spectrum, for example, in the 300-800 nm wavelength range (also referred to herein as the optical range for simplicity). Some of the optical range cameras 118 may use a global shutter, while others 118 may use a rolling shutter. The images may be two-dimensional projections of the driving environment 101 (or portions of the driving environment 101) onto the camera's projection surface (planar or non-planar). Some of the cameras 118 of the sensing system 110 may be video cameras configured to capture a continuous (or quasi-continuous) stream of images of the driving environment 101. The sensing system 110 may also include one or more sonars 116, such as ultrasonic sonars, for active sound probing of the driving environment 101, and one or more microphones for passively listening to sounds in the driving environment 101. The sensing system 110 may also include one or more infrared range cameras 119, also referred to herein as IR cameras 119. The IR camera(s) 119 may use focusing optics (e.g., made of germanium-based materials, silicon-based materials, etc.) configured to operate in the wavelength range of microns to tens of microns or more. The IR camera(s) 119 may include a phased array of IR detector elements. The pixels of the IR image generated by the camera(s) 119 may represent the total amount of IR radiation collected by the respective detector element (associated with the pixel), the temperature of the physical object from which IR radiation is collected by the respective detector element, or any other suitable physical quantity.

[0024] The sensory data acquired by the sensing system 110 may be processed by the data processing system 120 of the AV 100. For example, the data processing system 120 may include a perception system 130. The perception system 130 may be configured to detect and track objects in the driving environment 101 and recognize the detected objects. For example, the perception system 130 may analyze images captured by the camera(s) 118 and may be capable of detecting traffic signals, road signs, road layouts (e.g., lane boundaries, intersection topology, parking designations, etc.), the presence of obstacles, etc. The perception system 130 may also receive radar sensory data (Doppler data and ToF data) to determine the distances to various objects in the environment 101 and the velocities of such objects (line of sight and, in some embodiments, lateral, as described below). In some embodiments, the perception system 130 may use radar data in combination with data captured by the camera(s) 118, as described in more detail below.

[0025] Perception system 130 may include one or more modules to facilitate efficient and reliable detection, identification, and tracking of objects, including an object detection and classification model (ODCM) 132 with depth co-training, which may be used to process data provided by sensing system 110. In some embodiments, during inference, ODCM 132 may receive data from (optical range) camera(s) 118 and / or IR camera(s) 119. During training, as described in more detail in connection with FIGS. 2 and 4A-B , ODCM 132 may process data from camera(s) 118 and / or IR camera(s) 119 while using range (distance) data acquired by at least some of lidar(s) 112, radar(s) 114, sonar(s) 116, etc. ODCM 132 may include one or more trained machine learning models (MLMs) used to process received images to detect objects depicted in the images and classify the detected objects.

[0026] The perception system 130 may further receive information from a global navigation satellite system (GNSS) positioning subsystem (not shown in FIG. 1 ), which may include a GNSS transceiver (not shown) configured to obtain information about the position of the AV relative to the Earth and its surroundings. The positioning subsystem may use the positioning data (e.g., GNSS and inertial measurement unit (IMU) data) in conjunction with the sensing data to help accurately determine the position of the AV with respect to fixed objects (e.g., roadways, lane boundaries, intersections, sidewalks, crosswalks, road signs, curbs, surrounding buildings, etc.) in the driving environment 101, whose locations may be provided by the map information 124. In some embodiments, the data processing system 120 may receive non-electromagnetic data, such as audio data (e.g., ultrasonic sensor data from the sonar 116 or data from a microphone picking up an emergency vehicle siren), temperature sensor data, humidity sensor data, pressure sensor data, weather data (e.g., wind speed and direction, precipitation data), etc.

[0027] The data processing system 120 may further include an environment monitoring and prediction component 126, which can monitor how the driving environment 101 evolves over time, for example, by tracking the position (relative to the Earth) and velocity of moving objects. In some embodiments, the environment monitoring and prediction component 126 can track the changing appearance of the environment due to the movement of the AV relative to the environment. In some embodiments, the environment monitoring and prediction component 126 can make predictions about how various moving objects in the driving environment 101 will be positioned within a prediction time frame. The predictions may be based on the current state of the moving objects, including their current positions (coordinates) and velocities. Additionally, the predictions can be based on the history of the moving objects' motion (tracked dynamics) during a specific period preceding the current moment. For example, based on stored data regarding a first object indicating accelerated motion of the first object during the previous three seconds, the environment monitoring and prediction component 126 can conclude that the first object is resuming its motion from a stop sign or red light signal. Thus, the environmental monitoring and prediction component 126 can predict where a first object is likely to be within the next three or five seconds of movement, taking into account the layout of the roadway and the presence of other vehicles. As another example, based on stored data about a second object indicating the second object's slowed movement within the previous two seconds, the environmental monitoring and prediction component 126 can conclude that the second object is stopped at a stop sign or red light. Thus, the environmental monitoring and prediction component 126 can predict where the second object is likely to be within the next one or three seconds. The environmental monitoring and prediction component 126 can periodically check the accuracy of its prediction and revise the prediction based on new data obtained from the sensing system 110. The environmental monitoring and prediction component 126 can operate in conjunction with the ODCM 132. For example, the environmental monitoring and prediction component 126 can track the relative movement of the AV and various objects (e.g., reference objects that are stationary or moving relative to the Earth).

[0028] The data generated by the perception system 130, the GNSS processing module 122, and the environmental monitoring and prediction component 126 can be used by an autonomous driving system, such as the AV control system (AVCS) 140. The AVCS 140 may include one or more algorithms that control how the AV should behave in various driving situations and environments. For example, the AVCS 140 may include a navigation system for determining a global driving path to a destination. The AVCS 140 may also include a driving path selection system for selecting a particular path through the immediate driving environment, which may include selecting a lane, navigating traffic jams, selecting a location to make a U-turn, selecting a trajectory for a parking maneuver, etc. The AVCS 140 may also include an obstacle avoidance system for safely avoiding various obstacles (e.g., rocks, stopped vehicles, etc.) in the AV's driving environment. The obstacle avoidance system may be configured to assess the size of an obstacle and the trajectory of the obstacle (if the obstacle is moving) and select an optimal driving strategy (e.g., braking, steering, accelerating, etc.) to avoid the obstacle.

[0029] The algorithms and modules of AVCS 140 may generate instructions for various systems and components of the vehicle, such as powertrain, braking, and steering 150, vehicle electronics 160, signaling 170, and other systems and components not explicitly shown in FIG. 1 . Powertrain, braking, and steering 150 may include an engine (internal combustion engine, electric engine, etc.), transmission, differential, axles, wheels, steering mechanism, and other systems. Vehicle electronics 160 may include an on-board computer, engine management, ignition, communication systems, car computers, telematics, in-car entertainment systems, and other systems and components. Signaling 170 may include high and low headlights, stop lights, turn signals and taillights, horns and alarms, interior lighting systems, dashboard notification systems, passenger notification systems, radio and wireless network transmission systems, etc. Some of the commands output by AVCS 140 may be delivered directly to powertrain, braking, and steering 150 (or signaling 170), while other commands output by AVCS 140 are first delivered to vehicle electronics 160, which generates commands to powertrain, braking, and steering 150 and / or signaling 170.

[0030] In one example, the ODCM 132 can determine that an image acquired by the camera(s) 118 includes a depiction of an object and can further classify the object as a bicycle. The environmental monitoring and prediction component 126 can track the bicycle and determine that the bicycle is traveling at a speed of 15 mph along an intersecting road perpendicular to the vehicle's direction of motion. In response to such a determination, the data processing system 120 can determine that the vehicle needs to slow down to allow the bicycle to clear the intersection. The AVCS 140 can output instructions to the powertrain, brakes, and steering 150 (either directly or via the vehicle electronics 160) to: (1) change the throttle setting to reduce fuel flow to the engine and lower engine speed; (2) downshift the drivetrain, via the automatic transmission, into a lower gear; and (3) activate the brake unit (in cooperation with the engine and transmission) to reduce vehicle speed. After the ODCM 132 and / or the environmental monitoring and prediction component 126 determine that the bicyclist has crossed the intersection, the AVCS 140 can output commands to the powertrain, braking, and steering 150 to resume the vehicle's previous speed setting.

[0031] FIG. 2 illustrates an example architecture 200 of a portion of a perception system capable of efficient object detection and classification, according to some embodiments of the present disclosure. Input to a perception system (e.g., perception system 130 of FIG. 1 ) can include multiple camera images 202, which can be training images (during the training phase) or images taken onboard at runtime (during the estimation phase). The images 202 can be combined into frames. A frame should be understood as any set of images showing the external environment along any direction relative to the sensing system (e.g., sensing system 110 of an autonomous vehicle). In particular, camera images 202 can refer to panoramic images taken by peripheral vision cameras, images taken by directional cameras, such as front-view cameras, side-view cameras (SVCs), rear-view cameras, and the like, or any combination thereof. In some embodiments, images acquired by different cameras can be synchronized so that all images in a given frame have the same timestamp (up to the maximum synchronization accuracy). In some embodiments, some images in a given frame can have (controlled) time offsets, such as a time offset associated with the scanning operation of the SVC. The camera images 202 may be processed by the ODCM 132. Additional inputs to the ODCM 132 may include depth data 204, which may be data acquired using lidar sensor(s), radar sensor(s), sonar sensor(s), etc. In some implementations, the depth data 204 may be used during training of the ODCM 132 but may not be used during the inference phase. The images 202 and depth data 204 may include directional indexing. More specifically, various pixels of the images 202 and return points of the depth data 204 may be associated with known directions in space (e.g., from camera calibration). For example, the camera images may include an intensity map indexed by any suitable set of coordinates that characterize directions in space, e.g., I(w,h), where w, h may be Cartesian pixel coordinates (in the imaging plane), I(θ,φ), where θ and φ are polar and azimuth angles, respectively, or any other set of coordinates.In some embodiments, multiple sets of coordinates can be used for different tasks, facilitated by stored mappings (transformations) between the different sets. The depth data 204 may include, for example, the radial distance R(w,h) to an object associated with a particular pixel w,h in the image 202, determined from the ToF of a lidar / radar / sonar signal.

[0032] Each image 202 may have one color I(w,h) (for black and white images) or multiple colors I c (w,h) (for color images). At least some images 202 may be infrared (IR) camera images acquired by an array of IR detectors (pixels) capable of operating in a wavelength range from a few microns to tens of microns or more. The IR image may include an intensity IR(w,h) representing the total amount of IR radiation collected by each detector. In some embodiments, the IR image may include a pseudocolor map, in which the presence of a particular pseudocolor may represent the total collected intensity IR(w,h). In some implementations, the collected intensities may be used to determine a temperature map T(w,h) of the environment. Thus, different embodiments may represent IR camera data using different representations (e.g., intensity map, pseudocolor map, temperature map, etc.).

[0033] In some embodiments, architecture 200 may include a normalization module (not shown in FIG. 2) that can resize each image 202 to match the size of the input to ODCM 132. In some implementations, the normalization module normalizes the intensities of pixels in image 202, for example, as [I min ,I max ], where I min is the minimum intensity and I max is the maximum intensity that the ODCM 132 is configured to process. In some embodiments, the minimum intensity is zero, I min = 0. Additionally, the normalization module may perform other pre-processing of the image 202, including filtering, denoising, etc.

[0034] The normalized and preprocessed images may be processed by various components of the ODCM 132 to detect the presence of objects 232 in the driving environment and classify the detected objects 232. The ODCM 132 may include any suitable machine learning model, such as a lookup table, a geometric shape mapping, a mathematical formula, a decision tree algorithm, a support vector machine, a deep neural network, or the like, or any combination thereof. The deep neural network may include a convolutional neural network, a recurrent neural network (RNN), a fully connected neural network, a fully connected neural network, a long short-term memory neural network, a Boltzmann machine, or the like, or any combination thereof.

[0035] The depth estimation network (DEN) 210 can output a prediction of the depth of the object imaged by various pixels in the image(s) 202. The context feature network (CFN) 220 can output a feature vector for the various pixels. The feature vector can be a multi-element string of data in feature space. The depth prediction and the feature vector can be combined into a feature tensor. The feature tensor can undergo one or more transformations, as described in more detail in connection with FIGS. 3A-C, to map the pixel data (intensity and depth) from the perspective view to the top-down BEV view. The feature tensor combined into the projected BEV context tensor can be processed by the BEV feature network 230, which outputs the detected / classified object 232, which can be classified among multiple classes, e.g., car, truck, bus, pedestrian, unknown object, etc.

[0036] The detected / classified objects 232 may undergo post-processing 234, which may include object tracking to track the movement of the detected objects across multiple frames of the image 202. Each object may be assigned a detection track, which may be characterized by some or all of a bounding box for depiction of the particular object across multiple frames, object type, object size, object pose, object motion (e.g., velocity, acceleration, etc.), etc. Post-processing 234 may further include generating any graphical, e.g., pixel-based (e.g., heat map) or (curve-based) vectorized representation of the trajectory, including trajectory, pose, velocity regimes of various objects, etc. In some embodiments, post-processing 234 may include processing the detected tracks using one or more models that predict the motion of the detected objects, e.g., models that track the velocity, acceleration, etc. of the detected objects. For example, a Kalman filter or any other suitable filter that combines the predicted motion of a particular object with the detected motion of the object may be used to more accurately estimate the object's position and motion.

[0037] The detected / classified objects 232 and the tracking data generated by post-processing 234 can be provided to AVCS 140. AVCS 140 evaluates the trajectories of the objects in various trajectories and determines whether to modify the vehicle's current driving trajectory, taking into account the position and speed of the tracked objects. For example, if a tracked pedestrian or cyclist is within a certain distance from the vehicle, AVCS 140 can slow the vehicle to a speed that ensures it can safely avoid the pedestrian or cyclist. Alternatively, AVCS 140 can change lanes or perform some other driving maneuver, for example, if there is no obstacle in the adjacent lane.

[0038] Training may be performed by a training engine 242 hosted by a training server 240, which may be an external server deploying one or more processing units, such as a central processing unit (CPU), a graphics processing unit (GPU), etc. In some implementations, the ODCM 132 may be trained by the training engine 242 and then downloaded to a vehicle deploying the perception system 130. The ODCM 132 may be trained using training data including training inputs 244 and corresponding target outputs 246 (the correct matches for each training input), as shown in FIG. 2. During training of the ODCM 132, the training engine 242 may find patterns in the training data that map each training input 244 to the corresponding target output 246.

[0039] In some implementations, the ODCM 132 may be trained using images and other sensory data recorded during a driving mission and annotated with ground truth. For training the depth estimation network 210 of the ODCM 132, the ground truth may include distances to various pixels of the images in the training input 244. Training the BEV feature network 230 may involve ground truth including correct identification of the locations (e.g., bounding boxes) of various objects in the training input 244 and semantic (e.g., class, type, etc.) information for the objects. In some embodiments, the ground truth for training any or all of the depth estimation network 210, context feature network 220, and BEV feature network 230 may include the output of one or more teacher models, as described below in conjunction with FIGS. 4A-B. The ground truth may include correct connectivity of the same object across multiple batches of images / frames, correct velocity of the object, etc.

[0040] The training engine 242 may have access to a data repository 250 that stores multiple camera / IR camera images 252 and lidar / radar (or sonar) images 254 acquired during driving situations in various driving environments (e.g., urban driving missions, highway driving missions, rural driving missions, etc.). During training, the training engine 242 may select (e.g., randomly) a number of sets of camera / IR camera images 252 and sets of lidar / radar images 254 as training data. The training data may be annotated with correct object identification. In some embodiments, the annotation may be performed by a developer before the annotated training data is placed in the data repository 250. The annotated training data retrieved by the training server 240 from the data repository 250 may include one or more training inputs 244 and one or more target outputs 246. The training data may also include mapping data 248 that maps the training inputs 244 to the target outputs 246. For example, the mapping data 248 may identify a bounding box of a passenger's car in each of a batch of N consecutive frames acquired by the vehicle's forward-facing camera. Mapping data 248 may include an identifier for the training data, the location of the vehicle, the size and identification of the vehicle, the speed and direction of movement of the vehicle, and other suitable information.

[0041] During training of the ODCM 132, the training engine 242 may use an appropriate loss function 245 to evaluate the difference between the output of the ODCM 132 (or various networks and sub-networks of the ODCM 132) and the target output 246. In some embodiments, different loss functions 245 may be used to train the depth estimation network 210, the context feature network 220, and / or the BEV feature network 230. During training of the ODCM 132, the training engine 242 may vary parameters (e.g., weights and biases) of the various networks and sub-networks of the ODCM 132 until the model minimizes the loss function(s) 245 and successfully learns the target output 246, e.g., how to successfully identify and classify various objects in the external environment. In some embodiments, the various networks and sub-networks of the ODCM 132 may be trained separately, e.g., with the depth estimation network 210 trained before the training of the context feature network 220 and / or the BEV feature network 230. In some embodiments, various networks and sub-networks of ODCM 132 may be trained together (e.g., simultaneously). For example, depth estimation network 210, context feature network 220, and / or BEV feature network 230 may be sub-networks of a single end-to-end neural network architecture that are trained together.

[0042] Data repository 250 may be persistent storage capable of storing camera / IR camera images, lidar / radar / sonar data, and data structures configured to facilitate detection and identification according to embodiments of the present disclosure. Data repository 250 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage disks, tape, or hard drives, network-attached storage (NAS), storage area networks (SAN), etc. Although illustrated separately from training server 240, in embodiments, data repository 250 may be part of training server 240. In some embodiments, data repository 250 may be a network-attached file server, while in other embodiments, data repository 250 may be some other type of persistent storage, such as an object-oriented database, a relational database, etc., that may be hosted by a server machine or one or more different machines accessible to training server 240 via a network (not shown in FIG. 2 ).

[0043] FIG. 3A is a schematic diagram illustrating an example operation of a model 300 trained using depth ground truth data for efficient object detection and classification using a bird's-eye view, in accordance with some embodiments of the present disclosure. The model illustrated in FIG. 3A may be the ODCM 132 of the perception system 130 illustrated in FIG. 1 or any other similar model. In some embodiments, the model illustrated in FIG. 3A may include a set of neural networks (NNs), such as a depth estimation network (DEN) 310, a context feature network (CFN) 320, a background effect feature network (BEF FN) 350, etc. Although shown as separate blocks in FIG. 3A, the various illustrated NNs and sub-networks may be part of the same NN that is trained together. Neurons in a neural network are associated with learnable weights and biases. The neurons may be arranged in layers. Some of the layers may be hidden layers. Any of the NNs or sub-networks illustrated in FIG. 3A may include multiple hidden neuron layers and may be configured to perform one or more functions that facilitate object detection and classification.

[0044] Input to the DEN 310 and CFN 320 may include one or more images 302, which may be perspective views acquired by one or more cameras. Any number of images 302 may be processed simultaneously. The input image(s) 302 may depict any portion of the external environment, up to a 360-degree panoramic peripheral view. In some embodiments, the total number of pixels in all image(s) 302 may be W×H, where W is the number of pixels along a first direction (e.g., horizontal) and H is the number of pixels along a second direction (e.g., vertical). In some embodiments, a "pixel" of an image 302 may correspond to a single element of a charge-coupled device (CCD) of the camera. In some embodiments, a "pixel" of an image 302 may be a downsampled combination (superpixel) corresponding to multiple CCD elements of the camera. For example, one or more images of a camera may have a number of pixels different from W×H, e.g., the dimensions of the input of the trained DEN 310 and CFN 320. In such embodiments, the camera image may first be rescaled (e.g., using techniques such as interpolation, downsampling / upsampling, etc.) to the target input size W×H. Similarly, the depth data 304 may be rescaled to the target input size W×H. In some embodiments, the depth data 304 may be available for only a subset of the W×H pixels, e.g., one depth value every N pixels.

[0045] The image(s) 302 can be in any suitable digital format (JPEG, TIFF, GIG, BMP, CGM, SVG, etc.). The image(s) 302 are represented by one or more intensity matrices I k (w,h), where 0≦w≦W and 0≦h≦H, and the color index k has a single value (for black and white images), three values ​​(for RGB images), four values ​​(for CMYK images), etc. The intensity value I k can assume continuous or discrete values ​​between 0 and 1 (or between any other selected limits, e.g., 0 to 255).

[0046] The DEN 310 calculates the probability P(d) that various pixels, e.g., a given pixel w, h, indicate an object located at a distance d from the camera. w,h In some implementations, the distance d is a number of intervals D, Δd1, Δd2, ..., Δd D can be discretized between the interval Δd i are not necessarily of equal size, but are distances, e.g., Δd1≦Δd2≦...≦Δd D The final interval Δd N may extend from a particular distance (e.g., 100m, 200m, etc.) to an infinite distance.

[0047] The DEN 310 may be trained using training images depicting objects in various environments and using depth data 304 as ground truth. The depth data 304 may be provided by a lidar (radar, sonar, etc.) sensor(s) and may represent the actual distance d to an object (or portion of an object) imaged by a corresponding camera pixel w, h. True During the training phase, the training engine 242 of FIG. 2 calculates the distribution P(d) output by the DEN 310. w,h The center (e.g., mean, median, etc.) of and the ground truth value distance d True The parameters (e.g., weights and biases) of the neurons of the DEN 310 can be varied to minimize a loss function that characterizes the difference between (w, h) and (w, h). In some implementations, the distribution P(d) w,h can be modeled as a categorical distribution, a Laplace distribution, a Gaussian distribution, or any other suitable distribution. The loss function minimized during training may include a focal loss function, a negative log likelihood loss function, or any other suitable loss function, e.g., a mean squared loss function, a cross entropy loss function, etc.

[0048] The CFN 320 can be trained, for example, using the same training images, to output a feature vector 322. The feature vector FV(c) for pixel w,h w,hrepresents a learned digital encoding that characterizes the appearance of the corresponding pixel and the graphical context provided by other pixels in the image. The feature vector FV(c) can have any number C of components (bits), c∈[1,C], selected taking into account the desired target accuracy of object detection and the specific computational resources on which the trained model (e.g., ODCM132) will be deployed, e.g., C=32, 64, 128, 256, etc. Higher values ​​of C favor higher accuracy of object detection and classification, while lower values ​​of C facilitate faster processing and / or deployment of the NN on systems with limited computational resources.

[0049] In some embodiments, the DEN 310 and the CFN 320 may be deep convolutional neural networks, e.g., having a U-net architecture with an encoder stage and a decoder stage. Each stage may include multiple convolutional neuron layers and one or more fully connected layers. The convolutions performed by each of the DEN 310 and the CFN 320 may include any number of convolution kernels of different dimensions trained to capture both the local and global context of the input image. In some embodiments, the DEN 310 and the CFN 320 may be completely independent, e.g., there are no edges connecting the neurons of the two networks. In some embodiments, the DEN 310 and the CFN 320 may share a common backbone and may have separate heads that output the depth distribution 312 and the feature vector 322. Each head may have any number of neuron layers and, in some embodiments, may have its own encoder-decoder architecture. In some implementations, at least some of the heads may include one or more fully connected layers.

[0050] The depth distribution 312 and the feature vector 322 can be combined into a feature tensor 332 by a lift transform 330. More specifically, the lift transform 330 transforms the feature vector FV(c) w,hEach pixel w,h described by (d) can be complemented with depth information from the depth distribution 312. For example, the lift transform 330 can be used to compute the corresponding depth distribution P(d) for the same pixel. w,h (of dimensions D × 1) for each feature vector FV(c) w,h The cross product (of dimensions C×1) can be calculated to obtain a feature tensor 332, e.g., for pixels w, h, as shown schematically with an insert in FIG. 3A.

number

[0051] Then, the feature tensor FT(c,d) calculated for each pixel w,h For example, by concatenating the feature tensors 332 for different pixels, a combined feature tensor 334 for the entire image can be obtained using {FT(c,d) w,h} → CFT(c,d,w,h). The combined feature tensor CFT(c,d,w,h) has dimensions C×D×W×H. The combined feature tensor 334 may then undergo 2D mapping 340. More specifically, 2D mapping 340 may generate a projected feature tensor 342 that uses a convenient set of planar coordinates, e.g., Cartesian coordinates x and y, or polar coordinates r and θ, within the ground plane.

[0052] The 2D mapping 340 may be a two-part transformation. During the first part, viewpoint coordinates d, w, h may be transformed into 3D Cartesian coordinates d, w, h → x, y, z (or 3D cylindrical coordinates, w, h → r, θ, z), where z is the vertical coordinate (the direction perpendicular to the ground). The transformation d, w, h → x, y, z may be a projective transformation parameterized with the focal length of the camera, the direction of the camera's optical axis, and other similar parameters. In examples where the image 302 is acquired by multiple cameras (or cameras with rotating optical axes), the transformation d, w, h → x, y, z may include multiple projective transformations, e.g., with separate transformations used for pixels w, h provided by different cameras.

[0053] During the second part, a 2D mapping 340 can project the combined feature tensor, represented by new coordinates CFT(c,x,y,z), onto a horizontal plane to obtain a projected (BEV) feature tensor 342. For example, to obtain a C×W×H projected feature tensor PCT(c,x,y), the combined feature tensor can be divided into elements associated with each vertical pillar of a pixel, e.g., PCT(c,x,y)= Σi CFT(c,x,y,z i ) can be summed (or averaged) to the coordinate z i The summation for different coordinates z i The different weights w assigned to i PCT(c,x,y)= Σi w i ×CFT(c,x,y,z i ), e.g., the larger the weight w i is assigned to pixels that image objects within a certain elevation above the ground (e.g., up to a few meters), and lower weights are assigned to other elevations (e.g., to remove spurious objects such as tree branches, electrical cables, etc. that do not obstruct vehicle movement).

[0054] The projected feature tensor 342 characterizes objects and their locations within the BEV, effectively eliminating perspective distortion. The BEV FN 350 can then process the projected feature tensor 342 to identify objects shown in the image 302 and classify the identified objects. In some embodiments, the BEV FN 350 can be a network having both an encoder stage and a decoder stage. In some embodiments, the BEV FN 350 can be a network having a decoder stage, but the encoder stage is part of the CFN 320. In some embodiments, the BEV FN 350 can serve as the backbone of one or more classification heads 360-n. Each classification head 360-n can output a different type of information about the objects depicted in the image 302. For example, head A 360-1 can output the bounding box of the object, head B 360-2 can output the type and / or size of the object, head C 360-3 can output the pose (position and orientation) of the object, etc.

[0055] The DEN 310, CFN 320, and BEV FN 350 are shown as separate blocks in FIG. 3A and may each have an encoder stage and a decoder stage. Other embodiments are also within the scope of this disclosure. More specifically, FIG. 3B is a schematic diagram of one embodiment of a model 301 that uses a bird's-eye view and is trained using depth ground truth data for efficient object detection and classification. As shown in FIG. 3B, the DEN 310 and CFN 320 may be supported by a common backbone network, such as the encoder network 305. Each of the DEN 310 and the CFN 320 may be a separate decoder trained to process the common output of the encoder network 305. In some embodiments, the CFN 320 may be a decoder network, while the DEN 310 may be a head with one or more fully connected layers. In some embodiments, neither the DEN 310 nor the CFN 320 includes a decoder. In some embodiments, the DEN 310 may be a decoder network, while the CFN 320 may be a head with one or more fully connected layers. In some embodiments, one of the DEN 310 or the CFN 320 may be absent. For example, if the CFN 320 is not present, the feature vector 322 may be output directly by the encoder network 305, but the DEN 310 may include a decoder that further processes a copy of the output (the feature vector 322) to generate the depth distribution 312.

[0056] In some embodiments, any or all of the encoder network 305, DEN 310, CFN 320, and BEV FN 350 may be trained together. In some embodiments, some of the encoder network 305, DEN 310, CFN 320, or BEV FN 350 may be trained in stages. More specifically, FIG. 3C illustrates a model 303 in which the depth estimation network 310 (together with the encoder network 305, if deployed) is pre-trained using depth ground truth data in accordance with some embodiments of the present disclosure. Pre-training the DEN 310 is performed to estimate the center of distribution P(d)w,h and the ground truth distance d provided by the depth data 304. True (w,h) may be implemented using an appropriate loss function that evaluates the difference between (w,h). The difference may then be backpropagated through various layers of neurons in the DEN 310 (and encoder network 305, if deployed) until the DEN 310 (and encoder network 305, if deployed) learns to predict the estimated depth of the pixel with the desired accuracy. After pre-training 306 of the DEN 310 (and encoder network 305, if deployed) is complete, the output of the DEN 310 may be used as ground truth during training of the CFN 320 and / or BEV FN 350, which may be trained together or sequentially.

[0057] FIG. 4A is a schematic diagram illustrating a distillation framework 400 for training a bird's-eye view model using depth ground truth data, according to some embodiments of the present disclosure. The distillation framework 400 may include training a student model 401-S using the output of a previously trained teaching model 401-T. The student model 401-S may include a student DEN 410-S, a student CFN 420-S, and a student BEV FN 450-S, which may operate substantially as described in connection with their respective networks in FIGS. 3A-C. In particular, the student DEN 410-S and the student CFN 420-S may process one or more image(s) 402 and output depth distributions 412 and feature vectors 422 for various pixels of the image(s) 402, which may undergo a lifting transformation / 2D mapping 430, for example, similar to the lifting transformation 330 and 2D mapping 340, to obtain a projected feature tensor 442. The student BEV FN 450-S can process the projected feature tensor 442 to identify the detected (and classified) object 452.

[0058] Various outputs (including intermediate outputs) of the student ODCM 401-S can be compared to the ground truth obtained from the teacher ODCM 401-T. In some embodiments, as shown in FIG. 4A , the teacher ODCM 401-T can have an architecture similar to that of the student ODCM 401-S. In particular, the teacher DEN 410-T and the teacher CFN 420-T can process one or more image(s) 402, which may be the same images input to the student DEN 410-S and the student CFN 420-S. Additionally, the teacher DEN 410-T can process depth data 404 (e.g., lidar range data) that associates the depth of (distance to) objects shown in the image(s) 402. To accommodate the additional depth data input 404, the teacher DEN 410-T can have more input neurons compared to the student DEN-S 410-S. The taught DEN 410-T and taught CFN 420-T can output depth distributions 411 and feature vectors 421 for pixels of the image(s) 402, which can undergo a lift transformation / 2D mapping 430 to generate projected feature tensors 441, which can then be processed by the taught BEV FN 450-T to identify detected (and classified) objects 541.

[0059] To enable direct comparison of the various outputs (including intermediate outputs) of the student ODCM401-S with the outputs of the teacher ODCM401-T, at least some of the teacher DEN410-T, teacher CFN420-T, and teacher BEV FN450-T may have outputs of the same dimensions (number of neurons in the output layer) as the output dimensions of the student ODCM401-S, student DEN410-S, student CFN420-S, and student BEV FN450-S networks, respectively. Meanwhile, the teacher networks may have higher complexity, including the number of neuron layers, the number of connections between layers, the precision (number of bits) of the representation of the intermediate outputs, etc. In some embodiments, the teacher DEN 410-T and the teacher CFN 420-T may have more input neurons than the respective student networks and may therefore be configured to process images 402 at a higher resolution than the student networks. The Teach ODCM401-T can be used in an off-board setting, while the Student ODCM401-S can be used on a vehicle, traffic monitoring station, or any edge device with limited computational resources. As a result, the Teach ODCM401-T need not be limited in complexity and / or computation time.

[0060] The student ODCM401-S may be a lightweight model with substantially fewer neurons and neuron layers than the teacher ODCM401-T, allowing for faster processing (given the same processing and memory resources). The distillation process to obtain the student ODCM401-S may include identifying and removing (culling, shearing) nodes and / or edges that have little or reduced effect on the model's output, combining multiple neuron nodes and / or edges, etc. Furthermore, the output of the teacher ODCM401-T can be used as ground truth during training of the student ODCM401-S, as depicted schematically by the dashed arrows in Figure 4A. More specifically, the depth distribution 411 output by the teacher DEN410-T and the feature vector 421 output by the teacher CFN420-T can be used as ground truth during training of the student DEN410-S and the student CFN420-S. For example, depth distribution 412 (feature vector 422) may be compared with depth distribution 411 (feature vector 421) and parameters of student DEN 410-S (student CFN 420-S) until the difference is minimized. Similarly, projected feature tensor 441 and detected object 451 may be used as ground truth values ​​to be compared with projected feature tensor 442 and detected object 452. In some embodiments, the difference between detected object 452 and ground truth detected object 451 may be backpropagated through student BEV FN 450-S, but not backpropagated through student DEN 410-S and / or student CFN 420-S. In such an embodiment, student DEN410-S is trained based on the difference between depth distribution 412 and ground truth depth distribution 444 (similarly, student CFN420-S is trained based on the difference between feature vector 422 and ground truth feature vector 421).In other embodiments, the difference between the detected object 452 and the ground truth detected object 451 may be additionally backpropagated through the student DEN 410-S and / or the student CFN 420-S.

[0061] 4B is a schematic diagram illustrating another embodiment of a distillation framework 401 for training a bird's-eye view model using depth ground truth data, according to some embodiments of the present disclosure. The distillation framework 401 may differ from the distillation framework 400 of FIG. 4A in that there is no teacher DEN 410-T and, instead of a depth distribution 411, depth sensing data (e.g., lidar data or radar data) may be used as ground truth during training of the student DEN 410-S.

[0062] 5 is a schematic diagram illustrating operations 500 of a bird's-eye view model for providing instance segmentation and semantic segmentation, according to some embodiments of the present disclosure. The operations 500 may include processing image(s) 502 using a DEN 510 to output a depth distribution 512, a CFN 520 to output a feature vector 522, and a lifted transformation / 2D mapping 530 to obtain a projected feature tensor 542. The DEN 510 may be trained, for example, using depth data as described in conjunction with FIGS. 3A-C and / or 4A-B, or any other similar method. The BEV FN 550 may be a backbone network that processes the projected feature tensor 542 and generates intermediate outputs that are then provided to multiple classification heads 560-n. In some implementations, the classification head may include a semantic head 560-1 that outputs an instance segmentation map 562-1 containing classifications of various locations of the BEV, such as, for example, vehicles, vulnerable road users, roads, buildings, trees, roadside structures, and the like, among several types. The classification head 560-n may include one or more instance segmentation heads. For example, the instance center head 560-2 may output coordinates of the centers 562-2 (e.g., center-of-mass pixels) of various objects in the environment. The instance offset head 560-3 may output a map of offsets 560-3 characterizing the distances of various pixels of the object to the center of the object. The various maps 562-n may be further processed by one or more layers of neurons (not shown in FIG. 5) to obtain a detection and classification map (DCM) 564. The DCM 564 may combine the semantic segmentation and geometric (instance) segmentation generated by the classification head 560-n to determine the location of the object and further identify the class of the object. For example, a "vehicle" object of type identified by semantics head 560-1 may be further subdivided into classes such as "car", "pickup truck", "bus", "half-truck", etc., taking into account geometric information.Similarly, the type "vulnerable road user" may be further subdivided into classes such as "pedestrian," "bicycle," "motorcycle," and "skateboarder."

[0063] FIG. 6 is a schematic diagram illustrating operations 600 of a bird's-eye view model using temporal aggregation, according to some embodiments of the present disclosure. The operations 600 may include processing image(s) 602 using a DEN 610 to output a depth distribution 612, a CFN 620 to output a feature vector 622, and a lifted transform / 2D mapping 630 to obtain a projected feature tensor 642. The DEN 610 may be trained, for example, using depth data as described in conjunction with FIGS. 3A-C and / or 4A-B, or any other similar method. The images 602 may be associated with frames acquired at different times t1, t2, t3, .... The operations of blocks 610-630 may be performed separately for different frames, e.g., serially (using a single instance of the DEN 610 and the CFN 620) or in parallel (e.g., using multiple instances of the DEN 610 and the CFN 620). The projected feature tensor 642 may then be warped to a common reference time, e.g., the current time or the latest available time, which may be one of times t1, t2, t3..., using warping 644 (shown as time t3 in FIG. 6). Warping 644 may be a mathematical transformation that eliminates the (independently known) motion of the sensing system (e.g., the ego-motion of an autonomous vehicle) and generates warped feature tensor 645. As a result of warping 644, objects that are stationary relative to the ground are described by elements of warped feature tensor 645 associated with the same coordinates x, y, while moving objects are described by elements that extend along the direction of the object's motion. The warped feature tensor 645 may then be processed using aggregation network 646, which may be a convolutional network with a kernel that extends across two or more temporal components of warped feature tensor 645. The aggregation network 646 outputs an aggregation tensor 648 that may be processed by the BEV FN 650 (and further processed by various classification heads 660-n), similar to how the projected feature tensors are processed by the BEV feature network, for example, in the operation of Figures 3A-C or 4A-B.

[0064] 7-8 illustrate exemplary methods 700-800 for deploying machine learning models using a bird's-eye view and trained using depth ground truth data for efficient object detection and classification. A processing device having one or more processing units (CPUs) and a memory device communicatively coupled to the CPU(s) can perform methods 700-800 and / or each of their individual functions, routines, subroutines, or operations. A processing device executing methods 700-800 can execute instructions issued by various components of sensing system 110 or data processing system 120 of FIG. 1, such as ODCM 132. In some embodiments, methods 700-800 may be directed to systems and components of an autonomous vehicle, such as autonomous vehicle 100 of FIG. 1. Methods 700-800 can be used to improve the performance of processing system 120 and / or autonomous vehicle control system 140. In certain implementations, a single processing thread can perform methods 700-800. Alternatively, two or more processing threads may execute methods 700-800, with each thread executing one or more individual functions, routines, subroutines, or operations of the methods. In an exemplary embodiment, the processing threads executing methods 700-800 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads executing methods 700-800 may execute asynchronously with respect to each other. The various operations of methods 700-800 may be executed in an order different from that shown in FIG. 78. Some operations of methods 700-800 may be performed simultaneously with other operations. Some operations may be optional.

[0065] FIG. 7 illustrates an example method 700 for deploying a model trained using depth ground truth data for efficient object detection and classification using a bird's-eye view representation, according to some embodiments of the present disclosure. The method 700 may use real-time images acquired by one or more cameras on a vehicle or by cameras mounted on any other suitable application platform. The cameras may be optical range cameras and / or IR cameras, including panoramic (peripheral vision) cameras, partial panoramic cameras, high-definition (high-resolution) cameras, close-field cameras, cameras with fixed fields of view (relative to the vehicle), cameras with dynamic (adjustable) fields of view, cameras with fixed or adjustable focal lengths, cameras with fixed or adjustable numerical apertures, and any other suitable cameras. Optical range cameras may further include night vision cameras. The images acquired by the cameras may include various metadata providing geometric associations between image pixels and spatial locations of objects, correspondences between pixels in different images, etc. In some implementations, method 700 may be performed by ODCM 132 or any other similar model that may be part of the perception system of an autonomous vehicle, a vehicle incorporating driver assistance technology, or any other application platform that uses object detection and classification.

[0066] At block 710, method 700 may include obtaining one or more perspective camera images of an environment (e.g., image 302 of FIGS. 3A-C ). At block 720, method 700 may include using a first neural network to generate, for each pixel of a set of pixels in the one or more perspective camera images, a feature vector (FV) and a depth distribution of a portion of the environment imaged by the corresponding pixel. In some embodiments, the set of pixels may include all pixels in the one or more perspective camera images. In some embodiments, the set of pixels may include only a portion of all pixels in the one or more perspective camera images.

[0067] The first NN may include one or more sub-networks, including the encoder network 305, the DEN 310, the CFN 320, and / or other sub-networks. In some embodiments, a feature vector for each pixel of the set of pixels may be output by a first sub-network (e.g., the CFN 320) of the first NN. A depth distribution for each pixel of the set of pixels may be output by a second sub-network (e.g., the DEN 310) of the first NN. For example, the feature vector may be the FV 322 output by the CFN 320, and the depth distribution may be the depth distribution 312 output by the DEN 310. In some embodiments, the first NN may be trained using a plurality of training images and depth ground truth data for the plurality of training images. In some embodiments, the depth ground truth data may include lidar-determined ranges to one or more objects in at least a subset of the plurality of training images. In some embodiments, the second sub-network (e.g., DEN 310) may be trained using depth ground truth data prior to training of the first sub-network (e.g., CFN 320).

[0068] In some embodiments, the first NN may be part of a student model trained using a distillation technique, e.g., as described in conjunction with FIGS. 4A-B. More specifically, the depth ground truth data may include depth estimates for at least a subset of pixels in a plurality of training images. For example, a depth estimate may be available for each Nth pixel in the training images. The depth estimates (e.g., depth distribution 411) may be output by a first NN (e.g., training DEN 410-S) of a training model (e.g., training ODCM 401-T).

[0069] At block 730, the method 700 may include obtaining a feature tensor (FT) for each pixel of the set of pixels. The feature tensor (e.g., FT 432) may be obtained by considering a feature vector for each pixel and a depth distribution for each pixel. For example, the feature tensor may be obtained using a cross product of the feature vector and the depth distribution (for each pixel). Furthermore, obtaining the feature tensor may include performing a lift transform, for example, as described in connection with FIG. 3A.

[0070] At block 740, the method 700 may include processing the obtained feature tensors using a second NN to identify one or more objects in the environment. As indicated by the callouts in FIG. 7, processing the obtained feature tensors may include several operations. More specifically, at block 742, the method 700 may include using the feature tensors of each of the set of pixels to obtain a combined feature tensor (e.g., a combination FT 334). The method 700 may continue by mapping the combined feature tensor to a ground plane (or any other reference plane) to obtain a projected feature tensor (e.g., a projection FT 342). For example, mapping the combined feature tensor to the ground plane may include transforming the combined feature tensor to a set of coordinates associated with the ground plane (e.g., Cartesian coordinates x, y, z, cylindrical coordinates r, θ, z, or any other suitable set of coordinates) at block 744, and vertically aggregating elements of the combined feature tensor to obtain a projected feature tensor at block 746. For example, elements of the combined feature tensor associated with different values ​​of z (and the same values ​​of x, y or r, θ) may be added together to obtain the projected feature tensor.

[0071] At block 748, method 700 may continue using a second NN to process the projected feature tensor to identify one or more objects in the environment. In some implementations, the second NN may include a first classification head (e.g., semantic head 560-1 of FIG. 5) configured to output a semantic segmentation of one or more objects in the environment. The second NN may also include at least one second classification head (e.g., instance-centered head 560-2 and / or instance-offset head 560-3) configured to output geometric information associated with the position of one or more objects in the environment. In some embodiments, for example, if the NN is trained using a distillation framework, the second NN (e.g., student BEV FN 450-S) may be trained using the output of the second NN (e.g., teaching BEV FN 450-T) of the teaching training model (e.g., teaching ODCM 401-T).

[0072] In some embodiments, for example, as described in conjunction with FIG. 6 , a model (e.g., ODCM 132) can perform temporal aggregation of images acquired at different times. More specifically, one or more perspective camera images acquired in conjunction with block 710 may be associated with a first time (e.g., time t1). The temporal aggregation may include acquiring one or more additional perspective camera images associated with at least a second time (e.g., time t2, t3, etc.). The temporal aggregation may then include using the one or more additional perspective camera images to generate additional projected feature tensors and performing simultaneous processing of the projected feature tensors and the additional projected feature tensors (e.g., as exemplified by processing projected feature tensor 642). In some embodiments, the simultaneous processing may include application of a warping transform (e.g., warping 644) and an aggregation NN (e.g., aggregation network 646). The aggregation NN may include one or more convolution kernels configured to aggregate elements of a projected feature tensor with elements of additional projected feature tensors (or multiple additional feature tensors).

[0073] In those embodiments in which the perception system performing method 700 is deployed on a vehicle, method 700 may continue at block 750 with the perception system determining a driving path for the vehicle taking into account one or more identified objects in the vehicle's environment.

[0074] FIG. 8 illustrates an example method 800 for using depth ground truth data to train a model that develops bird's-eye view representations for efficient object detection and classification, according to some embodiments of the present disclosure. Method 800 can be used to train an ODCM 132 or any other similar model. Method 800 can use previously recorded images and other sensory data acquired by scanning the vehicle's (or any other relevant environment) environment using multiple sensors of the vehicle's sensing system, e.g., lidar, radar, sonar, etc. At block 810, method 800 may include acquiring training images, e.g., as part of a set of multiple training images. At block 815, method 800 may include processing the training images to generate multiple feature vectors and multiple depth distributions using a first NN of a student model. For example, the first NN of the student model may include some or both of student DEN 410-S and student CFN 420-S, as shown in FIGS. 4A-B . Each feature vector of the plurality of feature vectors (e.g., FV 422) and each depth distribution of the plurality of depth distributions (e.g., depth distributions 412) may be associated with a respective pixel of the plurality of pixels of the training images.

[0075] At block 820, method 800 may continue acquiring a plurality of ground truth feature vectors (e.g., feature vector 421) generated by the first NN of the training model. For example, the first NN of the training model may include some or both of the training DEN 410-T and the training CFN 420-T. Each ground truth feature vector of the plurality of ground truth feature vectors may be associated with a respective pixel of the plurality of pixels of the training image. At block 825, method 800 may continue acquiring a plurality of ground truth depth indicators. Each ground truth depth indicator of the plurality of ground truth depth indicators may be associated with a respective pixel of at least a subset of the plurality of pixels of the training image. In some implementations, each of the plurality of ground truth depth indicators may include a depth distribution (e.g., depth distribution 411 as shown in FIG. 4A ) acquired by the first NN of the training model for the associated pixel. In some implementations, each of the multiple ground truth depth indicators may include a particular distance obtained by a distance sensing device relative to a portion of the environment imaged by the associated pixel (e.g., depth data 404 as shown in FIG. 4B).

[0076] At block 830, the method 800 may include adjusting parameters of the first NN of the Student model based on a comparison of the plurality of feature vectors (e.g., feature vector 422) to a plurality of ground truth feature vectors (e.g., feature vector 422) and a comparison of the plurality of depth distributions (e.g., depth distribution 412) to a plurality of ground truth depth indicators.

[0077] At block 835, method 800 may include obtaining a plurality of feature tensors. Each feature tensor of the plurality of feature tensors may be obtained using a respective feature vector of the plurality of feature vectors and a respective depth distribution of the plurality of depth distributions (e.g., as described in conjunction with FIG. 3A ). At block 840, method 800 may continue using the plurality of feature tensors to obtain a combined feature tensor (e.g., combined feature tensor 334). At block 845, method 800 may include mapping the combined feature tensor to a ground plane (e.g., using lift transformation / 2D mapping 430) to obtain a projected feature tensor. At block 850, method 800 may continue processing the projected feature tensor using a second NN of the Student model (e.g., Student BEV FN 450-S) to identify one or more objects in the training images (e.g., detected object 452). At block 855, the method 800 may include obtaining one or more ground truth objects (e.g., detected objects 451) identified by the second NN of the teaching model (e.g., teaching BEV FN 450-T) in the training images. At block 860, the method 800 may continue with adjusting parameters of the second NN of the student model (e.g., student BEV FN 450-S) based on a comparison of the one or more objects identified by the second NN of the student model with one or more objects identified by the second NN of the teaching model.

[0078] FIG. 9 illustrates a block diagram of an exemplary computing device 900 capable of operating and / or training a model trained using depth ground truth data for efficient object detection and classification using a bird's-eye view, according to some embodiments of the present disclosure. An example computing device 900 may be connected to other computing devices within a LAN, an intranet, an extranet, and / or the Internet. The computing device 900 may operate in the capacity of a server in a client-server network environment. The computing device 900 may be a personal computer (PC), a set-top box (STB), a server, a network router, switch, or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Furthermore, while only a single example computing device is shown, the term "computer" shall also be considered to include any group of computers that, individually or jointly, execute a set (or sets) of instructions to perform any one or more of the methodologies discussed herein.

[0079] An embodiment of computing device 900 may include a processing unit 902 (also referred to as a processor or CPU), a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., data storage device 918), which may communicate with each other via a bus 930.

[0080] Processing unit 902 (which may include logic processing 903) represents one or more general-purpose processing units, such as a microprocessor, a central processing unit, or the like. More specifically, processing unit 902 may be a complex instruction set computer (CISC) microprocessor, a reduced instruction set computer (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor executing other instruction sets, or a processor executing a combination of instruction sets. Processing unit 902 may also be one or more special-purpose processing units, such as an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), a digital signal processor (DSP), a network processor, or the like. According to one or more aspects of the present disclosure, processing unit 902 may be configured to execute instructions to perform method 700 of using depth ground truth data to train a model for developing a bird's-eye view representation and / or method 800 of using depth ground truth data to train a model for developing a bird's-eye view representation for efficient object detection and classification.

[0081] An embodiment of computing device 900 may further include a network interface device 908, which may be communicatively coupled to a network 920. An embodiment of computing device 900 may further include a video display 910 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and an audio signal generating device 916 (e.g., a speaker).

[0082] Data storage device 918 may include a computer-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 928 having stored thereon one or more sets of executable instructions 922. According to one or more aspects of the present disclosure, executable instructions 922 may include executable instructions for performing method 700 of using depth ground truth data for training a model that develops bird's-eye view representations and / or method 800 of using depth ground truth data for training a model that develops bird's-eye view representations for efficient object detection and classification.

[0083] The executable instructions 922 may also reside, completely or at least partially, within the main memory 904 and / or within the processing unit 902 during execution thereof by the exemplary computing device 900, with the main memory 904 and the processing unit 902 also constituting computer-readable storage media. The executable instructions 922 may also be transmitted or received over a network via the network interface device 908.

[0084] While computer-readable storage medium 928 is illustrated in FIG. 9 as a single medium, the term "computer-readable storage medium" should be considered to include a single medium or multiple media (e.g., centralized or distributed databases, and / or associated caches and servers) that store one or more sets of operating instructions. The term "computer-readable storage medium" should also be considered to include any medium capable of storing or encoding a set of instructions for execution by a machine, causing the machine to perform any one or more of the methods described herein. Thus, the term "computer-readable storage medium" should be considered to include, but not be limited to, solid-state memory, and optical and magnetic media.

[0085] Some portions of the above detailed descriptions are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, understood to be a self-consistent sequence of steps leading to a desired result. The steps require physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0086] It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless otherwise indicated, and as will be apparent from the discussion that follows, throughout the description, discussions using terms such as "identify," "determine," "store," "adjust," "produce," "return," "compare," "generate," "stop," "load," "copy," "inject," "replace," "perform," or the like, will be understood to refer to the actions and processes of a computer system, or similar electronic computing device, that manipulate and transform data represented as physical (electronic) quantities in the computer system's registers and memory into other data similarly represented as physical quantities in the computer system's memory or registers or other such information storage, transmission, or display device.

[0087] Examples of the present disclosure also relate to an apparatus for performing the methods described herein. This apparatus may be specially constructed for the required purposes, or it may be a general-purpose computer system selectively programmed by a computer program stored within the computer system. Such a computer program may be stored on a computer-readable storage medium, such as, but not limited to, any type of disk, including optical disks, CD-ROMs, and magneto-optical disks, read-only memory (ROM), random-access memory (RAM), EPROM, EEPROM, magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0088] The methods and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct more specialized apparatus to perform the required method steps. The required structure for a variety of these systems appears as set forth in the description below. Additionally, the scope of the present disclosure is not limited to any particular programming language. It will be understood that a variety of programming languages ​​can be used to implement the teachings of the present disclosure.

[0089] It should be understood that the above description is intended to be illustrative, and not restrictive. Many other example embodiments will become apparent to those skilled in the art upon reading and understanding the above description. While the present disclosure describes particular examples, it will be recognized that the disclosed systems and methods are not limited to the examples described herein, but may be modified and practiced within the scope of the appended claims. Accordingly, the specification and drawings should be regarded in an illustrative, and not a restrictive, sense. The scope of the present disclosure should, therefore, be determined with reference to the appended claims, along with the full scope of equivalents to which such claims are entitled.

Claims

1. 1. A method comprising: acquiring one or more perspective camera images of an environment; using a first neural network (NN), for each pixel of the set of pixels of the one or more perspective camera images: a feature vector (FV); generating a depth distribution for a portion of the environment imaged by corresponding pixels, the first neural network being trained using a plurality of training images and depth ground truth data for the plurality of training images; For each pixel of the set of pixels, obtaining a feature tensor (FT) taking into account (i) the FV for the respective pixel, and (ii) the depth distribution for the respective pixel; and processing the obtained FT using a second NN to identify one or more objects in the environment.

2. processing the obtained FT, using the FTs for the set of pixels to obtain a combined FT; Mapping the combined FT onto a ground plane to obtain a projected FT; and processing the projected FT using the second NN.

3. Mapping the combined FT to the ground plane transforming the combined FT into a set of coordinates associated with the ground plane; and vertically aggregating elements of the combined FT to obtain the projected FT.

4. The one or more fluoroscopic camera images are associated with a first time, and the method comprises: acquiring one or more additional fluoroscopic camera images associated with at least a second time; generating additional projected FTs using the one or more additional perspective camera images; The method of claim 2 , further comprising: performing simultaneous processing of the projected FT and the additional projected FT.

5. 5. The method of claim 4, wherein the simultaneous processing is performed by an aggregation NN including one or more convolution kernels configured to aggregate elements of the projected FT with elements of the additional projected FT.

6. The second NN is a first classification head configured to output a semantic segmentation of the one or more objects in the environment; and at least one second classification head configured to output geometric information associated with the position of the one or more objects in the environment.

7. The method of claim 1 , wherein the depth ground truth data includes depth estimates for at least a subset of pixels of the plurality of training images, the depth estimates being output by a first neural network of a training model.

8. The method of claim 7 , wherein the second NN is trained using the output of the teaching model second NN.

9. 2. The method of claim 1, wherein the FT for each pixel of the set of pixels is output by a first sub-network of the first NN, and the depth distribution for each pixel of the set of pixels is output by a second sub-network of the first NN, the second sub-network being trained using the depth ground truth data prior to training of the first sub-network.

10. The method of claim 1 , wherein the depth ground truth data includes lidar-determined ranges to one or more objects in at least a subset of the plurality of training images.

11. 1. A method for training a student model, comprising: acquiring training images; processing the training images using a first neural network (NN) of the student model; a plurality of feature vectors (FVs); generating a plurality of depth distributions, each FV of the plurality of FVs and each depth distribution of the plurality of depth distributions being associated with a respective pixel of the plurality of pixels of the training images; Obtaining a plurality of ground truth FVs generated by a first neural network of a training model, wherein each ground truth FV of the plurality of ground truth FVs is associated with a respective pixel of the plurality of pixels of the training image; obtaining a plurality of ground truth depth indicators, each ground truth depth indicator of the plurality of ground truth depth indicators being associated with a respective pixel of at least a subset of the plurality of pixels of the training image; The parameters of the first neural network of the student model are comparing the plurality of FVs with the plurality of ground truth FVs; and adjusting based on a comparison of the plurality of depth distributions to the plurality of ground truth depth indicators.

12. obtaining a plurality of feature tensors (FTs), each FT of the plurality of FTs being obtained using a respective FV of the plurality of FVs and a respective depth distribution of the plurality of depth distributions; using the plurality of FTs to obtain a combined FT; Mapping the combined FT onto a ground plane to obtain a projected FT; processing the projected FT using a second NN of the student model to identify one or more objects in the training images; Obtaining one or more ground truth objects identified by the second neural network of the teaching model in the training images; adjusting parameters of the second NN of the student model based on a comparison of the one or more objects identified by the second NN of the student model with the one or more objects identified by the second NN of the teaching model; The method of claim 11 further comprising:

13. 12. The method of claim 11 , wherein each of the plurality of ground truth depth indicators comprises at least one of: (i) a depth distribution obtained by the first neural network of the teaching model for the associated pixel; or (ii) a distance to a portion of an environment imaged by the associated pixel, obtained by a range sensing device.

14. 1. A system comprising: Memory and a processing unit communicatively coupled to the memory, acquiring one or more perspective camera images of an environment; using a first neural network (NN), for each pixel of the set of pixels of the one or more perspective camera images: a feature vector (FV); generating a depth distribution for a portion of the environment imaged by corresponding pixels, the first neural network being trained using a plurality of training images and depth ground truth data for the plurality of training images; For each pixel of the set of pixels, obtaining a feature tensor (FT) taking into account (i) the FV for the respective pixel, and (ii) the depth distribution for the respective pixel; a processing unit configured to process the obtained FT using a second NN to identify one or more objects in the environment; and A system comprising:

15. To process the obtained FT, the processing device using the FTs for the set of pixels to obtain a combined FT; Mapping the combined FT onto a ground plane to obtain a projected FT; and processing the projected FT using the second NN.

16. To map the combined FT to the ground plane, the processing unit: transforming the combined FT into a set of coordinates associated with the ground plane; and vertically aggregating elements of the combined FT to obtain the projected FT.

17. The one or more fluoroscopic camera images are associated with a first time, and the processing device: acquiring one or more additional fluoroscopic camera images associated with at least a second time; generating additional projected FTs using the one or more additional perspective camera images; 16. The system of claim 15, further comprising: performing simultaneous processing of the projected FT and the additional projected FT, the simultaneous processing being performed by an aggregation NN including one or more convolution kernels configured to aggregate elements of the projected FT with elements of the additional projected FT.

18. The second NN is a first classification head configured to output a semantic segmentation of the one or more objects in the environment; and at least one second classification head configured to output geometric information associated with the position of the one or more objects in the environment.

19. 15. The system of claim 14, wherein the FT for each pixel of the set of pixels is output by a first sub-network of the first NN, and the depth distribution for each pixel of the set of pixels is output by a second sub-network of the first NN, the second sub-network being trained using the depth ground truth data prior to training of the first sub-network.

20. The system of claim 14 , wherein the depth ground truth data includes lidar-determined ranges to one or more objects in at least a subset of the plurality of training images.

Images