Object recognition in bird's-eye view reference frame with explicit depth estimation co-training

Abstract

The described aspects and implementations enable efficient detection and classification of objects using machine learning models that deploy bird's-eye view representations and are trained using depth fiducial truth data. In one embodiment, systems and techniques are disclosed that include obtaining an image, generating a feature vector (FV) and depth distribution pixels for the image using a first neural network (NN) trained using a training image and depth reference truth data for the training image. The technique further includes obtaining a feature tensor (FT) in view of the FV and the depth distribution, and processing the obtained FT using a second NN to identify one or more objects depicted in the image.

Description

Technical Field

[0001] The present specification generally relates to systems and applications for detecting and classifying objects, and more particularly to autonomous vehicles and vehicles deploying driver assistance technologies. More specifically, the present specification relates to processing of perspective camera images using machine learning techniques for faster and more resource-efficient detection and classification of objects, including but not limited to vehicles, pedestrians, bicyclists, animals, etc. Background Art

[0002] Autonomous (fully or partially autonomous) vehicles (AVs) operate by sensing the external environment using various electromagnetic (e.g., radar and optical) and non-electromagnetic (e.g., audio and humidity) sensors. Some autonomous vehicles plot a driving path through the environment based on the sensed data. The driving path can be determined based on global navigation satellite system (GNSS) data and road map data. While GNSS and road map data can provide information about static aspects of the environment (buildings, street layout, road closures, etc.), dynamic information (such as information about other vehicles, pedestrians, street lights, etc.) is obtained from the sensed data collected at the same time. The accuracy and safety of the driving path and speed mechanism selected by the autonomous vehicle depends on the timely and accurate identification of the various objects present in the driving environment and the ability of the driving algorithm to process information about the environment and provide correct instructions to the vehicle controller and transmission system. BRIEF DESCRIPTION OF THE DRAWINGS

[0003] The present disclosure is illustrated by way of example and not limitation and may be more fully understood with reference to the following detailed description when considered in conjunction with the accompanying drawings, in which:

[0004] Figure 1 is a diagram showing components of an example autonomous vehicle (AV) that deploys a model that uses a bird's eye view and is trained using deep ground truth data for efficient detection and classification of objects in accordance with some embodiments of the present disclosure.

[0005] Figure 2 is a diagram illustrating an example architecture of portions of a perception system capable of efficiently detecting and classifying objects according to some embodiments of the present disclosure.

[0006] Figure 3A is a schematic diagram illustrating example operations of a model using a bird's eye view and trained using depth ground truth data for efficient detection and classification of objects according to some embodiments of the present disclosure.

[0007] Figure 3B is a schematic diagram of one embodiment of a model that uses a bird's eye view and is trained using depth ground truth data for efficient detection and classification of objects.

[0008] Figure 3C A model in which a depth estimation network is pre-trained using depth ground truth data is shown according to some embodiments of the present disclosure.

[0009] Figure 4A is a schematic diagram illustrating a distillation framework for training a bird's eye view model using depth ground truth data according to some embodiments of the present disclosure.

[0010] Figure 4B is a schematic diagram illustrating another embodiment of a distillation framework for training a bird's eye view model using depth ground truth data according to some embodiments of the present disclosure.

[0011] Figure 5 is a schematic diagram illustrating the operation of a bird's eye view model providing instance segmentation and semantic segmentation according to some embodiments of the present disclosure.

[0012] Figure 6 is a schematic diagram illustrating the operation of a bird's eye view model using time aggregation according to some embodiments of the present disclosure.

[0013] Figure 7 An example method of deploying a model using a bird's-eye view representation and trained using deep ground truth data for efficient detection and classification of objects is shown in accordance with some embodiments of the present disclosure.

[0014] Figure 8 An example method of using deep ground truth data to train a model that deploys a bird's-eye view representation for efficient detection and classification of objects is shown in accordance with some embodiments of the present disclosure.

[0015] Fig. 9 Depicted is a block diagram of an example computer device capable of operating and / or training a model that uses a bird's eye view and is trained using deep ground truth data for efficient detection and classification of objects in accordance with some embodiments of the present disclosure. Summary of the invention

[0016] In one embodiment, a method is disclosed that includes obtaining one or more perspective camera images of an environment, and generating, using a first neural network (NN), for each pixel in a set of pixels of the one or more perspective camera images, a feature vector (FV) and a depth distribution of a portion of the environment imaged by the corresponding pixel. The first NN is trained using a plurality of training images and depth ground truth data for the plurality of training images. The method also includes, for each pixel in the set of pixels, obtaining a feature tensor (FT) given (i) the FV of the corresponding pixel and (ii) the depth distribution of the corresponding pixel. The method also includes processing the obtained FT using a second NN to identify one or more objects in the environment.

[0017] In another embodiment, a method for training a student model is disclosed, the method comprising obtaining a training image and processing the training image using a first NN of the student model to generate a plurality of FVs and a plurality of depth distributions. Each of the plurality of FVs and each of the plurality of depth distributions is associated with a corresponding pixel in a plurality of pixels of the training image. The method also comprises obtaining a plurality of baseline truth FVs generated by the first NN of the teacher model, each of the plurality of baseline truth FVs being associated with a corresponding pixel in a plurality of pixels of the training image. The method also comprises obtaining a plurality of baseline truth depth indicators, each of the plurality of baseline truth depth indicators being associated with a corresponding pixel in at least a subset of a plurality of pixels of the training image. The method also comprises adjusting parameters of the first NN of the student model. The adjustment is based on a comparison of the plurality of FVs with the plurality of baseline truth FVs, and further based on a comparison of the plurality of depth distributions with the plurality of baseline truth depth indicators.

[0018] In another embodiment, a system including a memory and a processing device is disclosed. The processing device is configured to obtain one or more perspective camera images of an environment, and to generate, using a first NN, for each pixel in a set of pixels of the one or more perspective camera images, a FV and a depth distribution of a portion of the environment imaged by the corresponding pixel. The first NN is trained using a plurality of training images and depth ground truth data for the plurality of training images. The processing device is further configured to obtain, for each pixel in the set of pixels, a FT in view of (i) the FV of the corresponding pixel and (ii) the depth distribution of the corresponding pixel. The processing device is further configured to process the obtained FT using a second NN to identify one or more objects in the environment. DETAILED DESCRIPTION

[0019] Although various embodiments may be described below using autonomous driving systems and driver assistance systems as examples for purposes of illustration, it should be understood that the techniques and systems described herein can be used to track objects in a wide range of applications, including aviation applications, marine applications, traffic control, animal control, industrial and academic research, public and personal safety, or any other application in which automatic detection of objects is advantageous.

[0020] In one example, for the safety of autonomous driving operations, it may be desirable to develop and deploy technologies for rapid and accurate detection, classification and tracking of various road users and other objects (such as road obstacles, construction equipment, roadside structures, etc.) encountered on or near the road. Autonomous vehicles (and various driver assistance systems) can utilize multiple sensors to facilitate the detection of objects in the driving environment and determine the movement of such objects. Sensors typically include radio detection and ranging sensors (radar), light detection and ranging sensors (lidar), various types of digital cameras, sonar, positioning sensors, etc. Different types of sensors provide different and usually complementary benefits. For example, radar and lidar emit electromagnetic signals (radio signals or light signals) that are reflected from objects and carry information that allows determination of the distance to the object (e.g., from the flight time of the signal) and the speed of the object (e.g., from the Doppler shift of the frequency of the signal). Radar and lidar can cover a full 360-degree view, for example, by using a scanning transmitter of a sensing beam. The sensing beam can cause many reflections of the driving environment in a dense grid covering the return point. Each return point may be associated with a distance to the corresponding reflecting object and the radial velocity (the component of the velocity along the line of sight) of the reflecting object.

[0021] Existing systems and methods for object recognition and tracking use various sensing modalities (e.g., lidar, radar, camera, etc.) to obtain images of the environment. The images can then be processed by a trained machine learning model to identify the locations of various objects in the image (e.g., in the form of bounding boxes), the motion state of the objects (e.g., the speed detected by lidar or radar Doppler effect-based sensors), the type of object (e.g., vehicle or pedestrian), etc. The motion of the object (or any other evolution, such as splitting a single object into multiple objects) can be performed by creating and maintaining tracks associated with specific objects.

[0022] The use of multiple sensing modalities (e.g., lidar, radar, camera, etc.) to obtain generally complementary data improves the accuracy of object detection, recognition and tracking, but brings considerable costs in sensing hardware and processing software. For example, lidar sensors can provide valuable information about the distance to various reflective surfaces in the external environment. However, lidar sensors are expensive optical and electronic devices that operate by actively detecting the external environment using light signals and require a lot of maintenance and periodic calibration. The lidar return (point cloud) must be processed, segmented into groups associated with separate hypothetical objects, and matched with objects detected using other sensing modalities (e.g., camera), which requires additional processing and memory resources. On the other hand, the camera operates by passively collecting light (and / or infrared electromagnetic waves) emitted (or reflected) by objects in the environment, and is significantly simpler and cheaper in design, installation and operation. Therefore, various driver assistance systems that do not deploy lidar (for cost and maintenance reasons) are usually equipped with one or more cameras. Cameras can also be more easily installed in various fixed locations and are used for traffic monitoring and control, public and private safety applications, etc. Cameras have certain advantages over radar, which, while allowing the distance of an object (and the object's velocity) to be detected, operates in a wavelength range that has an inherently lower resolution than cameras. Therefore, the ability to detect and identify objects based on camera images alone is beneficial.

[0023] However, the camera produces a projection of the three-dimensional (3D) external environment onto a two-dimensional imaging surface (e.g., the camera's photodetector array), which can be a plane or a curved surface. This causes two related challenges. On the one hand, the distance to the object (usually referred to as the depth of the object in the image) is not immediately known (but can usually be determined based on the context of the imaged object). On the other hand, the camera image has perspective distortion, resulting in the same number of pixels of the separated object image corresponding to different distances between the objects, depending on the depth of the object. In addition, objects depicted close to each other can still be separated by a significant distance (e.g., a car and a pedestrian visible behind the car). Existing machine learning techniques for object detection sometimes attempt to map objects from a perspective view to a top view, also known as a bird's-eye view (BEV), where the object is represented on a convenient manifold, for example, a plane viewed from above and characterized by a set of simple Cartesian coordinates. Object recognition and tracking can then be performed directly within the BEV representation. The success of such a technique depends on accurate mapping of objects to the BEV. This in turn requires accurate estimates of the distances to various objects, because the misalignment of objects within the BEV may not only lead to errors in determining the distance to the road user, but may also lead to the loss of important contextual information.

[0024] Aspects and embodiments of the present disclosure address these and other challenges of the prior art by enabling methods and systems for depth-informed training of machine learning object detection, classification, and tracking models. In particular, the disclosed technology includes a system of neural networks (NNs) trained to process perspective camera images. A first NN can be configured to associate context with various pixels (e.g., in the form of feature vectors) and further estimate the probability that the pixel has multiple discrete depth values. As described in more detail below, the first neural network can be trained using training camera images annotated with ground truth values, which ground truth values ​​include depth data for at least some pixels of the image. The depth data can be obtained by a suitable sensor capable of detecting the distance of an object (e.g., a lidar, radar, sonar, or any other distance-aware sensor). During inference, the trained first NN can process a collection of new perspective camera images and can estimate the depth to various pixels, for example, as the probability P(d) that a given pixel depicts an object (or a portion of an object) located at a distance d from the camera. The distance distribution can be combined with the feature vector FV(c) of the pixel (where c is a context space index) to obtain a feature tensor FT(c, d) that characterizes both the possible position (depth or distance d to the object) and the context of the object. The feature tensors of various pixels with coordinates w,h in the image can be used to obtain the combined feature tensor {FT(c,d) w,h}→CFT(c,d,w,h). A mapping transformation d,w,h→x,y,z from the perspective coordinates w,h,d (width, height, depth) to the set of Cartesian coordinates x,y,z can then be performed. The combined feature tensor CFT(c,x,y,z) in the new coordinates can then be projected onto a horizontal surface, for example, by averaging or summing the elements of the combined feature tensor along a vertical column of pixels to obtain a BEV-projected context tensor PCT(c,x,y)=∑ i CFT(c,x,y,z i ). The projected context tensor PCT(c,x,y) can then be processed by a second trained NN that recognizes the object, extracts semantic information (e.g., the type of the object), the precise location of the object, etc.

[0025] Many variations of these techniques are described herein. In some embodiments, the first NN may include a first subnetwork trained to output a depth distribution P(d) and a separate second NN trained to output a feature vector FV(c). In some embodiments, the first NN and the second NN may be trained simultaneously. In such embodiments, the first NN and the second NN are subnetworks of an end-to-end model architecture that is trained jointly. In some embodiments, the first NN may be first trained using both training images and depth ground truth, followed by training the second NN using the training images instead of the depth ground truth. Any number of images taken at the same time (or approximately the same time) may be processed simultaneously, for example, multiple images taken by a surround view camera (SVC) during a single cycle of the camera. In some embodiments, images taken at different times may be processed simultaneously. For example, at a particular time t j The captured image can be used to generate the corresponding BEV context tensor PCT (c, x, y; t j ). Multiple BEV context tensors can then be processed at once by the second NN. Multiple depictions of the same object at the same or different locations can increase the likelihood of correctly segmenting and identifying objects in the image. In some embodiments, the second NN can have a common backbone and multiple classification heads. For example, one classification head can be trained to output a semantic segmentation of an input image. The second classification head can be trained to output the geometric center of various objects. The third classification head can be trained to output the distances from various pixels in the BEV representation to the geometric center of the object. The combined output of the classification heads can be used to provide identification of the boundaries of the object and classification (type, category) of the object. In some embodiments, the training of the first NN and / or the second NN can be performed using a teacher-student distillation framework. More specifically, the output of a teacher model trained with data obtained using multiple sensing modalities (cameras, lidars, radars, sonars, etc.) can be used as a baseline truth value for the NN that trains the student model. The student model can be a distillation of a teacher model, for example, a model with a reduced number of neuron layers and / or neurons within a particular layer. As a result, the student model can maintain the substantial functionality of the teacher model while being more easily deployed on vehicles with less efficient processing and memory resources.

[0026] The described embodiments depart from the conventional object detection and classification paradigm in a complementary BEV segmentation technique that uses deep ground truth data for efficient co-training of models. Advantages of the described embodiments include, but are not limited to, rapid and accurate detection, identification, and tracking of objects in a manner that avoids the large computational overhead of processing data from multiple sensing modalities. Because the trained and deployed machine learning models as disclosed herein are capable of efficient object detection based on camera images, the models can be deployed on a variety of platforms (e.g., autonomous vehicles and vehicles equipped with driver assistance technologies), including systems with modest computational resources.

[0027] Figure 1 is a diagram showing components of an example autonomous vehicle (AV) 100 that deploys a model that uses a bird's eye view and is trained using deep ground truth data for efficient detection and classification of objects in accordance with some embodiments of the present disclosure. Autonomous vehicles may include motor vehicles (cars, trucks, buses, motorcycles, all-terrain vehicles, recreational vehicles, any specialized agricultural or construction vehicles, etc.), aircraft (planes, helicopters, drones, etc.), naval vehicles (ships, boats, yachts, submarines, etc.), spacecraft (controllable objects operating outside the Earth's atmosphere), or any other self-propelled vehicle (e.g., a robot, a factory or warehouse robotic vehicle, a sidewalk delivery robotic vehicle, etc.) that is capable of operating in an autonomous driving mode (without or with reduced human input).

[0028] Vehicles such as those described herein may be configured to operate in one or more different driving modes. For example, in a manual driving mode, the driver may directly control acceleration, deceleration, and steering via inputs such as an accelerator pedal, a brake pedal, a steering wheel, and the like. Vehicles may also operate in one or more autonomous driving modes, including, for example, semi-autonomous or partially autonomous driving modes, in which a person has a certain amount of direct or remote control over the driving operations, or fully autonomous driving modes, in which the vehicle handles the driving operations without direct or remote control by a person. These vehicles may be known by different names, including, for example, autonomous driving vehicles, self-driving vehicles, and the like.

[0029] As described herein, in semi-autonomous or partially autonomous driving modes, even if the vehicle assists with one or more driving operations (e.g., steering, braking, and / or accelerating to perform lane centering, adaptive cruise control, advanced driver assistance systems (ADAS), or emergency braking), the human driver is expected to be situationally aware of the vehicle's surroundings and supervise the assisted driving operations. Here, even if the vehicle can perform all driving tasks in certain situations, the human driver is expected to be responsible for taking control as needed.

[0030] Although various systems and methods may be described below in conjunction with autonomous vehicles for the sake of brevity, similar techniques may be used for various driver assistance systems that do not rise to the level of fully autonomous driving systems. In the United States, the Society of Automotive Engineers (SAE) has defined different levels of autonomous driving operation to indicate the degree or extent to which a vehicle controls driving, although different organizations in the United States or other countries may classify the levels differently. More specifically, the disclosed systems and methods may be used for SAE Level 2 driver assistance systems that implement steering, braking, acceleration, lane centering, adaptive cruise control, etc., as well as other driver support. The disclosed systems and methods may be used for SAE Level 3 driver assistance systems that are capable of autonomous driving under limited (e.g., highway) conditions. Similarly, the disclosed systems and methods may be used in vehicles using SAE Level 4 autonomous driving systems that operate autonomously in most routine driving situations and require only occasional attention from a human operator. In all such driver assistance systems, accurate lane estimation can be performed automatically without driver input or control (e.g., when the vehicle is in motion) and results in improved reliability of vehicle positioning and navigation and improved overall safety of autonomous, semi-autonomous, and other driver assistance systems. As previously mentioned, in addition to the way that SAE classifies the levels of automated driving operation, other organizations in the United States or other countries may classify the levels of automated driving operation differently. Without limitation, the systems and methods disclosed herein may be used for driver assistance systems defined by the levels of automated driving operation of these other organizations.

[0031] The driving environment 101 may include any object (animate or inanimate) located outside the AV, such as roads, buildings, trees, bushes, sidewalks, bridges, mountains, other vehicles, pedestrians, docks, banks, landing strips, animals, birds, etc. The driving environment 101 may be a city, suburbs, rural areas, etc. In some embodiments, the driving environment 101 may be an off-road environment (e.g., agriculture or other agricultural land) outside the road. In some embodiments, the driving environment may be an indoor environment, such as an environment of an industrial plant, a shipping warehouse, a dangerous area of ​​a building, etc. In some embodiments, the driving environment 101 may be substantially flat, wherein various objects move parallel to the surface (e.g., parallel to the surface of the earth). In other embodiments, the driving environment may be three-dimensional and may include objects (e.g., balloons, fallen leaves, etc.) that can move in all three directions. Hereinafter, the term "driving environment" should be understood to include all environments in which autonomous motion (e.g., SAE Level 5 and SAE Level 4 systems), conditional autonomous motion (e.g., SAE Level 3 systems), and / or motion of a vehicle equipped with driver assistance technology (e.g., SAE Level 2 systems) may occur. Additionally, the "driving environment" may include any possible flight environment for an aircraft (or spacecraft) or a marine environment for a naval vessel. Objects of the driving environment 101 may be located at any distance from the AV, from a close distance of a few feet (or less) to several miles (or more).

[0032] The example AV 100 may include a sensing system 110. The sensing system 110 may include various electromagnetic (e.g., optical, infrared, radio waves, etc.) and non-electromagnetic (e.g., acoustic) sensing subsystems and / or devices. The sensing system 110 may include one or more lidars 112, which may be laser-based units capable of determining the distance to an object in the driving environment 101 and the speed of the object. The sensing system 110 may include one or more radars 114, which may be any system that utilizes radio or microwave frequency signals to sense objects within the driving environment 101 of the AV 100. The lidar 112 and / or radar 114 may be configured to sense the spatial position of an object (including their spatial dimensions) and the speed of the object (e.g., using Doppler shift techniques). Hereinafter, "speed" refers to the speed at which an object is moving (the speed of the object) and the direction in which the object is moving. Each of the lidar 112 and radar 114 may include a coherent sensor, such as a frequency modulated continuous wave (FMCW) lidar or radar sensor. For example, the lidar 112 and / or radar 114 may use heterodyne detection for speed determination. In some embodiments, the functionality of ToF and coherent lidar (or radar) are combined into a lidar (or radar) unit that is capable of simultaneously determining the distance to a reflecting object and the radial velocity of the reflecting object. Such a unit can be configured to operate in an incoherent sensing mode (ToF mode) and / or a coherent sensing mode (e.g., a mode using heterodyne detection) or in both modes simultaneously. In some embodiments, multiple lidars 112 and / or radars 114 can be mounted on the AV 100.

[0033] The lidar 112 (and / or radar 114) may include one or more light sources (and / or radio / microwave sources) that generate and transmit signals and one or more detectors of signals reflected from an object. In some embodiments, the lidar 112 and / or radar 114 may perform a 360-degree scan in the horizontal direction. In some embodiments, the lidar 112 and / or radar 114 may be capable of spatial scanning in both the horizontal and vertical directions. In some embodiments, the field of view may be up to 90 degrees in the vertical direction (e.g., scanning at least a portion of the area above the horizon using a lidar or radar signal). In some embodiments (e.g., aerospace applications), the field of view may be a complete sphere (consisting of two hemispheres).

[0034] The sensing system 110 may also include one or more cameras 118 to capture images of the driving environment 101. The cameras 118 may operate in the visible portion of the electromagnetic spectrum, for example, the 300-800 nm wavelength range (also referred to herein as the optical range for simplicity). Some optical range-finding cameras 118 may use a global shutter, while other cameras 118 may use a rolling shutter. The image may be a two-dimensional projection of the driving environment 101 (or a portion of the driving environment 101) onto a projection surface (flat or non-flat) of the camera. Some of the cameras 118 of the sensing system 110 may be video cameras configured to capture a continuous (or quasi-continuous) stream of images of the driving environment 101. The sensing system 110 may also include one or more sonars 116, such as ultrasonic sonars, for active sound detection of the driving environment 101, and one or more microphones for passively listening to the sounds of the driving environment 101. The sensing system 110 may also include one or more infrared range-finding cameras 119, also referred to herein as IR cameras 119. IR camera 119 may use focusing optics (e.g., made of germanium-based materials, silicon-based materials, etc.) configured to operate in a wavelength range from microns to tens of microns or more. IR camera 119 may include a phased array of IR detector elements. A pixel of an IR image produced by camera 119 may represent the total amount of IR radiation collected by a corresponding detector element (associated with the pixel), the temperature of a physical object whose IR radiation is collected by the corresponding detector element, or any other suitable physical quantity.

[0035] The sensing data obtained by the sensing system 110 may be processed by the data processing system 120 of the AV 100. For example, the data processing system 120 may include a perception system 130. The perception system 130 may be configured to detect and track objects in the driving environment 101 and to identify detected objects. For example, the perception system 130 may analyze images captured by the camera 118 and may be able to detect traffic light signals, road signs, road layout (e.g., boundaries of traffic lanes, topology of intersections, designation of parking places, etc.), the presence of obstacles, etc. The perception system 130 may also receive radar sensing data (Doppler data and ToF data) to determine the distance to various objects in the environment 101 and the speed (radial and, in some embodiments, lateral, as described below) of such objects. In some embodiments, the perception system 130 may use the radar data in conjunction with the data captured by the camera 118, as described in more detail below.

[0036] The perception system 130 may include one or more modules to facilitate efficient and reliable detection, identification, and tracking of objects, including an object detection and classification model with deep collaborative training (ODCM) 132, which may be used to process data provided by the sensing system 110. In some embodiments, during inference, the ODCM 132 may receive data from the (optical range) camera 118 and / or the IR camera 119. During training, such as in conjunction with Figure 2 and Figure 4A-4B As described in more detail, the ODCM 132 may process data from the camera 118 and / or the IR camera 119 while using range (distance) data obtained using at least some of the lidar 112, radar 114, sonar 116, etc. The ODCM 132 may include one or more trained machine learning models (MLMs) for processing received images to detect objects depicted in the images and to classify the detected objects.

[0037] The perception system 130 can also obtain information from the Global Navigation Satellite System (GNSS) positioning subsystem ( Figure 1 The GNSS positioning subsystem may include a GNSS transceiver (not shown) configured to obtain information about the position of the AV relative to the earth and its surroundings. The positioning subsystem may use positioning data (e.g., GNNS and inertial measurement unit (IMU) data) in conjunction with sensory data to help accurately determine the position of the AV relative to fixed objects (e.g., roads, lane boundaries, intersections, sidewalks, crosswalks, road signs, curbs, surrounding buildings, etc.) of the driving environment 101, the positions of which may be provided by the map information 124. In some embodiments, the data processing system 120 may receive non-electromagnetic data, such as audio data (e.g., ultrasonic sensor data from sonar 116 or data from a microphone that picks up an emergency vehicle sirens), temperature sensor data, humidity sensor data, pressure sensor data, meteorological data (e.g., wind speed and direction, precipitation data), etc.

[0038] The data processing system 120 may also include an environment monitoring and prediction component 126, which can monitor how the driving environment 101 evolves over time, for example, by tracking the position and speed of animate objects (e.g., relative to the earth). In some embodiments, the environment monitoring and prediction component 126 can track changes in the appearance of the environment due to the movement of the AV relative to the environment. In some embodiments, the environment monitoring and prediction component 126 can make predictions about how various animate objects of the driving environment 101 will be positioned within the predicted time range. The prediction can be based on the current state of the animate object, including the current position (coordinates) and speed of the animate object. In addition, the prediction can be based on the movement history (tracked dynamics) of the animate object during a specific time period before the current moment. For example, based on the stored data of the first object indicating the accelerated movement of the first object during the previous 3-second time period, the environment monitoring and prediction component 126 can conclude that the first object is resuming its movement from the stop sign or the red traffic light signal. Therefore, given the layout of the road and the presence of other vehicles, the environment monitoring and prediction component 126 can predict the position of the first object that may be in the next 3 seconds or 5 seconds of movement. As another example, based on stored data of a second object indicating decelerated motion of the second object during a previous 2 second time period, the environment monitoring and prediction component 126 can conclude that the second object is stopping at a stop sign or a red traffic light signal. Therefore, the environment monitoring and prediction component 126 can predict where the second object is likely to be in the next 1 second or 3 seconds. The environment monitoring and prediction component 126 can perform periodic checks on the accuracy of its predictions and modify the predictions based on new data obtained from the sensing system 110. The environment monitoring and prediction component 126 can operate in conjunction with the ODCM 132. For example, the environment monitoring and prediction component 126 can track the relative motion of the AV and various objects (e.g., reference objects that are stationary or moving relative to the earth).

[0039] The data generated by the perception system 130, the GNSS processing module 122, and the environmental monitoring and prediction component 126 can be used by an autonomous driving system such as an AV control system (AVCS) 140. The AVCS 140 may include one or more algorithms that control how the AV behaves in various driving situations and environments. For example, the AVCS 140 may include a navigation system for determining a global driving route to a destination point. The AVCS 140 may also include a driving path selection system for selecting a specific path through the instant driving environment, which may include selecting a traffic lane, negotiating traffic jams, selecting a location for a U-turn, selecting a trajectory for parking maneuvers, etc. The AVCS 140 may also include an obstacle avoidance system for safely avoiding various obstacles (rocks, stalled vehicles, etc.) within the driving environment of the AV. The obstacle avoidance system may be configured to evaluate the size of the obstacle and the trajectory of the obstacle (if the obstacle is alive), and select the best driving strategy for avoiding the obstacle (e.g., braking, steering, acceleration, etc.).

[0040] The algorithms and modules of the AVCS 140 may generate instructions for various systems and components of the vehicle, such as the powertrain, braking and steering 150, vehicle electronics 160, signaling 170, and Figure 1 Other systems and components not explicitly shown in the figure. The powertrain, brake and steering 150 may include an engine (internal combustion engine, electric motor, etc.), a transmission, a differential, an axle, a wheel, a steering mechanism, and other systems. The vehicle electronics 160 may include an onboard computer, engine management, an ignition, a communication system, a carputer, telematics, an in-car entertainment system, and other systems and components. The signal 170 may include high and low headlights, parking lights, turn signals and reverse lights, horns and alarms, interior lighting systems, dashboard notification systems, passenger notification systems, radio and wireless network transmission systems, etc. Some of the instructions output by AVCS140 can be delivered directly to the powertrain, brake and steering 150 (or signal 170), while other instructions output by AVCS140 are first delivered to the vehicle electronics 160, and the vehicle electronics 160 generates commands to the powertrain, brake and steering 150 and / or signal 170.

[0041] In one example, the ODCM 132 may determine that the image obtained by the camera 118 includes a depiction of an object, and may further classify the object as a bicyclist. The environmental monitoring and prediction component 126 may track the bicyclist and determine that the bicyclist is traveling at a speed of 15 mph along an intersection perpendicular to the direction of motion of the vehicle. In response to such a determination, the data processing system 120 may determine that the vehicle needs to slow down to allow the bicyclist to pass through the intersection. The AVCS 140 may output instructions to the powertrain, braking and steering 150 (directly or via the vehicle electronics 160) to: (1) reduce fuel flow to the engine by modifying the throttle setting to reduce the engine engine rpm; (2) downshift the driveline to a lower gear via the automatic transmission; and (3) engage the brake unit to reduce the speed of the vehicle (acting in concert with the engine and transmission). After the ODCM 132 and / or the environmental monitoring and prediction component 126 determine that the bicyclist has crossed the intersection, the AVCS 140 may output commands to the powertrain, braking, and steering 150 to restore the vehicle's previous speed setting.

[0042] Figure 2 is a diagram illustrating an example architecture 200 of a portion of a perception system capable of efficiently detecting and classifying objects according to some embodiments of the present disclosure. Figure 1The input of the perception system 130 of the autonomous vehicle may include a plurality of camera images 202, which may be training images (during the training phase) or images taken on the vehicle at runtime (during the inference phase). The images 202 may be combined into frames. A frame should be understood as a collection of any images depicting the external environment along any direction relative to the sensing system (e.g., the sensing system 110 of the autonomous vehicle). In particular, the camera images 202 may refer to panoramic images taken by a surround camera, images taken by a directional camera (e.g., a forward-looking camera, a side-view camera (SVC), a rear-view camera, etc.), or any combination thereof. In some embodiments, the images obtained by different cameras may be synchronized so that all images in a given frame have the same (to the accuracy of synchronization) timestamp. In some embodiments, some images in a given frame may have a (controlled) time offset, for example, a time offset associated with the scanning motion of the SVC. The camera images 202 may be processed by the ODCM 132. Additional inputs to the ODCM 132 may include depth data 204, which may be data obtained using a lidar sensor, a radar sensor, a sonar sensor, etc. In some embodiments, the depth data 204 may be used during the training of the ODCM 132 rather than during the inference phase. The image 202 and the depth data 204 may include a direction index. More specifically, the various pixels of the image 202 and the returned points of the depth data 204 may be associated with known (e.g., from camera calibration) directions in space. For example, a camera image may include an intensity map indexed by any suitable set of coordinates representing directions in space, such as an intensity map indexed by I(w,h), where w,h may be Cartesian pixel coordinates (in the imaging plane); I(θ,φ), where θ and φ are polar and azimuthal angles, respectively; or any other set of coordinates. In some embodiments, multiple sets of coordinates may be used for different tasks, facilitated by mappings (transformations) between different sets stored. The depth data 204 may include a radial distance R(w,h) to an object associated with a particular pixel w,h in the image 202, for example, as determined from the ToF of a lidar / radar / sonar signal.

[0043] Each image 202 may have one color I(w,h) (for a black and white image) or multiple colors I C(w,h) (for color images). At least some of the images 202 may be infrared (IR) camera images obtained by an array of IR detectors (pixels), which may operate in a wavelength range from microns to tens of microns or more. The IR images may include intensities IR(w,h) representing the total amount of IR radiation collected by the corresponding detectors. In some embodiments, the IR images may include a pseudo-color map, wherein the presence of a particular pseudo-color may represent the total intensity IR(w,h) collected. In some embodiments, the collected intensities may be used to determine a temperature map T(w,h) of the environment. Thus, in different embodiments, different representations (e.g., intensity maps, pseudo-color maps, temperature maps, etc.) may be used to represent the IR camera data.

[0044] In some embodiments, architecture 200 may include a normalization module ( Figure 2 ), which can adjust the size of each image 202 to match the size of the input to the ODCM 132. In some embodiments, the normalization module can further normalize the intensity of the pixels of the image 202 to a preset intensity range, for example, [I min ,I max ], where I min is the minimum intensity, I max is the maximum intensity that the ODCM 132 is configured to handle. In some embodiments, the minimum intensity may be zero, and I min = 0. In addition, the normalization module can perform other pre-processing of the image 202, including filtering, denoising, etc.

[0045] The normalized and pre-processed images may be processed by various components of the ODCM 132 to detect the presence of objects 232 in the driving environment and to classify the detected objects 232. The ODCM 132 may include any suitable machine learning model, such as a lookup table, a geometric shape mapping, a mathematical formula, a decision tree algorithm, a support vector machine, a deep neural network, or the like, or any combination thereof. The deep neural network may include a convolutional neural network, a recurrent neural network (RNN), a fully connected neural network, a fully connected neural network, a long short-term memory neural network, a Boltzmann machine, or the like, or any combination thereof.

[0046] Depth estimation network (DEN) 210 may output a prediction of the depth of an object imaged by various pixels of image 202. Context feature network (CFN) 220 may output a feature vector for each pixel. A feature vector may be a multi-element data string in a feature space. The depth prediction and feature vector may be combined into a feature tensor. The feature tensor may undergo one or more transformations, such as combining Figure 3A-3CAs described in more detail, the feature tensors combined into a projected BEV context tensor can be processed by a BEV feature network 230 that outputs detected / classified objects 232, which can be classified into multiple categories, such as cars, trucks, buses, pedestrians, unknown objects, etc.

[0047] The detected / classified objects 232 may undergo post-processing 234, which may include object tracking that tracks the motion of the detected objects across multiple frames of the image 202. Each object may be assigned a detection track and may be characterized by some or all of the following: a bounding box that depicts the particular object across multiple frames, the type of the object, the size of the object, the pose of the object, the motion of the object (e.g., velocity, acceleration, etc.), etc. The post-processing 234 may also include generating any graphical representation of the track, such as a pixel-based (e.g., heat map) or vectorized (curve-based) representation, including the tracks, poses, velocity ranges, etc. of various objects. In some embodiments, the post-processing 234 may include processing the detected track using one or more models that predict the motion of the detected objects (e.g., models that track the velocity, acceleration, etc. of the detected objects). For example, a Kalman filter or any other suitable filter that combines the predicted motion of a particular object with the detected object motion may be used to more accurately estimate the position and motion of the object.

[0048] The detected / classified objects 232 and the tracking data generated by the post-processing 234 can be provided to the AVCS 140. The AVCS 140 evaluates the trajectories of the objects in the various trajectories and determines whether to modify the current driving trajectory of the vehicle in view of the position and velocity of the tracked objects. For example, if a tracked pedestrian or bicyclist is within a certain distance from the vehicle, the AVCS 140 can slow the vehicle to a speed that ensures that the pedestrian or bicyclist can be safely avoided. Alternatively, the AVCS 140 can change lanes, for example, if the adjacent lane is clear of obstacles, or perform some other driving maneuver.

[0049] The training may be performed by a training engine 242 hosted by a training server 240, which may be an external server that deploys one or more processing devices (e.g., a central processing unit (CPU), a graphics processing unit (GPU), etc.). In some embodiments, the ODCM 132 may be trained by the training engine 242 and then downloaded to a vehicle where the perception system 130 is deployed. Figure 2As shown, the ODCM 132 may be trained using training data including training inputs 244 and corresponding target outputs 246 (correct matches of the respective training inputs). During training of the ODCM 132, the training engine 242 may find patterns in the training data that map each training input 244 to a corresponding target output 246.

[0050] In some embodiments, the ODCM 132 can be trained using images and other sensory data that have been recorded during the driving task and annotated with ground truth. To train the depth estimation network 210 of the ODCM 132, the ground truth may include distances to various pixels of the images in the training input 244. Training of the BEV feature network 230 may involve ground truth that includes correct identification of the locations (e.g., bounding boxes) of various objects in the training input 244 and semantic (e.g., category, type, etc.) information of the objects. In some embodiments, the ground truth used to train any or all of the depth estimation network 210, the context feature network 220, and the BEV feature network 230 may include the output of one or more teacher models, as described below in conjunction with Figure 4A-4B The ground truth may include the correct linkage of the same object across multiple batches of images / frames, the correct velocity of the object, etc.

[0051] The training engine 242 may access a data repository 250 storing a plurality of camera / IR camera images 252 and lidar / radar (or sonar) images 254 obtained during driving situations in various driving environments (e.g., city driving tasks, highway driving tasks, rural driving tasks, etc.). During training, the training engine 242 may (e.g., randomly) select a set of the plurality of camera / IR camera images 252 and a set of the plurality of lidar / radar images 254 as training data. The training data may be annotated with correct object recognition. In some embodiments, the annotations may be made by a developer before the annotated training data is placed in the data repository 250. The annotated training data retrieved from the data repository 250 by the training server 240 may include one or more training inputs 244 and one or more target outputs 246. The training data may also include mapping data 248 that maps the training inputs 244 to the target outputs 246. For example, the mapping data 248 may identify a bounding box of a passenger car in each of a batch of N consecutive frames obtained by a forward-facing camera of the vehicle. Mapping data 248 may include identifiers of the training data, locations of buses, sizes and identifications of buses, speeds and directions of movement of buses, and other suitable information.

[0052] During the training of the ODCM 132, the training engine 242 may use a suitable loss function 245 to evaluate the difference between the output of the ODCM 132 (or the various networks and sub-networks of the ODCM 132) and the target output 246. In some embodiments, different loss functions 245 may be used for the training of the depth estimation network 210, the contextual feature network 220, and / or the BEV feature network 230. During the training of the ODCM 132, the training engine 242 may change the parameters (e.g., weights and biases) of the various networks and sub-networks of the ODCM 132 until the model successfully learns how to minimize the loss function 245 and successfully recognizes and classifies the target output 246, such as various objects in the external environment. In some embodiments, the various networks and sub-networks of the ODCM 132 may be trained separately, for example, the depth estimation network 210 is trained before the contextual feature network 220 and / or the BEV feature network 230. In some embodiments, the various networks and sub-networks of the ODCM 132 may be trained together (e.g., simultaneously). For example, the depth estimation network 210, the contextual feature network 220, and / or the BEV feature network 230 may be subnetworks of a single end-to-end neural network architecture that is trained jointly.

[0053] According to an embodiment of the present disclosure, the data repository 250 may be a persistent storage device capable of storing camera / IR camera images, lidar / radar / sonar data, and data structures configured to facilitate detection and recognition. The data repository 250 may be hosted by one or more storage devices, such as main memory, magnetic or optical storage disks, tapes or hard drives, network-attached storage (NAS), storage area network (SAN), etc. Although depicted as being separate from the training server 240, in an embodiment, the data repository 250 may be part of the training server 240. In some embodiments, the data repository 250 may be a network-attached file server, while in other embodiments, the data repository 250 may be some other type of persistent storage device, such as an object-oriented database, a relational database, etc., which may be accessed by a server machine or the training server 240 via a network ( Figure 2 The system may be hosted on one or more different machines accessible by the system (not shown).

[0054] Figure 3A is a schematic diagram illustrating example operations of a model 300 using a bird's eye view and trained using depth ground truth data for efficient detection and classification of objects according to some embodiments of the present disclosure. Figure 3A The model shown in can be Figure 1In some embodiments, the ODCM 132 of the perception system 130 depicted in FIG. 1 or any other similar model. Figure 3A The model shown may include a collection of neural networks (NNs), such as a depth estimation network (DEN) 310, a context feature network (CFN) 320, a BEF feature network (BEF FN) 350, etc. Figure 3A , but the various NNs and subnetworks shown may be parts of the same NN, which are trained jointly. Neurons in a neural network are associated with learnable weights and biases. Neurons may be arranged in layers. Some layers may be hidden layers. Figure 3A Any NN or sub-network depicted in may include multiple hidden neuron layers and may be configured to perform one or more functions that facilitate detection and classification of objects.

[0055] The input to the DEN 310 and CFN 320 may include one or more images 302, which may be perspective images acquired by one or more cameras. Any number of images 302 may be processed simultaneously. The input images 302 may depict any portion of the external environment, up to a 360-degree panoramic surround view. In some embodiments, the total number of pixels in all images 302 may be W×H, where W is the number of pixels along a first direction (e.g., horizontal direction) and H is the number of pixels along a second direction (e.g., vertical direction). In some embodiments, a “pixel” of an image 302 may correspond to a single element of a charged-coupled device (CCD) of a camera. In some embodiments, a “pixel” of an image 302 may correspond to a downsampled combination (superpixel) of multiple CCD elements of a camera. For example, one or more images of a camera may have a number of pixels different from W×H, such as the size of the input to the trained DEN 310 and CFN 320. In such embodiments, the camera image may first be rescaled (e.g., using interpolation, downsampling / upsampling, etc.) to a target input size W x H. Similarly, the depth data 304 may be rescaled to the target input size W x H. In some embodiments, the depth data 304 may be available only for a subset of the W x H pixels, e.g., one depth value for every N pixels.

[0056] Image 302 may be in any suitable digital format (JPEG, TIFF, GIG, BMP, CGM, SVG, etc.). Image 302 may be obtained by quantizing the intensity of the image 302 using one or more intensity matrices I k (w,h), where 0≤w≤W and 0≤h≤H, and the color index k has a single value (for black and white images), three values ​​(for RGB images), four values ​​(for CMYK images), etc. The intensity value I kContinuous or discrete values ​​between 0 and 1 (or between any other chosen limits, such as 0 and 255) may be assumed.

[0057] DEN 310 may be any suitable NN trained to recognize a collection of depth distributions 312 for various pixels, e.g., the probability P(d) that a given pixel w,h depicts an object located at a distance d from the camera. w,h In some embodiments, the distance d may be spaced in D intervals Δd1, Δd2, ... Δd D The interval Δd i do not need to be of equal size and may vary with distance (e.g., Δd1≤Δd2≤…≤Δd D ) increases. The last interval Δd N It can extend from a certain distance (e.g., 100m, 200m, etc.) to an infinite distance.

[0058] DEN 310 may be trained using training images depicting objects in various environments and using depth data 304 as ground truth. Depth data 304 may be provided by a lidar (radar, sonar, etc.) sensor and may include the actual distance d to the object (or portion of the object) imaged by the corresponding camera pixel w, h. True (w,h). During the training phase, Figure 2 The training engine 242 may change the parameters (e.g., weights and biases) of the neurons of the DEN 310 to minimize a loss function that characterizes the distribution P(d) output by the DEN 310. w,h The distance d between the center (e.g., mean, median, etc.) of True In some embodiments, the distribution P(d) is the difference between (w,h). w,h It can be modeled as a categorical distribution, a Laplace distribution, a Gaussian distribution, or any other suitable distribution. The loss function minimized during training can include a focal loss function, a negative log-likelihood loss function, or any other suitable loss function, such as a mean square loss function, a cross entropy loss function, etc.

[0059] CFN 320 may be trained, for example, using the same training images to output feature vector 322. Feature vector FV(c) for pixel w,h w,hRepresents a learned digital encoding that characterizes the appearance of the corresponding pixel and the graphical context provided by other pixels of the image. The feature vector FV(c) can have any number C of components (bits) c∈[1,C] selected in view of the desired target accuracy for object detection and the specific computing resources on which the trained model (e.g., ODCM 132) is to be deployed, such as C=32, 64, 128, 256 components, etc. Higher values ​​of C are conducive to higher accuracy of object detection and classification, while lower values ​​of C facilitate faster processing and / or deployment of the NN on systems with limited computing resources.

[0060] In some embodiments, DEN 310 and CFN 320 can be deep convolutional NNs, for example, with a U-net architecture with an encoder stage and a decoder stage. Each stage can include multiple convolutional neuron layers and one or more fully connected layers. The convolution performed by each of DEN 310 and CFN 320 can include convolution kernels of any number of different dimensions trained to capture both the local context and the global context of the input image. In some embodiments, DEN 310 and CFN 320 can be completely independent, for example, there is no edge connecting the neurons of the two networks. In some embodiments, DEN 310 and CFN 320 can share a common backbone and can have separate heads that output depth distribution 312 and feature vector 322. Each head can have any number of neuron layers, and in some embodiments, can have its own encoder-decoder architecture. In some embodiments, at least some heads can include one or more fully connected layers.

[0061] The depth distribution 312 and the feature vector 322 may be combined into a feature tensor 332 by a lifting transformation 330. More specifically, the lifting transformation 330 may supplement the feature vector FV(c) with the depth information from the depth distribution 312. w,h For example, the lifting transformation 330 can calculate the feature vector FV(c) (of dimension C×1) w,h Each of the depth distributions P(d) corresponding to the same pixel (of dimension D×1) w,h This generates a feature tensor 332, e.g., the outer product of pixel w,h like Figure 3A The illustration in FIG. 1 is schematically shown.

[0062] The feature tensor FT(c,d) calculated for each pixel can then be used w,h To obtain the combined feature tensor 334 of the entire image, for example, by concatenating the feature tensors 332 of different pixels: {FT(c,d) w,h}→CFT(c,d,w,h). The combined feature tensor CFT(c,d,w,h) has dimensions C×D×W×H. The combined feature tensor 334 may then undergo a 2D mapping 340. More specifically, the 2D mapping 340 may produce a projected feature tensor 342 that uses a convenient set of planar coordinates, such as Cartesian coordinates x and y or polar coordinates r and θ in a ground plane.

[0063] The 2D mapping 340 can be a two-part transformation. During the first part, the perspective coordinates d, w, h can be transformed into 3D Cartesian coordinates d, w, h→x, y, z (or 3D cylindrical coordinates, w, h→r, θ, z), where z is the vertical coordinate (in the direction perpendicular to the ground). The transformation d, w, h→x, y, z can be a projective transformation, parameterized by the focal length of the camera, the direction of the optical axis of the camera, and other similar parameters. In the case where the image 302 is acquired by multiple cameras (or cameras with rotating optical axes), the transformation d, w, h→x, y, z can include multiple projective transformations, for example, with separate transformations for pixels w, h provided by different cameras.

[0064] During the second portion, the 2D mapping 340 may project the combined feature tensor represented by the new coordinates CFT(c,x,y,z) onto a horizontal surface to obtain a projected (BEV) feature tensor 342. For example, to obtain a C×W×H projected feature tensor PCT(c,x,y), the elements associated with each vertical pixel column (e.g., PCT(c,x,y)=∑ i CFT(c,x,y,z i )) sums (or averages) the combined feature tensors. In some embodiments, the coordinate z i The summation over the different coordinates z can be i :PCT(c,x,y)=∑ i w i ×CFT(c,x,y,z i ) with different weights w i This can be performed, for example, with a larger weight w assigned to pixels that image objects within a certain height (e.g., up to several meters) above the ground. i and lower weights assigned to other heights (e.g., to eliminate pseudo objects that do not hinder vehicle motion, such as tree branches, cables, etc.).

[0065] The projected feature tensor 342 characterizes the objects and their locations in the BEV, where perspective distortion has been effectively removed. The BEV FN 350 can then process the projected feature tensor 342 to identify the objects depicted in the image 302 and classify the identified objects. In some embodiments, the BEV FN 350 can be a network with both an encoder stage and a decoder stage. In some embodiments, the BEV FN 350 can be a network with a decoder stage, and the encoder stage is part of the CFN 320. In some embodiments, the BEV FN 350 can be used as a backbone for one or more classification heads 360-n. Each classification head 360-n can output different kinds of information about the objects depicted in the image 302. For example, head A 360-1 can output a bounding box of an object, head B 360-2 can output the type and / or size of the object, head C 360-3 can output the pose (position and orientation) of the object, and so on.

[0066] DEN 310, CFN 320 and BEV FN 350 Figure 3A are depicted as separate blocks. And each may have an encoder stage and a decoder stage. Other implementations are also within the scope of the present disclosure. More specifically, Figure 3B is a schematic diagram of one embodiment of a model 301 that uses a bird's eye view and is trained using depth ground truth data for efficient detection and classification of objects. Figure 3B As shown, DEN 310 and CFN 320 can be supported by a common backbone network such as encoder network 305. Each of DEN 310 and CFN 320 can be a separate decoder trained to process the common output of encoder network 305. In some embodiments, CFN 320 can be a decoder network, and DEN 310 can be a head of one or more fully connected layers. In some embodiments, neither DEN 310 nor CFN 320 includes a decoder. In some embodiments, DEN 310 can be a decoder network, and CFN 320 can be a head with one or more fully connected layers. In some embodiments, one of DEN 310 or CFN 320 can be absent. For example, if CFN 320 does not exist, feature vector 322 can be directly output by encoder network 305, and DEN 310 can include a decoder that further processes a copy of the output (feature vector 322) to generate a depth distribution 312.

[0067] In some embodiments, any or all of the encoder network 305, DEN 310, CFN 320, and BEV FN 350 may be trained together. In some embodiments, some of the encoder network 305, DEN 310, CFN 320, or BEV FN 350 may be trained in stages. More specifically, Figure 3C 303 is shown, in accordance with some embodiments of the present disclosure, where a depth estimation network 310 is pre-trained (along with the encoder network 305, if deployed) using depth ground truth data. Pre-training of the DEN 310 may be performed using a suitable loss function that evaluates the distribution P(d) w,h The center of the depth data 304 is at a distance d from the reference truth value True The difference between (w,h). The difference can then be back-propagated through the various layers of neurons of the DEN 310 (encoder network 305, if deployed) until the DEN 310 (and encoder network 305) learns to predict the likely depth of the pixel with the desired accuracy. After completing the pre-training 306 of the DEN 310 (and encoder network 305, if deployed), the output of the DEN 310 can be used as the ground truth during the training of the CFN 320 and / or BEV FN 350, which can be trained together or sequentially.

[0068] Figure 4A is a schematic diagram illustrating a distillation framework 400 for training a bird's eye view model using depth ground truth data according to some embodiments of the present disclosure. The distillation framework 400 may involve training a student model 401-S using the output of a previously trained teacher model 401-T. The student model 401-S may include a student DEN 410-S, a student CFN 420-S, and a student BEV FN 450-S may be substantially as described with respect to Figure 3A-3C In particular, the student DEN 410-S and the student CFN 420-S can process one or more images 402 and output depth distributions 412 and feature vectors 422 for various pixels of the image 402, which can be subjected to a lifting transform / 2D mapping 430, e.g., similar to the lifting transform 330 and the 2D mapping 340 to obtain a projected feature tensor 442. The student BEV FN 450-S can process the projected feature tensor 442 to identify detected (and classified) objects 452.

[0069] Various outputs (including intermediate outputs) of the student ODCM 401-S can be compared with the ground truth values ​​obtained from the teacher ODCM 401-T. Some of the embodiments, such as Figure 4AAs shown, the teacher ODCM 401-T can have an architecture similar to that of the student ODCM 401-S. In particular, the teacher DEN 410-T and the teacher CFN 420-T can process one or more images 402, which can be the same images as the images input into the student DEN 410-S and the student CFN 420-S. In addition, the teacher DEN 410-T can process depth data 404 (e.g., LiDAR range data) associated with the depth of objects depicted in the image 402 (the distance to the object). To accommodate the additional depth data input 404, the teacher DEN 410-T can have more input neurons than the student DEN-S 410-S. The teacher DEN 410-T and the teacher CFN 420-T may output a depth distribution 411 and feature vectors 421 of pixels of the image 402, which may undergo a lifting transform / 2D mapping 430 to generate a projected feature tensor 441, which is then processed by the teacher BEV FN 450-T to identify detected (and classified) objects 541.

[0070] To allow direct comparison of various outputs (including intermediate outputs) of the student ODCM 401-S with the outputs of the teacher ODCM 401-T, at least some of the teacher DEN 410-T, teacher CFN 420-T, and teacher BEV FN 450-T may have outputs of the same dimensions (number of neurons in the output layer) as the output dimensions of the student ODCM 401-S, student DEN 410-S, student CFN 420-S, and student BEV FN 450-S networks, respectively. On the other hand, the teacher network may have a higher complexity, including the number of neuron layers, the number of connections between layers, the representation accuracy (number of bits) of the intermediate outputs, etc. In some embodiments, the teacher DEN 410-T and the teacher CFN 420-T may have more input neurons than the corresponding student network, and therefore, may be configured to process the image 402 at a higher resolution than the student network. The teacher ODCM 401-T can be used in an off-vehicle setting, while the student ODCM 401-S can be used on a vehicle, a traffic monitoring station, or any edge device with limited computing resources. As a result, the teacher ODCM 401-T does not need to be limited in complexity and / or computing time.

[0071] The student ODCM 401-S can be a lightweight model with far fewer neurons and neuron layers than the teacher ODCM 401-T and capable of faster processing (given the same processing and memory resources). The distillation process to obtain the student ODCM 401-S can include identifying and removing (culling, pruning) nodes and / or edges that have little or reduced impact on the output of the model, combining multiple neuron nodes and / or edges, etc. In addition, the output of the teacher ODCM 401-T can be used as a ground truth during the training of the student ODCM 401-S, such as Figure 4A Schematically depicted by dashed arrows in . More specifically, the depth distribution 411 output by the teacher DEN 410-T and the feature vector 421 output by the teacher CFN 420-T can be used as a ground truth during the training of the student DEN 410-S and the student CFN 420-S. For example, the depth distribution 412 (feature vector 422) can be compared with the depth distribution 411 (feature vector 421) and the parameters of the student DEN 410-S (student CFN 420-S) until the difference is minimized. Similarly, the projected feature tensor 441 and the detected object 451 can be used as a ground truth to compare with the projected feature tensor 442 and the detected object 452. In some embodiments, the difference between the detected object 452 and the ground truth detected object 451 can be back-propagated through the student BEV FN 450-S, but not through the student DEN 410-S and / or the student CFN 420-S. In such an embodiment, the student DEN 410-S is trained based on the difference between the depth distribution 412 and the ground truth depth distribution 444 (and similarly, the student CFN 420-S is trained based on the difference between the feature vector 422 and the ground truth feature vector 421). In other embodiments, the difference between the detected object 452 and the ground truth detected object 451 can be additionally back-propagated through the student DEN 410-S and / or the student CFN 420-S.

[0072] Figure 4B is a schematic diagram illustrating another embodiment of a distillation framework 401 for training a bird's eye view model using depth ground truth data according to some embodiments of the present disclosure. The distillation framework 401 may be used with Figure 4A The distillation framework 400 differs in that the teacher DEN 410-T does not exist, and depth sensing data (e.g., lidar data or radar data) may be used as ground truth instead of the depth distribution 411 during the training of the student DEN 410-S.

[0073] Figure 55 is a schematic diagram illustrating operations 500 of a bird's eye view model for providing instance segmentation and semantic segmentation according to some embodiments of the present disclosure. Operations 500 may include processing an image 502 using a DEN 510 to output a depth distribution 512, processing the image 502 using a CFN 520 to output a feature vector 522, and processing the image 502 using a lifting transform / 2D mapping 530 to obtain a projected feature tensor 542. The DEN 510 may be trained using depth data, for example, as combined with Figure 3A-3C and / or Figure 4A-4B as described, or in any other similar manner. The BEV FN 550 can be a backbone network that processes the projected feature tensor 542 and generates an intermediate output, which is then provided to a plurality of classification heads 560-n. In some embodiments, the classification heads can include a semantics head 560-1 that outputs an instance segmentation map 562-1 that contains classifications of various locations of the BEV among a variety of types (e.g., vehicles, vulnerable road users, roads, buildings, trees, roadside structures, etc.). The classification heads 560-n can include one or more instance segmentation heads. For example, the instance center head 560-2 can output the coordinates of the center 562-2 (e.g., the centroid pixel) of various objects in the environment. The instance offset head 560-3 can output a map of the offset 560-3 of the distances of various pixels of an object to the center of the object. The various maps 562-n can be composed of one or more layer neurons ( Figure 5 560-n) are further processed to obtain a detection and classification map (DCM) 564. DCM 564 can combine the semantic segmentation and geometric (instance) segmentation generated by the classification head 560-n to determine the location of the object and further identify the class of the object. For example, given the geometric information, the type "vehicle" object identified by the semantic head 560-1 can be further subdivided into categories such as "car", "pickup truck", "bus", "semi-truck", etc. Similarly, the type "vulnerable road user" can be further subdivided into categories such as "pedestrian", "bicyclist", "motorcycle rider", "skateboarder", etc.

[0074] Figure 6 6 is a schematic diagram illustrating operation 600 of a bird's-eye view model using temporal aggregation according to some embodiments of the present disclosure. Operation 600 may include processing an image 602 using a DEN 610 to output a depth distribution 612, processing the image 602 using a CFN 620 to output a feature vector 622, and processing the image 602 using a lifting transform / 2D mapping 630 to obtain a projected feature tensor 642. The DEN 610 may be trained using depth data, for example, as combined with Figure 3A-3C and / or Figure 4A-4BAs described, or in any other similar manner. The image 602 may be associated with frames acquired at different times t1, t2, t3, ... The operations of blocks 610-630 may be performed separately for different frames, for example, sequentially (using a single instance of DEN 610 and CFN 620) or in parallel (for example, using multiple instances of DEN 610 and CFN 620). The projected feature tensor 642 may then be warped to a common reference time using warping 644, which may be one of the times t1, t2, t3, ..., for example, the current time or the most recent available time (in Figure 6 , indicated as time t3 in the figure). The deformation 644 can be a mathematical transformation that eliminates the (independently known) motion of the sensing system (e.g., the ego-motion of the autonomous vehicle) and generates a deformation feature tensor 645. As a result of the deformation 644, objects that are stationary relative to the ground will be described by elements of the deformation feature tensor 645 associated with the same coordinates x, y, while moving objects will be described by elements that are spread out along the direction of motion of the object. The deformation feature tensor 645 can then be processed using an aggregation network 646, which can be a convolutional network with a kernel that extends over two or more temporal components of the deformation feature tensor 645. The aggregation network 646 outputs an aggregate tensor 648 that can be processed by the BEV FN 650 (and also further processed by various classification heads 660-n), for example, similar to that in Figure 3A-3C or Figure 4A-Figure 3B The operation of projecting feature tensors in is shown in Figure 2. How the projected feature tensor is processed by the BEV feature network.

[0075] Figure 7-Figure 8 Example methods 700-800 of deploying a machine learning model that uses a bird's eye view and is trained using deep ground truth data to effectively detect and classify objects are shown. A processing device having one or more processing units (CPUs) and a memory device communicatively coupled to the CPU may perform methods 700-800 and / or each of their individual functions, routines, subroutines, or operations. The processing device executing methods 700-800 may perform the processing performed by Figure 1 In some embodiments, methods 700-800 may involve autonomous driving of a vehicle (such as a Figure 1Systems and components of the autonomous vehicle 100). Methods 700-800 may be used to improve the performance of the processing system 120 and / or the autonomous vehicle control system 140. In some embodiments, a single processing thread may execute methods 700-800. Alternatively, two or more processing threads may execute methods 700-800, with each thread executing one or more separate functions, routines, subroutines, or operations of the method. In an illustrative example, the processing threads implementing methods 700-800 may be synchronized (e.g., using semaphores, critical sections, and / or other thread synchronization mechanisms). Alternatively, the processing threads implementing methods 700-800 may execute asynchronously relative to each other. Figure 7-8 The various operations of methods 700-800 may be performed in a different order than shown. Some operations of methods 700-800 may be performed simultaneously with other operations. Some operations may be optional.

[0076] Figure 7 An example method 700 of deploying a model according to some embodiments of the present disclosure is shown, which is represented using a bird's-eye view and trained using deep reference truth data for effective detection and classification of objects. Method 700 can use real-time images obtained by one or more cameras of a vehicle or by a camera installed on any other suitable application platform. The camera can be an optical rangefinder camera and / or an IR camera, including a panoramic (surround view) camera, a partial panoramic camera, a high-definition (high-resolution) camera, a close-up camera, a camera with a fixed field of view (relative to the vehicle), a camera with a dynamic (adjustable) field of view, a camera with a fixed or adjustable focal length, a camera with a fixed or adjustable numerical aperture, and any other suitable camera. The optical rangefinder camera can also include a night vision camera. The image acquired by the camera can include various metadata that provide geometric associations between image pixels and the spatial position of the object, correspondences between pixels of different images, etc. In some embodiments, method 700 can be performed by ODCM 132 or any other similar model, which can be part of the perception system of an autonomous vehicle, a vehicle that deploys driver assistance technology, or part of any other application platform that uses object detection and classification.

[0077] At block 710, method 700 may include obtaining one or more perspective camera images of an environment (e.g., Figure 3A-3CAt block 720, method 700 may include generating, using the first NN, for each pixel in a set of pixels of the one or more perspective camera images, a feature vector (FV) and a depth distribution of a portion of an environment imaged by the corresponding pixel. In some embodiments, the set of pixels may include all pixels of the one or more perspective camera images. In some embodiments, the set of pixels may include only a portion of all pixels of the one or more perspective camera images.

[0078] The first NN may include one or more subnetworks, including an encoder network 305, a DEN 310, a CFN 320, and / or other subnetworks. In some embodiments, a feature vector for each pixel in a set of pixels may be output by a first subnetwork of the first NN (e.g., CFN 320). A depth distribution for each pixel in a set of pixels may be output by a second subnetwork of the first NN (e.g., DEN 310). For example, the feature vector may be FV 322 output by CFN 320, and the depth distribution may be depth distribution 312 output by DEN 310. In some embodiments, the first NN may be trained using a plurality of training images and depth ground truth data for the plurality of training images. In some embodiments, the depth ground truth data may include a lidar determined distance to one or more objects in at least a subset of the plurality of training images. In some embodiments, the second subnetwork (e.g., DEN 310) may be trained using the depth ground truth data prior to training of the first subnetwork (e.g., CFN 320).

[0079] In some embodiments, the first NN can be part of a student model trained using a distillation technique, for example, as combined with Figure 4A-4B More specifically, the depth ground truth data may include depth estimates for at least a subset of pixels of a plurality of training images. For example, a depth estimate may be available for every Nth pixel of a training image. The depth estimate (e.g., depth distribution 411) may be output by a first NN (e.g., teacher DEN 410-S) of a teacher model (e.g., teacher ODCM 401-T).

[0080] At block 730, method 700 may include obtaining a feature tensor (FT) for each pixel in the set of pixels. The feature tensor (e.g., FT 432) may be obtained in view of a feature vector for the corresponding pixel and a depth distribution for the corresponding pixel. For example, the feature tensor may be obtained using an outer product of the feature vector and the depth distribution (for each pixel). Additionally, obtaining the feature tensor may include performing a lifting transformation, such as combining Figure 3A As described.

[0081] At block 740, method 700 may include processing the obtained feature tensor using a second NN to identify one or more objects in the environment. Figure 7 As shown in the labeled portion of , processing the obtained feature tensor may include multiple operations. More specifically, at box 742, method 700 may include using the feature tensor of each pixel in the set of pixels to obtain a combined feature tensor (e.g., combined FT 334). Method 700 may continue to map the combined feature tensor to the ground (or any other reference surface) to obtain a projected feature tensor (e.g., projected FT 342). For example, mapping the combined feature tensor to the ground may include transforming the combined feature tensor to a set of coordinates associated with the ground (e.g., Cartesian coordinates x, y, z, cylindrical coordinates r, θ, z, or any other suitable set of coordinates) at box 744, and aggregating the elements of the combined feature tensor in a vertical direction at box 746 to obtain a projected feature tensor. For example, to obtain the projected feature tensor, the elements of the combined feature tensor associated with different values ​​of z (and the same values ​​of x, y or r, θ) may be added together.

[0082] At block 748, method 700 may continue to process the projected feature tensor using the second NN to identify one or more objects in the environment. In some embodiments, the second NN may include a first classification head (e.g., Figure 5 The second NN may also include at least one second classification head (e.g., instance center head 560-2 and / or instance offset head 560-3) configured to output geometric information associated with the location of one or more objects in the environment. In some embodiments, for example, when the NN is trained using a distillation framework, the output of the second NN (e.g., teacher BEV FN 450-T) of the teacher training model (e.g., teacher ODCM 401-T) may be used to train the second NN (e.g., student BEV FN 450-S).

[0083] In some embodiments, for example, as combined Figure 6As described, a model (e.g., ODCM 132) can perform temporal aggregation of images acquired at different times. More specifically, one or more perspective camera images obtained in conjunction with block 710 can be associated with a first time (e.g., time t1). Temporal aggregation can include obtaining one or more additional perspective camera images associated with at least a second time (e.g., time t2, t3, etc.). Temporal aggregation can then include generating an additional projected feature tensor using the one or more additional perspective camera images, and performing concurrent processing of the projected feature tensor and the additional projected feature tensor (e.g., as shown in the processing of the projected feature tensor 642). In some embodiments, the concurrent processing can include the application of a deformation transformation (e.g., deformation 644) and an aggregation NN (e.g., aggregation network 646). The aggregation NN can include one or more convolution kernels configured to aggregate elements of the projected feature tensor with elements of the additional projected feature tensor (or with multiple additional feature tensors).

[0084] In those embodiments where the perception system performing method 700 is deployed on a vehicle, method 700 may continue at block 750 where the perception system determines a driving path for the vehicle based on one or more identified objects in the vehicle's environment.

[0085] Figure 8 An example method 800 is shown for training a model that deploys a bird's-eye view representation using depth ground truth data for effective detection and classification of objects in accordance with some embodiments of the present disclosure. The method 800 may be used to train the ODCM 132 or any other similar model. The method 800 may use previously recorded images and other sensory data obtained by scanning the vehicle's environment (or any other relevant environment) using multiple sensors of the vehicle's sensing system (e.g., lidar, radar, sonar, etc.). At box 810, the method 800 may include obtaining a training image, for example, as part of a set of multiple training images. At box 815, the method 800 may include processing the training image using a first NN of a student model to generate a plurality of feature vectors and a plurality of depth distributions. For example, the first NN of the student model may include some or both of the student DEN 410-S and the student CFN 420-S, for example, as Figure 4A-4B Each feature vector in the plurality of feature vectors (eg, FV 422) and each depth distribution in the plurality of depth distributions (eg, depth distribution 412) may be associated with a corresponding pixel in the plurality of pixels of the training image.

[0086] At box 820, method 800 can continue to obtain a plurality of ground truth feature vectors (e.g., feature vector 421) generated by the first NN of the teacher model. For example, the first NN of the teacher model can include some or both of the teacher DEN 410-T and the teacher CFN 420-T. Each of the plurality of ground truth feature vectors can be associated with a corresponding pixel in a plurality of pixels of the training image. At box 825, method 800 can continue to obtain a plurality of ground truth depth indicators. Each of the plurality of ground truth depth indicators can be associated with a corresponding pixel in at least a subset of the plurality of pixels of the training image. In some embodiments, each of the plurality of ground truth depth indicators can include a depth distribution obtained by the first NN of the teacher model for the associated pixel (e.g., depth distribution 411, such as Figure 4A In some embodiments, each of the plurality of ground truth depth indicators may include a particular distance (e.g., depth data 404, such as 404, obtained by a distance sensing device) to a portion of an environment imaged by an associated pixel. Figure 4B shown).

[0087] At box 830, method 800 may include adjusting parameters of the first NN of the student model based on comparing multiple feature vectors (e.g., feature vector 422) with multiple baseline true feature vectors (e.g., feature vector 422) and comparing multiple depth distributions (e.g., depth distribution 412) with multiple baseline true depth indicators.

[0088] At block 835, method 800 may include obtaining a plurality of feature tensors. Each feature tensor in the plurality of feature tensors may be obtained using a corresponding feature vector in the plurality of feature vectors and a corresponding depth distribution in the plurality of depth distributions (e.g., as combined with Figure 3A). At box 840, method 800 may continue to obtain a combined feature tensor (e.g., combined feature tensor 334) using the plurality of feature tensors. At box 845, method 800 may include mapping the combined feature tensor to a ground plane to obtain a projected feature tensor (e.g., using lifting transform / 2D mapping 430). At box 850, method 800 may continue to process the projected feature tensor using a second NN of the student model (e.g., student BEV FN 450-S) to identify one or more objects in the training image (e.g., detected object 452). At box 855, method 800 may include obtaining one or more ground truth objects (e.g., detected object 451) identified by a second NN of a teacher model (e.g., teacher BEV FN 450-T) in the training image. At box 860, method 800 may continue to adjust parameters of the second NN of the student model (e.g., student BEV FN 450-S) based on a comparison of the one or more objects identified by the second NN of the student model with the one or more objects identified by the second NN of the teacher model.

[0089] Fig. 9 A block diagram of an example computer device 900 capable of operating and / or training a model according to some embodiments of the present disclosure is depicted, the model uses a bird's eye view and is trained using deep reference truth data for effective detection and classification of objects. Example computer device 900 can be connected to other computer devices in a LAN, intranet, extranet, and / or the Internet. Computer device 900 can operate with the capabilities of a server in a client-server network environment. Computer device 900 can be a personal computer (PC), a set-top box (STB), a server, a network router, a switch or a bridge, or any device capable of executing a set (sequential or otherwise) of instructions specifying an action to be taken by the device. In addition, although only a single example computer device is shown, the term "computer" should also be considered to include a set (or multiple sets) of instructions that are executed individually or jointly to perform any set of computers of any one or more methods discussed herein.

[0090] The example computer device 900 may include a processing device 902 (also referred to as a processor or CPU), a main memory 904 (e.g., read-only memory (ROM), flash memory, dynamic random access memory (DRAM) such as synchronous DRAM (SDRAM), etc.), a static memory 906 (e.g., flash memory, static random access memory (SRAM), etc.), and a secondary memory (e.g., a data storage device 918), which may communicate with each other via a bus 930.

[0091] The processing device 902 (which may include processing logic 903) represents one or more general-purpose processing devices, such as a microprocessor, a central processing unit, etc. More specifically, the processing device 902 may be a complex instruction set computing (CISC) microprocessor, a reduced instruction set computing (RISC) microprocessor, a very long instruction word (VLIW) microprocessor, a processor implementing other instruction sets, or a processor implementing a combination of instruction sets. The processing device 902 may also be one or more special-purpose processing devices, such as an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), a digital signal processor (DSP), a network processor, etc. According to one or more aspects of the present disclosure, the processing device 902 may be configured to execute instructions that execute the method 700 of using deep reference truth data to train a model for deploying a bird's-eye view representation and / or the method 800 of using deep reference truth data to train a model for deploying a bird's-eye view representation for effective detection and classification of objects.

[0092] The example computer device 900 may also include a network interface device 908, which may be communicatively coupled to a network 920. The example computer device 900 may also include a video display 910 (e.g., a liquid crystal display (LCD), a touch screen, or a cathode ray tube (CRT)), an alphanumeric input device 912 (e.g., a keyboard), a cursor control device 914 (e.g., a mouse), and an acoustic signal generating device 916 (e.g., a speaker).

[0093] The data storage device 918 may include a computer-readable storage medium (or more specifically, a non-transitory computer-readable storage medium) 928 having stored thereon a set of one or more executable instructions 922. According to one or more aspects of the present disclosure, the executable instructions 922 may include executable instructions for executing the method 700 of training a model for deploying a bird's-eye view representation using depth ground truth data and / or the method 800 of training a model for deploying a bird's-eye view representation using depth ground truth data for efficient detection and classification of objects.

[0094] The executable instructions 922 may also reside completely or at least partially within the main memory 904 and / or within the processing device 902 during execution by the example computer device 900, the main memory 904 and the processing device 902 also constituting computer-readable storage media. The executable instructions 922 may also be sent or received over a network via the network interface device 908.

[0095] Although the computer readable storage medium 928 is Fig. 9Although shown as a single medium in the text, the term "computer-readable storage medium" should be considered to include a single medium or multiple media (e.g., a centralized or distributed database and / or associated caches and servers) that store a set of one or more operating instructions. The term "computer-readable storage medium" should also be considered to include any medium that can store or encode a set of instructions executed by a machine, which set of instructions causes the machine to perform any one or more of the methods described herein. Therefore, the term "computer-readable storage medium" should be considered to include, but is not limited to, solid-state memories and optical and magnetic media.

[0096] Some portions of the above detailed description are presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the art of data processing to most effectively convey the substance of their work to others skilled in the art. An algorithm is here and generally considered to be a self-consistent sequence of steps leading to a desired result. These steps are those requiring physical manipulations of physical quantities. Typically, although not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, transferred, combined, compared, and otherwise manipulated. Mainly for reasons of common usage, it has proven convenient at times to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

[0097] It should be remembered, however, that all of these and similar terms will be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless expressly stated otherwise, as will be clear from the following discussion, it should be understood that throughout the specification, discussions utilizing terms such as "identify," "determine," "store," "adjust," "cause," "return," "compare," "create," "stop," "load," "copy," "throw," "replace," "execute," and the like refer to the actions and processes of a computer system or similar electronic computing device that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system's memories or registers or other such information storage, transmission, or display devices.

[0098] Examples of the present disclosure also relate to an apparatus for performing the methods described herein. The apparatus may be specially constructed for the desired purpose, or it may be a general-purpose computer system selectively programmed by a computer program stored in a computer system. Such a computer program may be stored in a computer-readable storage medium, such as, but not limited to, any type of disk, including optical disks, CD-ROMs and magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic disk storage media, optical storage media, flash memory devices, other types of machine-accessible storage media, or any type of medium suitable for storing electronic instructions, each coupled to a computer system bus.

[0099] The method and display presented herein are not inherently related to any particular computer or other device. According to the teachings herein, various general purpose systems can be used together with the program, or it can be proved that it is convenient to construct a more specialized device to perform the required method steps. The structure required for various these systems will appear as described in the following description. In addition, the scope of the present disclosure is not limited to any specific programming language. It should be understood that various programming languages ​​can be used to implement the teachings of the present disclosure.

[0100] It should be understood that the above description is intended to be illustrative rather than restrictive. After reading and understanding the above description, many other implementation examples will be clear to those skilled in the art. Although the present disclosure describes specific examples, it will be appreciated that the system and method of the present disclosure are not limited to the examples described herein, but can be modified to practice within the scope of the attached claims. Therefore, the description and the drawings are considered to be illustrative rather than restrictive. Therefore, the scope of the present disclosure should be determined with reference to the attached claims and the full scope of equivalents given by these claims.

Claims

1. A method comprising: obtaining one or more perspective camera images of the environment; Generating, using a first neural network (NN), for each pixel in a set of pixels of the one or more perspective camera images: feature vector (FV), and a depth distribution of a portion of an environment imaged by the corresponding pixel, wherein the first NN is trained using a plurality of training images and depth ground truth data for the plurality of training images; For each pixel in the set of pixels, obtaining a feature tensor (FT) given (i) the FV of the corresponding pixel and (ii) the depth distribution of the corresponding pixel; and The obtained FT is processed using a second NN to recognize one or more objects in the environment.

2. The method according to claim 1, wherein: The FT obtained by processing includes: Using the FT of the set of pixels to obtain a combined FT; Mapping the combined FT to the ground to obtain a projected FT; and The projected FT is processed using a second NN.

3. The method according to claim 2, wherein: Mapping the combined FT to the ground involves: transforming the combined FT into a set of coordinates associated with the ground; and Aggregate the elements of the combined FT in the vertical direction to obtain the projected FT.

4. The method according to claim 2, wherein: The one or more perspective camera images are associated with a first time, the method further comprising: obtaining one or more additional perspective camera images associated with at least a second time; generating additional projections FT using the one or more additional perspective camera images; and Perform concurrent processing of projection FT and additional projection FT.

5. The method according to claim 4, wherein: The concurrent processing is performed by an aggregation NN comprising one or more convolution kernels configured to aggregate elements of the projected FT with elements of additional projected FTs.

6. The method according to claim 1, wherein: The second NN includes: A first classification head configured to output a semantic segmentation of one or more objects in the environment; and At least one second classification head is configured to output geometric information associated with the location of the one or more objects in the environment.

7. The method according to claim 1, wherein: The depth ground truth data comprises depth estimates for at least a subset of pixels of the plurality of training images, wherein the depth estimates are output by a first NN of the teacher model.

8. The method according to claim 7, wherein: The second NN is trained using the output of the second NN of the teacher training model.

9. The method according to claim 1, wherein: The FT of each pixel in the set of pixels is output by a first subnetwork of the first NN, wherein the depth distribution of each pixel in the set of pixels is output by a second subnetwork of the first NN, and wherein, before training of the first subnetwork, the second subnetwork is trained using depth reference truth data.

10. The method according to claim 1, wherein: The depth ground truth data includes lidar determined distances to one or more objects in at least a subset of the plurality of training images.

11. A method for training a student model, the method comprising: Obtain training images; The first neural network (NN) using the student model processes the training images to generate multiple feature vectors (FV), and a plurality of depth profiles, wherein each FV of the plurality of FVs and each depth profile of the plurality of depth profiles is associated with a corresponding pixel of a plurality of pixels of a training image; obtaining a plurality of ground truth values ​​FV generated by a first NN of a teacher model, wherein each ground truth value FV of the plurality of ground truth values ​​FV is associated with a corresponding pixel of the plurality of pixels of a training image; obtaining a plurality of ground truth depth indicators, wherein each ground truth depth indicator of the plurality of ground truth depth indicators is associated with a corresponding pixel of at least a subset of the plurality of pixels of a training image; and Adjust the parameters of the first NN of the student model based on comparison of the plurality of FVs with the plurality of reference true values ​​FV, ​​and A comparison of the plurality of depth profiles to the plurality of ground truth depth indicators.

12. The method according to claim 11, further comprising: Obtaining a plurality of feature tensors (FTs), wherein each FT of the plurality of FTs is obtained using a corresponding FV of the plurality of FVs and a corresponding depth distribution of the plurality of depth distributions; obtaining a combined FT using the plurality of FTs; Map the combined FT to the ground to obtain the projected FT; Processing the projection FT using a second NN of the student model to recognize one or more objects in the training image; obtaining one or more ground truth objects in the training image recognized by the second NN of the teacher model; and Parameters of the second NN of the student model are adjusted based on a comparison of the one or more objects recognized by the second NN of the student model and the one or more objects recognized by the second NN of the teacher model.

13. The method according to claim 11, wherein: Each of the plurality of ground truth depth indicators comprises at least one of (i) a depth distribution obtained by a first NN of a teacher model for an associated pixel, or (ii) a distance to a portion of an environment imaged by the associated pixel obtained by a distance sensing device.

14. A system comprising: Memory; and A processing device is communicatively coupled to the memory, the processing device being configured to: obtaining one or more perspective camera images of the environment; Generating, using a first neural network (NN), for each pixel in a set of pixels of the one or more perspective camera images: feature vector (FV), and a depth distribution of a portion of an environment imaged by the corresponding pixel, wherein the first NN is trained using a plurality of training images and depth ground truth data for the plurality of training images; For each pixel in the set of pixels, obtaining a feature tensor (FT) given (i) the FV of the corresponding pixel and (ii) the depth distribution of the corresponding pixel; and The obtained FT is processed using a second NN to recognize one or more objects in the environment.

15. The system of claim 14, wherein: To process the obtained FT, processing equipment is used to: Using the FT of the set of pixels to obtain a combined FT; Mapping the combined FT to the ground to obtain a projected FT; and The projected FT is processed using a second NN.

16. The system of claim 15, wherein: To map the combined FT to the ground, processing equipment is used to: transforming the combined FT into a set of coordinates associated with the ground; and Aggregate the elements of the combined FT in the vertical direction to obtain the projected FT.

17. The system of claim 15, wherein: The one or more perspective camera images are associated with a first time, and wherein the processing device is further configured to: obtaining one or more additional perspective camera images associated with at least a second time; generating additional projections FT using the one or more additional perspective camera images; and Concurrent processing of the projection FT and the additional projection FT is performed, wherein the concurrent processing is performed by an aggregation NN comprising one or more convolution kernels configured to aggregate elements of the projection FT with elements of the additional projection FT.

18. The system of claim 14, wherein: The second NN includes: A first classification head configured to output a semantic segmentation of the one or more objects in the environment; and At least one second classification head is configured to output geometric information associated with the location of the one or more objects in the environment.

19. The system of claim 14, wherein: The FT of each pixel in the set of pixels is output by a first subnetwork of the first NN, wherein the depth distribution of each pixel in the set of pixels is output by a second subnetwork of the first NN, and wherein, before training of the first subnetwork, the second subnetwork is trained using depth reference truth data.

20. The system of claim 14, wherein: The depth ground truth data includes lidar determined distances to one or more objects in at least a subset of the plurality of training images.

Images