Distance estimation using geometric distance aware machine learning model

Abstract

Techniques and systems are provided for generating depth information for an image. For example, a process may include obtaining one or more images of an environment. The process may also include generating a set of features for the one or more images. The process may also include combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate a distance based on a relative height above a ground level. The process may also include generating depth information for the environment based on the combined feature distance information, and outputting the depth information for the environment.

Description

Technical Field

[0001] The present disclosure relates generally to object localization, and more particularly, to techniques, systems, and methods for improving localization using geometric distance-aware machine learning models. Background Art

[0002] Increasingly, systems and devices (e.g., autonomous vehicles such as autonomous and semi-autonomous cars, drones, mobile robots, mobile devices, extended reality (XR) devices, and other suitable systems or devices) include multiple sensors for collecting information about the environment, and processing systems for processing the collected information, such as for route planning, navigation, collision avoidance, etc. An example of such a system is an advanced driver assistance system (ADAS) for a vehicle. Sensor data (such as images captured from one or more cameras) can be collected, transformed, and analyzed to detect objects. An example of a transformed view or model is a semantically segmented bird's eye view (BEV) (e.g., a top-down view) of the environment.

[0003] Generating semantically segmented BEVs is useful for many applications and systems, including XR (e.g., augmented reality (AR), virtual reality (VR), mixed reality (MR)), robotic systems, manufacturing systems, quality assurance, automotive and aerospace (e.g., manufacturing, autonomous driving or navigation, etc.), 3D scene understanding, object grasping, object tracking, video analytics, security systems, and many others. For example, a semantically segmented BEV may include an image where each pixel of the image is annotated according to the object category to predict the object in view. A semantically segmented BEV can facilitate the efficient operation of a variety of systems. Summary of the invention

[0004] The following presents a simplified summary of the invention related to one or more aspects disclosed herein. Therefore, the following summary of the invention should neither be considered as an exhaustive overview related to all conceived aspects, nor should it be considered to identify key or decisive elements related to all conceived aspects or to delineate the scope associated with any particular aspect. Accordingly, the following summary of the invention presents certain concepts related to one or more aspects of the mechanisms disclosed herein in a simplified form before the detailed description presented below.

[0005] In one illustrative example, an apparatus for generating depth information is provided. The apparatus includes at least one memory having instructions and at least one processor coupled to the at least one memory. The at least one processor is configured to: obtain one or more images of an environment; generate a set of features for the one or more images; combine the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generate depth information of the environment based on the combined feature distance information; and output the depth information of the environment.

[0006] In another example, a method for generating depth information includes: obtaining one or more images of an environment; generating a set of features for the one or more images; combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generating depth information for the environment based on the combined feature distance information; and outputting the depth information for the environment.

[0007] In another example, a non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to: obtain one or more images of an environment; generate a set of features for the one or more images; combine the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distance based on a relative height above a ground level; generate depth information for the environment based on the combined feature distance information; and output the depth information for the environment.

[0008] In another example, a device for generating depth information includes: a component for obtaining one or more images of an environment; a component for generating a set of features for the one or more images; a component for combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; a component for generating depth information of the environment based on the combined feature distance information; and a component for outputting the depth information of the environment. In some cases, the device may also include: a component for performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and a component for outputting the set of transformed features, wherein the depth information of the environment is implicitly output in the set of transformed features.

[0009] In some aspects, the apparatus is, is part of, and / or may include a vehicle, a computing device or component of a vehicle (e.g., an autonomous vehicle), a camera, a mobile device (e.g., a mobile phone or so-called "smartphone" or other mobile device), a wearable device, an extended reality device (e.g., a virtual reality (VR) device, an augmented reality (AR) device, or a mixed reality (MR) device), a personal computer, a laptop computer, a server computer, or other device. In some aspects, the apparatus includes a camera or multiple cameras for capturing one or more images. In some aspects, the apparatus also includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the apparatus described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyroscopes, one or more accelerometers, any combination thereof, and / or other sensors).

[0010] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all drawings, and each claim.

[0011] The foregoing and other features and embodiments will become more apparent upon reference to the following description, claims and accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Exemplary embodiments of the present application are described in detail below with reference to the following drawings:

[0013] Figure 1A and Figure 1B is a block diagram illustrating a vehicle suitable for implementing the various techniques described herein, according to aspects of the present disclosure.

[0014] Figure 1C is a block diagram illustrating components of a vehicle suitable for implementing the various techniques described herein in accordance with aspects of the present disclosure;

[0015] Figure 1D illustrates example implementations of a system on a chip (SOC) according to some examples;

[0016] Figure 2A is a component block diagram illustrating components of an example vehicle management system according to aspects of the present disclosure;

[0017] Figure 2B is a component block diagram illustrating components of another example vehicle management system according to aspects of the present disclosure;

[0018] FIG. 3A to FIG. 4is a diagram illustrating an example of a neural network according to aspects of the present disclosure;

[0019] Figure 5 is a diagram illustrating an example of a system for processing an image for use by a device according to aspects of the present disclosure;

[0020] Figure 6 A set of range map images based on known camera characteristics according to aspects of the present disclosure are illustrated;

[0021] Figure 7 is a diagram illustrating an example of a system for predicting a distance of a detected object according to aspects of the present disclosure;

[0022] Figure 8 is a flow chart illustrating an example of a process for generating one or more depth maps according to aspects of the present disclosure;

[0023] Fig. 9 An example computing device architecture is illustrated that illustrates an example computing device that may implement the techniques described herein. DETAILED DESCRIPTION

[0024] Certain aspects of the present disclosure are provided below. Some of these aspects may be applied independently, and some of them may be applied in combination, which is obvious to those skilled in the art. In the following description, specific details are set forth for explanation purposes to provide a thorough understanding of various aspects of the application. However, it is apparent that various aspects may be practiced without these specific details. Each of the drawings and descriptions is not intended to be restrictive.

[0025] The following description provides only example aspects and is not intended to limit the scope, applicability or configuration of the present disclosure. On the contrary, the following description of the example aspects will provide a description that can be used to implement the example aspects to those skilled in the art. It should be understood that various changes may be made to the function and arrangement of the elements without departing from the essence and scope of the present application as set forth in the appended claims.

[0026] As previously noted, one or more sensors (e.g., image sensors such as cameras, ranging sensors such as radar and / or light detection and ranging (lidar) sensors, etc.) of a device or system may be used to obtain information about the environment in which the device or system is located. A processing system of the device or system may be used to process the information for one or more operations, such as route planning, navigation, collision avoidance, etc. For example, in some cases, sensor data (e.g., one or more images captured from one or more cameras, depth information captured or determined by one or more radar and / or lidar sensors, etc.) may be obtained from one or more sensors and transformed and analyzed to detect an object.

[0027] An example of a transformed view or model is a semantically segmented bird's eye view (BEV) (e.g., a top-down view) of an environment. In a semantically segmented BEV, each pixel corresponds to a grid cell in the real world (e.g., an imaging environment). Each pixel can be assigned a value and the value corresponds to a category (e.g., a group) representing the real-world content at the corresponding location. The semantically segmented BEV can be created by generating features from an image and then transforming those generated features into the BEV space. These features in the transformed space can then be further processed, for example, by a machine learning (ML) model to obtain semantically segmented BEV information. In some cases, 3D object detection can be performed in parallel with or instead of the semantically segmented BEV.

[0028] Generating semantically segmented BEVs is useful for many applications and systems, including XR applications (e.g., AR systems, VR systems, MR systems, etc.), robotic systems, manufacturing systems, quality assurance, manufacturing, autonomous driving or navigation, three-dimensional (3D) scene understanding, object grasping, object tracking, video analysis, security systems, and the like. For example, a semantically segmented BEV may include an image where each pixel of the image is annotated according to an object category to predict objects in view. A semantically segmented BEV may facilitate efficient operation of a variety of systems. In an illustrative example, an autonomous vehicle may use a BEV representation of the environment surrounding the autonomous vehicle to navigate in traffic.

[0029] The exact positioning of any object in a BEV image and 3D object detection based on a monocular image is non-trivial and a challenging task (e.g., in computer vision). This problem may be challenging, at least in part because camera images do not contain accurate depth information. Accurately determining the positioning or position of the detected object may be useful for the various applications and systems noted above, such as autonomous vehicle perception, XR applications, etc. For example, both BEV segmentation and 3D object detection may be useful for autonomous vehicle guidance. Both of these tasks may require 3D perception of the world, which may be challenging in the case of a set of monocular cameras that capture two-dimensional (2D) projections of the 3D world. Existing solutions for distance estimation typically employ multiple sensors, such as radar sensors, lidar sensors, or multiple cameras. Techniques for implicit or explicit distance estimation that reduce or eliminate the need for multiple sensors may be useful.

[0030] Systems, apparatus, electronic devices, methods (also referred to as processes), and computer-readable media (collectively referred to herein as "systems and techniques") are described for providing (and training) machine learning-based algorithms for accurately estimating depth from images from a monocular camera using geometric prior knowledge. The implicit depth information of the image can be used, for example, to generate explicit depth information using an ML model to improve localization accuracy, which typically decreases with the distance of an object from the sensor.

[0031] The systems and techniques described herein can be used to improve positioning accuracy for a variety of applications and systems, including autonomous driving, XR systems, robotics, scene understanding, etc. Higher accuracy for computer vision tasks can be achieved to replace expensive sensors (e.g., lidar sensors), which can result in large cost benefits.

[0032] Various aspects of the application will be described with respect to the accompanying drawings.

[0033] The systems and techniques described herein can be implemented by any type of system or device. An illustrative example of a system that can be used to implement the systems and techniques described herein is a vehicle (e.g., an autonomous or semi-autonomous vehicle) or a system or component of the vehicle (e.g., an ADAS or other system or component). Figure 1A and Figure 1B is a diagram illustrating an example vehicle 100 in which the systems and techniques described herein may be implemented. Figure 1A and Figure 1B, the vehicle 100 may include a control unit 140 and a plurality of sensors 102 to 138, including a satellite geo-positioning system receiver (e.g., sensor) 108, occupancy sensors 112, 116, 118, 126, 128, tire pressure sensors 114, 120, cameras 122, 136, microphones 124, 134, impact sensors 130, radar 132, and lidar 138. The plurality of sensors 102 to 138 disposed in or on the vehicle may be used for various purposes, such as autonomous and semi-autonomous navigation and control, collision avoidance, location determination, etc., and provide sensor data about objects and people in or on the vehicle 100. The sensors 102 to 138 may include one or more of a wide variety of sensors capable of detecting a variety of information useful for navigation and collision avoidance. Each of the sensors 102 to 138 may communicate with the control unit 140 and each other in wired or wireless communication. Specifically, the sensors may include one or more cameras 122, 136 or other optical or photoelectric sensors. The sensors may also include other types of object detection and ranging sensors, such as radar 132, lidar 138, IR sensors, and ultrasonic sensors. The sensors may also include tire pressure sensors 114, 120, humidity sensors, temperature sensors, satellite geolocation sensors 108, accelerometers, vibration sensors, gyroscopes, gravity meters, impact sensors 130, force gauges, pressure gauges, strain sensors, fluid sensors, chemical sensors, gas content analyzers, pH sensors, radiation sensors, Geiger counters, neutron detectors, biomaterial sensors, microphones 124, 134, occupancy sensors 112, 116, 118, 126, 128, proximity sensors, and other sensors.

[0034] The vehicle control unit 140 may be configured with processor-executable instructions to use information received from various sensors (particularly the cameras 122, 136, the radar 132, and the lidar 138) to perform various embodiments. In some embodiments, the control unit 140 may supplement the processing of camera images with distance and relative positioning information (e.g., relative azimuth) available from the radar 132 and / or lidar 138 sensors. The control unit 140 may also be configured to use information about other vehicles determined using various embodiments to control the steering, braking, and speed of the vehicle 100 when operating in an autonomous or semi-autonomous mode.

[0035] Figure 1C is a component block diagram of a system 150 illustrating components and supporting systems suitable for implementing various embodiments. Figure 1A , 1B 1C, the vehicle 100 may include a control unit 140, which may include various circuits and devices for controlling the operation of the vehicle 100. Figure 1CIn the example illustrated in FIG. 1 , control unit 140 includes processor 164, memory 166, input module 168, output module 170, and radio module 172. Control unit 140 may be coupled to and configured to control driving control component 154, navigation component 156, and one or more sensors 158 of vehicle 100.

[0036] The control unit 140 may include a processor 164 that may be configured with processor-executable instructions to control maneuvering, navigation, and / or other operations of the vehicle 100, including operations of various embodiments. The processor 164 may be coupled to a memory 166. The control unit 140 may include an input module 168, an output module 170, and a radio module 172.

[0037] The radio module 172 may be configured for wireless communication. The radio module 172 may exchange signals 182 (e.g., command signals for controlling maneuvers, signals from navigation aids, etc.) with the network node 180, and may provide the signals 182 to the processor 164 and / or the navigation component 156. In some embodiments, the radio module 172 may enable the vehicle 100 to communicate with the wireless communication device 190 via the wireless communication link 92. The wireless communication link 92 may be a two-way or one-way communication link, and may use one or more communication protocols.

[0038] Input module 168 may receive sensor data from one or more vehicle sensors 158 and electronic signals from other components, including driving control component 154 and navigation component 156. Output module 170 may be used to communicate with or activate various components of vehicle 100, including driving control component 154, navigation component 156, and sensors 158.

[0039] The control unit 140 may be coupled to a driving control assembly 154 to control physical elements of the vehicle 100 related to maneuvering and navigation of the vehicle, such as the engine, motor, throttle, steering elements, other control elements, braking or deceleration elements, etc. The driving control assembly 154 may also include components for controlling other devices of the vehicle, including environmental controls (e.g., air conditioning and heating), exterior and / or interior lighting, interior and / or exterior information displays (which may include display screens or other devices for displaying information), safety devices (e.g., tactile devices, audible alarms, etc.), and other similar devices.

[0040] The control unit 140 may be coupled to the navigation component 156 and may receive data from the navigation component 156. The control unit 140 may be configured to use such data to determine the current location and orientation of the vehicle 100 and the appropriate route to the destination. In various embodiments, the navigation component 156 may include or be coupled to a GNSS receiver system (e.g., one or more global positioning system (GPS) receivers) that enables the vehicle 100 to use a global navigation satellite system (GNSS) signal to determine its current location. Alternatively or in addition, the navigation component 156 may include a radio navigation receiver for receiving navigation beacons or other signals from radio nodes (such as Wi-Fi access points, cellular network sites, radio stations, remote computing devices, other vehicles, etc.). Through the control of the driving control component 154, the processor 164 can control the vehicle 100 to navigate and maneuver. The processor 164 and / or the navigation component 156 may be configured to communicate with a server 184 on a network 186 (e.g., the Internet) using wireless signals 182 exchanged over a cellular data network via a network node 180 to receive commands for controlling maneuvers, receiving data useful in navigation, providing real-time location reports, and evaluating other data.

[0041] Control unit 140 may be coupled to one or more sensors 158. Sensors 158 may include sensors 102-138 as described, and may be configured to provide various data to processor 164.

[0042] Although the control unit 140 is described as including separate components, in some embodiments, some or all of the components (e.g., the processor 164, the memory 166, the input module 168, the output module 170, and the radio module 172) may be integrated into a single device or module (such as a system-on-chip (SOC) processing device). Such an SOC processing device may be configured for use in a vehicle and configured (such as configured with processor-executable instructions executed in the processor 164) to perform the operations of the various embodiments when installed in the vehicle.

[0043] Figure 1DAn example implementation of a system on chip (SOC) 105 is illustrated, which may include a central processing unit (CPU) 110 or a multi-core CPU configured to perform one or more of the functions described herein. In some cases, the SOC 105 may be based on the ARM instruction set. In some cases, the CPU 110 may be similar to the processor 164. Parameters or variables (e.g., neural signals and synaptic weights), system parameters associated with a computing device (e.g., a neural network with weights), delays, frequency bin information, task information, and other information may be stored in a memory block associated with a neural processing unit (NPU) 125, in a memory block associated with the CPU 110, in a memory block associated with a graphics processing unit (GPU) 115, in a memory block associated with a digital signal processor (DSP) 106, in a memory block 185, and / or may be distributed in multiple blocks. Instructions executed at the CPU 110 may be loaded from a program memory associated with the CPU 110, or may be loaded from the memory block 185.

[0044] The SOC 105 may also include additional processing blocks tailored for specific functions, such as a GPU 115, a DSP 106, a connectivity block 135 (which may include fifth generation (5G) connectivity, fourth generation long term evolution (4G LTE) connectivity, Wi-Fi connectivity, USB connectivity, Bluetooth connectivity, etc.), and a multimedia processor 145 that may, for example, detect and recognize gestures. In one specific implementation, the NPU is implemented in the CPU 110, the DSP 106, and / or the GPU 115. The SOC 105 may also include a sensor processor 155, an image signal processor (ISP) 175, and / or a navigation module 195, which may include a global positioning system. In some cases, the navigation module 195 may be similar to the navigation component 156, and the sensor processor 155 may accept inputs from, for example, one or more sensors 158. In some cases, the connectivity block 135 may be similar to the radio module 172.

[0045] Figure 2A An example of a method that can be used in a vehicle such as Figure 1A Examples of vehicle applications, subsystems, computing elements or units within a vehicle management system 200 utilized within a vehicle 100 of FIG. Figures 1A to 2AIn some embodiments, various vehicle applications, computing elements, or units within the vehicle management system 200 may be implemented within a system of interconnected computing devices (i.e., subsystems) that communicate data and commands to each other. In other embodiments, the vehicle management system 200 may be implemented as multiple vehicle applications, such as separate threads, processes, algorithms, or computing elements, executed within a single computing device. However, the use of the term vehicle application when describing various embodiments is not intended to imply or require that the corresponding functionality be implemented within a single autonomous (or semi-autonomous) vehicle management system computing device, although this is a potential implementation embodiment. Instead, the use of the term vehicle application is intended to include subsystems with independent processors, computing elements (e.g., threads, algorithms, subroutines, etc.) running in one or more computing devices, and combinations of subsystems and computing elements.

[0046] In various embodiments, vehicle applications executed in the vehicle management system 200 may include, but are not limited to, a radar perception vehicle application 202, a camera perception vehicle application 204, a positioning engine vehicle application 206, a map fusion and arbitration vehicle application 208, a route vehicle planning application 210, a sensor fusion and road world model (RWM) management vehicle application 212, a motion planning and control vehicle application 214, and a behavior planning and prediction vehicle application 216. The vehicle applications 202 to 216 are merely examples of some of the vehicle applications in one example configuration of the vehicle management system 200. In other configurations consistent with various embodiments, other vehicle applications may be included, such as additional vehicle applications for other perception sensors (e.g., a lidar perception layer, etc.), additional vehicle applications for planning and / or control, additional vehicle applications for modeling, etc., and / or some of the vehicle applications 202 to 216 may be excluded from the vehicle management system 200. Each of the vehicle applications 202 to 216 may exchange data, calculation results, and commands.

[0047] The vehicle management system 200 may receive and process data from sensors (e.g., radar, lidar, camera, inertial measurement unit (IMU), etc.), navigation systems (e.g., GPS receiver, IMU, etc.), vehicle networks (e.g., controller area network (CAN) bus), and databases in memory (e.g., digital map data). The vehicle management system 200 may output vehicle control commands or signals to a drive-by-wire (DBW) system / control unit 220, which is a system, subsystem, or computing device that interfaces directly with the vehicle's steering, throttle, and brake controls. Figure 2A The configurations of the vehicle management system 200 and the DBW system / control unit 220 illustrated in FIG. 1 are merely example configurations and other configurations of the vehicle management system and other vehicle components may be used in various embodiments. As an example, Figure 2AThe configuration of the vehicle management system 200 and DBW system / control unit 220 illustrated in FIG. 2 may be used in a vehicle configured for autonomous or semi-autonomous operation, while a different configuration may be used in a non-autonomous vehicle.

[0048] The radar-aware vehicle application 202 may receive data from one or more detection and ranging sensors, such as radar (e.g., 132) and / or lidar (e.g., 138), and process the data to identify and determine the location of other vehicles and objects in the vicinity of the vehicle 100. The radar-aware vehicle application 202 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management vehicle application 212.

[0049] The camera awareness vehicle application 204 may receive data from one or more cameras, such as cameras (e.g., 122, 136), and process the data to identify and determine the location of other vehicles and objects in the vicinity of the vehicle 100. The camera awareness vehicle application 204 may include the use of neural network processing and artificial intelligence methods to identify objects and vehicles and pass such information to the sensor fusion and RWM management vehicle application 212.

[0050] The positioning engine vehicle application 206 can receive data from various sensors and process the data to determine the location of the vehicle 100. The various sensors may include, but are not limited to, GPS sensors, IMUs, and / or other sensors connected via a CAN bus. The positioning engine vehicle application 206 may also utilize input from one or more cameras, such as cameras (e.g., 122, 136) and / or any other available sensors, such as radar, lidar, etc.

[0051] The map fusion and arbitration vehicle application 208 may access data within a high definition (HD) map database and receive output received from the positioning engine vehicle application 206 and process the data to further determine the location of the vehicle 100 within the map, such as a location within a traffic lane, a location within a street map, etc. The HD map database may be stored in a memory (e.g., memory 166). For example, the map fusion and arbitration vehicle application 208 may convert latitude and longitude information from a GPS into a location within a ground road map contained in the HD map database. The GPS positioning position includes errors, so the map fusion and arbitration vehicle application 208 may be used to determine the best guess position of the vehicle 100 within the road based on arbitration between the GPS coordinates and the HD map data. For example, although the GPS coordinates may place the vehicle 100 near the middle of a two-lane road in the HD map, the map fusion and arbitration vehicle application 208 may determine based on the direction of travel that the vehicle 100 is most likely aligned with a driving lane consistent with the direction of travel. The map fusion and arbitration vehicle application 208 may pass the map-based location information to the sensor fusion and RWM management vehicle application 212.

[0052] The route planning vehicle application 210 can utilize the HD map and input from the operator or dispatcher to plan a route that the vehicle 100 will follow to a specific destination. The route planning vehicle application 210 can pass the map-based location information to the sensor fusion and RWM management vehicle application 212. However, other vehicle applications (such as the sensor fusion and RWM management vehicle application 212, etc.) do not require the use of a priori maps. For example, other stacks can operate and / or control the vehicle based solely on sensory data without a provided map, building concepts of lanes, boundaries, and local maps as the sensory data is received.

[0053] The sensor fusion and RWM management vehicle application 212 may receive data and output generated by one or more of the radar perception vehicle application 202, the camera perception vehicle application 204, the map fusion and arbitration vehicle application 208, and the route planning vehicle application 210, and use some or all of such inputs to estimate or refine the position and state of the vehicle 100 relative to the road, other vehicles on the road, and other objects near the vehicle 100. For example, the sensor fusion and RWM management vehicle application 212 may combine image data from the camera perception vehicle application 204 with arbitrated map position information from the map fusion and arbitration vehicle application 208 to refine the determined positioning of the vehicle within the traffic lane. As another example, the sensor fusion and RWM management vehicle application 212 may combine object recognition and image data from the camera perception vehicle application 204 with object detection and ranging data from the radar perception vehicle application 202 to determine and refine the relative positioning of other vehicles and objects near the vehicle. As another example, the sensor fusion and RWM management vehicle application 212 may receive information about the location and direction of travel of other vehicles from vehicle-to-vehicle (V2V) communications (such as via a CAN bus) and combine this information with information from the radar perception vehicle application 202 and the camera perception vehicle application 204 to refine the location and motion of the other vehicles. The sensor fusion and RWM management vehicle application 212 may output the refined location and state information of the vehicle 100 and the refined location and state information of other vehicles and objects in the vicinity of the vehicle to the motion planning and control vehicle application 214 and / or the behavior planning and prediction vehicle application 216.

[0054] As a further example, the sensor fusion and RWM management vehicle application 212 can use dynamic traffic control instructions to guide the vehicle 100 to change speed, lane, direction of travel, or other navigation elements, and combine this information with other received information to determine refined position and state information. The sensor fusion and RWM management vehicle application 212 can output the refined position and state information of the vehicle 100 and the refined position and state information of other vehicles and objects in the vicinity of the vehicle 100 to the motion planning and control vehicle application 214, the behavior planning and prediction vehicle application 216 and / or devices remote from the vehicle 100, such as data servers, other vehicles, etc., via wireless communication (such as through a C-V2X connection, other wireless connections, etc.).

[0055] As yet a further example, the sensor fusion and RWM management vehicle application 212 may monitor perception data from various sensors (such as perception data from the radar perception vehicle application 202, the camera perception vehicle application 204, other perception vehicle applications, etc.) and / or data from one or more sensors themselves to analyze conditions in the vehicle sensor data. The sensor fusion and RWM management vehicle application 212 may be configured to detect sensor data conditions, such as sensor measurements being at, above, or below thresholds, certain types of sensor measurements occurring, etc., and may output the sensor data as part of the refined location and state information of the vehicle 100 that is provided to the behavior planning and prediction vehicle application 216 and / or to devices (such as data servers, other vehicles, etc.) remote from the vehicle 100 via wireless communications (such as through a C-V2X connection, other wireless connections, etc.).

[0056] Detailed location and status information may include vehicle descriptors associated with the vehicle 100 and the vehicle owner and / or operator, such as: vehicle specifications (e.g., size, weight, color, onboard sensor types, etc.); vehicle location, speed, acceleration, direction of travel, attitude, orientation, destination, fuel / power level, and other status information; vehicle emergency status (e.g., whether the vehicle is an emergency vehicle or a private individual is in an emergency); vehicle restrictions (e.g., heavy / wide loads, turning restrictions, high occupancy vehicle (HOV) authorizations, etc.); vehicle capabilities (e.g., all-wheel drive, four-wheel drive, snow tires, chains, supported connection types, onboard sensor operating status, onboard sensor resolution level, etc.); equipment issues (e.g., low tire pressure, weak brakes, sensor failure, etc.); owner / operator travel preferences (e.g., preferred lanes, roads, routes and / or destinations, preference for avoiding tolls or highways, preference for fastest routes, etc.); permission to provide sensor data to a data proxy server (e.g., 184); and / or owner / operator identification information.

[0057] The behavior planning and prediction vehicle application 216 of the autonomous vehicle system 200 can use the refined position and state information of the vehicle 100 and the position and state information of other vehicles and objects output from the sensor fusion and RWM management vehicle application 212 to predict the future behavior of other vehicles and / or objects. For example, the behavior planning and prediction vehicle application 216 can use such information to predict the future relative position of other vehicles based on its own vehicle position and speed and the position and speed of other vehicles in the vicinity of the vehicle. Such predictions can take into account information from HD maps and route planning to anticipate changes in relative vehicle positioning as the host vehicle and other vehicles travel along the road. The behavior planning and prediction vehicle application 216 can output other vehicle and object behavior and position predictions to the motion planning and control vehicle application 214.

[0058] In addition, the behavior planning and prediction vehicle application 216 can use the object behavior in conjunction with the position prediction to plan and generate control signals for controlling the movement of the vehicle 100. For example, based on the route planning information, the refined position in the road information, and the relative position and movement of other vehicles, the behavior planning and prediction vehicle application 216 can determine that the vehicle 100 needs to change lanes and accelerate, such as to maintain or achieve a minimum spacing with other vehicles, and / or to prepare for a turn or exit. As a result, the behavior planning and prediction vehicle application 216 can calculate or otherwise determine the steering angle of the wheels and the change to the throttle setting, which will be commanded to the motion planning and control vehicle application 214 and the DBW system / control unit 220 along with various such parameters necessary to achieve such lane change and acceleration. One such parameter can be a calculated steering wheel command angle.

[0059] The motion planning and control vehicle application 214 may receive data and information output from the sensor fusion and RWM management vehicle application 212, and other vehicle and object behavior and position predictions from the behavior planning and prediction vehicle application 216, and use the information to plan and generate control signals for controlling the motion of the vehicle 100, and to verify that such control signals meet the safety requirements of the vehicle 100. For example, based on the route planning information, the refined position in the road information, and the relative position and motion of other vehicles, the motion planning and control vehicle application 214 may verify and pass various control commands or instructions to the DBW system / control unit 220.

[0060] The DBW system / control unit 220 may receive these commands or instructions from the motion planning and control vehicle application 214 and translate such information into mechanical control signals for controlling the wheel angles, brakes, and throttle of the vehicle 100. For example, the DBW system / control unit 220 may respond to the calculated steering wheel command angle by sending corresponding control signals to a steering wheel controller.

[0061] In various embodiments, the vehicle management system 200 may include functionality for performing safety checks or supervision of various commands, plans, or other decisions of various vehicle applications that may affect the safety of the vehicle and passengers. Such safety checks or supervision functionality may be implemented within a dedicated vehicle application or distributed between various vehicle applications and included as part of the functionality. In some embodiments, various safety parameters may be stored in a memory, and the safety check or supervision functionality may compare the determined values ​​(e.g., relative spacing with nearby vehicles, distance from the centerline of the road, etc.) with the corresponding safety parameters, and issue a warning or command in the event of a violation or violation of the safety parameters. For example, the safety or supervision functionality in the behavior planning and prediction vehicle application 216 (or in a separate vehicle application) may determine the current or future spacing between another vehicle (such as refined by the sensor fusion and RWM management vehicle application 212) and the vehicle 100 (e.g., based on a world model refined by the sensor fusion and RWM management vehicle application 212), compare the spacing with the safety spacing parameters stored in the memory, and issue instructions to the motion planning and control vehicle application 214 to accelerate, decelerate, or turn when the current or predicted spacing violates the safety spacing parameters. As another example, safety or supervisory functionality in the motion planning and control vehicle application 214 (or in a separate vehicle application) may compare a determined or commanded steering wheel command angle to a safe steering wheel angle limit or parameter and issue an override command and / or alert in response to the commanded angle exceeding the safe steering wheel angle limit.

[0062] Some safety parameters stored in memory may be static (i.e., do not change over time), such as maximum vehicle speed. Other safety parameters stored in memory may be dynamic in that they are continuously or periodically determined or updated based on vehicle state information and / or environmental conditions. Non-limiting examples of safety parameters include: maximum safe speed, maximum brake pressure, maximum acceleration, and safe steering wheel angle limits, all of which may vary depending on road and weather conditions.

[0063] Figure 2B Examples of vehicle applications, subsystems, computing elements, or units within the vehicle management system 250 that may be utilized within the vehicle 100 are illustrated. Figures 1A to 2B In some embodiments, the vehicle applications 202, 204, 206, 208, 210, 212, and 216 of the vehicle management system 200 may be similar to those of reference Figure 2A The vehicle management system 250 may be similar to the vehicle management system 200, except that the vehicle management system 250 may pass various data or instructions to the vehicle safety and collision avoidance system 252 instead of the DBW system / control unit 220. For example, Figure 2B The configuration of the vehicle management system stack 250 and the vehicle safety and collision avoidance system 252 illustrated in FIG. 2 may be used in a non-autonomous vehicle.

[0064] In various embodiments, the behavior planning and prediction vehicle application 216 and / or the sensor fusion and RWM management vehicle application 212 may output data to the vehicle safety and collision avoidance system 252. For example, the sensor fusion and RWM management vehicle application 212 may output sensor data as part of the refined position and state information of the vehicle 100 provided to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 may use the refined position and state information of the vehicle 100 to make safety determinations about the vehicle 100 and / or the occupants of the vehicle 100. As another example, the behavior planning and prediction vehicle application 216 may output behavior models and / or predictions related to the movement of other vehicles to the vehicle safety and collision avoidance system 252. The vehicle safety and collision avoidance system 252 may use the behavior models and / or predictions related to the movement of other vehicles to make safety determinations about the vehicle 100 and / or the occupants of the vehicle 100.

[0065] In various embodiments, the vehicle safety and collision avoidance system 252 may include functionality to perform safety checks or oversight of various commands, plans or other decisions of various vehicle applications and human driver actions that may affect the safety of the vehicle and its occupants. In some embodiments, various safety parameters may be stored in memory, and the vehicle safety and collision avoidance system 252 may compare determined values ​​(e.g., relative spacing to nearby vehicles, distance to the centerline of the road, etc.) to corresponding safety parameters and issue warnings or commands if the safety parameters are or will be violated. For example, the vehicle safety and collision avoidance system 252 may determine the current or future spacing (e.g., based on a world model refined by the sensor fusion and RWM management vehicle application 212) between another vehicle (as refined by the sensor fusion and RWM management vehicle application 212) and the vehicle, compare the spacing to the safety spacing parameters stored in memory, and issue instructions to the driver to accelerate, decelerate, or turn if the current or predicted spacing violates the safety spacing parameters. As another example, the vehicle safety and collision avoidance system 252 may compare a human driver's steering wheel angle changes to safe steering wheel angle limits or parameters and issue an override command and / or alert in response to the steering wheel angle exceeding the safe steering wheel angle limits.

[0066] As indicated above, different modes of data received from different sensors can be fused into a single data modality that can provide more information about the environment than can be obtained from a single type of sensor. In some cases, one or more machine learning techniques can be used as part of sensor fusion.

[0067] A neural network is an example of a machine learning system, and a neural network may include an input layer, one or more hidden layers, and an output layer. Data is provided from the input nodes of the input layer, processed by the hidden nodes of one or more hidden layers, and output is generated by the output nodes of the output layer. A deep learning network typically includes multiple hidden layers. Each layer of a neural network may include a feature map or activation map, which may include artificial neurons (or nodes). Feature maps may include filters, kernels, etc. A node may include one or more weights to indicate the importance of a node in one or more layers in the layer. In some cases, a deep learning network may have a series of many hidden layers, where early layers are used to determine simple and low-level features of the input, while later layers establish a hierarchy of more complex and abstract features.

[0068] A deep learning architecture can learn a hierarchy of features. For example, if presented with visual data, the first layer can learn to recognize relatively simple features in the input stream, such as edges. In another example, if presented with auditory data, the first layer can learn to recognize spectral power in specific frequencies. The second layer, taking the output of the first layer as input, can learn to recognize combinations of features, such as simple shapes for visual data or combinations of sounds for auditory data. For example, higher layers can learn to represent complex shapes in visual data or words in auditory data. Even higher layers can learn to recognize common visual objects or spoken phrases.

[0069] Deep learning architectures perform particularly well when applied to problems that have a natural hierarchical structure. For example, the classification of motorized vehicles can benefit from first learning to recognize wheels, windshields, and other features. These features can be combined in different ways at higher levels to identify cars, trucks, and airplanes.

[0070] Neural networks can be designed with a variety of connection patterns. In a feedforward network, information is passed from lower layers to higher layers, with each neuron in a given layer communicating with neurons in higher layers. As described above, hierarchical representations can be built in successive layers of a feedforward network. Neural networks can also have recurrent or feedback (also known as top-down) connections. In a recurrent connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. Recurrent architectures can help identify patterns that span more than one block of input data that is sequentially delivered to the neural network. Connections from neurons in a given layer to neurons in lower layers are called feedback (or top-down) connections. A network with many feedback connections can be helpful when the recognition of high-level concepts can assist in discerning specific low-level features of the input. The connections between layers of a neural network can be fully connected, or locally connected. Various examples of neural network architectures are described below in detail with respect to FIG. 3A to FIG. 4 describe.

[0071] Some sensor fusion systems utilize neural networks or other machine learning systems to fuse data of different modalities, such as image data, radar data, and lidar data. Neural networks can be designed to have a variety of connection patterns. In a feedforward network, information is passed from a lower layer to a higher layer, where each neuron in a given layer communicates with a neuron in a higher layer. As described above, hierarchical representations can be constructed in successive layers of a feedforward network. Neural networks can also have loops or feedback (also known as top-down) connections. In a loop connection, the output from a neuron in a given layer can be communicated to another neuron in the same layer. The loop architecture can help identify patterns that span more than one input data block in the input data blocks that are delivered to the neural network in sequence. The connection from a neuron in a given layer to a neuron in a lower layer is called a feedback (or top-down) connection. When the recognition of high-level concepts can assist in discerning specific low-level features of the input, a network with many feedback connections may be helpful.

[0072] The connections between neural network layers can be fully connected or partially connected. Figure 3A An example of a fully connected neural network 302 is illustrated. In a fully connected neural network 302, a neuron in a first layer may communicate its output to every neuron in a second layer, such that every neuron in the second layer will receive input from every neuron in the first layer. Figure 3B An example of a locally connected neural network 304 is illustrated. In the locally connected neural network 304, neurons in a first layer may be connected to a limited number of neurons in a second layer. More generally, the locally connected layers of the locally connected neural network 304 may be configured such that each neuron in the layer will have the same or similar connectivity pattern, but the connection strengths may have different values ​​(e.g., 310, 312, 314, and 316). The connectivity patterns of the local connections may produce spatially different receptive fields in higher layers because higher layer neurons in a given region may receive input that is tuned through training to the characteristics of a limited portion of the total input to the network.

[0073] An example of a locally connected neural network is a convolutional neural network. Figure 3C An example of a convolutional neural network 306 is illustrated. The convolutional neural network 306 can be configured such that the connection strengths associated with the inputs of each neuron in the second layer are shared (e.g., 308). Convolutional neural networks may be well suited for problems where the spatial location of the inputs is meaningful. According to aspects of the present disclosure, the convolutional neural network 306 can be used to perform one or more aspects of video compression and / or decompression.

[0074] One type of convolutional neural network is the deep convolutional network (DCN). Figure 3DA detailed example of a DCN 300 is illustrated, which is designed to recognize visual features from an image 326 input from an image capture device 330 (such as a vehicle-mounted camera). The DCN 300 of the current example can be trained to identify traffic signs and numbers provided on traffic signs. Of course, the DCN 300 can be trained for other tasks such as identifying lane markings or identifying traffic lights.

[0075] The DCN 300 may be trained using supervised learning. During training, an image (such as an image 326 of a speed limit sign) may be presented to the DCN 300, and a forward pass may then be calculated to produce an output 322. The DCN 300 may include a feature extraction portion and a classification portion. Upon receiving the image 326, a convolution layer 332 may apply a convolution kernel (not shown) to the image 326 to generate a first set of feature maps 318. As an example, the convolution kernel of the convolution layer 332 may be a 5x5 kernel that generates a 28x28 feature map. In this example, because four different feature maps are generated in the first set of feature maps 318, four different convolution kernels are applied to the image 326 at the convolution layer 332. A convolution kernel may also be referred to as a filter or a convolution filter.

[0076] The first set of feature maps 318 may be subsampled by a max pooling layer (not shown) to generate a second set of feature maps 320. The max pooling layer reduces the size of the first set of feature maps 318. That is, the size of the second set of feature maps 320 (such as 14x14) is smaller than the size of the first set of feature maps 318 (such as 28x28). The reduced size provides similar information to subsequent layers while reducing memory consumption. The second set of feature maps 320 may be further convolved via one or more subsequent convolutional layers (not shown) to generate one or more sets of subsequent feature maps (not shown).

[0077] exist Figure 3D In the example of , the second set of feature maps 320 is convolved to generate a first feature vector 324. In addition, the first feature vector 324 is further convolved to generate a second feature vector 328. Each feature of the second feature vector 328 may include a number corresponding to a possible feature of the image 326, such as "sign", "60", and "100". A softmax function (not shown) may convert the numbers in the second feature vector 328 into probabilities. Therefore, the output 322 of the DCN 300 is the probability that the image 326 includes one or more features.

[0078] In this example, the probability of "logo" and "60" in output 322 is higher than the probability of other numbers (such as "30", "40", "50", "70", "80", "90", and "100") in output 322. Before training, the output 322 produced by DCN 300 may be incorrect. Therefore, the error between the output 322 and the target output can be calculated. The target output is the ground truth of image 326 (e.g., "logo" and "60"). The weights of DCN 300 can then be adjusted so that the output 322 of DCN 300 is more closely aligned with the target output.

[0079] To adjust the weights, the learning algorithm may calculate a gradient vector for the weights. The gradient may indicate the amount by which the error will increase or decrease when the weights are adjusted. At the top layer, the gradient may directly correspond to the value of the weights connecting the activated neurons in the penultimate layer and the neurons in the output layer. In lower layers, the gradient may depend on the value of the weights and the calculated error gradients of the higher layers. The weights may then be adjusted to reduce the error. This way of adjusting weights may be referred to as "backward propagation" because it involves a "backward pass" through the neural network.

[0080] In practice, the error gradient of the weights can be calculated over a small number of examples so that the calculated gradient is close to the true error gradient. This approximation method can be called stochastic gradient descent. Stochastic gradient descent can be repeated until the achievable error rate of the entire system stops decreasing or until the error rate reaches a target level. After learning, new images can be presented to the DCN, and a forward pass through the network can produce an output 322 that can be considered an inference or prediction of the DCN.

[0081] A Deep Belief Network (DBN) is a probabilistic model that includes multiple layers of hidden nodes. A DBN can be used to extract a hierarchical representation of a training dataset. A DBN can be obtained by stacking layers of Restricted Boltzmann Machines (RBMs). RBMs are a type of artificial neural network that can learn a probability distribution over a set of inputs. Because RBMs can learn a probability distribution without information about the class to which each input should be classified, RBMs are often used for unsupervised learning. Using a hybrid unsupervised and supervised paradigm, the bottom RBM of a DBN can be trained in an unsupervised manner and can be used as a feature extractor, while the top RBM can be trained in a supervised manner (on the joint distribution of inputs and target classes from the previous layer) and can be used as a classifier.

[0082] A deep convolutional network (DCN) is a network that is a convolutional network configured with additional pooling and normalization layers. DCNs have achieved state-of-the-art performance on many tasks. DCNs can be trained using supervised learning, where both the input targets and the output targets are known for many examples and are used to modify the weights of the network by using a gradient descent method.

[0083] The DCN may be a feed-forward network. In addition, as described above, the connections from a neuron in the first layer of the DCN to a set of neurons in the next higher layer are shared across the neurons in the first layer. The feed-forward and shared connections of the DCN may be used for fast processing. For example, the computational burden of the DCN may be much smaller than that of a similarly sized neural network that includes loops or feedback connections.

[0084] The processing of each layer of the convolutional network can be thought of as a spatially invariant template or basis projection. If the input is first decomposed into multiple channels, such as the red, green, and blue channels of a color image, then the convolutional network trained on this input can be thought of as three-dimensional, where two spatial dimensions are along the axes of the image and the third dimension captures color information. The output of the convolutional connection can be thought of as forming a feature map in a subsequent layer, where each element in the feature map (e.g., feature map 320) receives input from a range of neurons in the previous layer (e.g., feature map 318) and from each of the multiple channels. The values ​​in the feature map can be further processed with nonlinearities (such as rectification, max(0,x)). The values ​​from neighboring neurons can be further pooled, which corresponds to downsampling and can provide additional local invariance and dimensionality reduction.

[0085] Figure 4 4 is a block diagram illustrating an example of a deep convolutional network 450. The deep convolutional network 450 may include multiple different types of layers based on connectivity and weight sharing. Figure 4 As shown, the deep convolution network 450 includes convolution blocks 454A and 454B. Each of the convolution blocks 454A and 454B may be configured with a convolution layer (CONV) 456, a normalization layer (LNorm) 458, and a maximum pooling layer (MAX POOL) 460.

[0086] The convolution layer 456 may include one or more convolution filters that may be applied to the input data 452 to generate a feature map. Although only two convolution blocks 454A, 454B are shown, the present disclosure is not limited thereto, and any number of convolution blocks (e.g., convolution blocks 454A, 454B) may be included in the deep convolution network 450 according to design preferences. The normalization layer 458 may normalize the output of the convolution filter. For example, the normalization layer 458 may provide whitening or lateral suppression. The maximum pooling layer 460 may provide spatial downsampling aggregation to achieve local invariance and dimensionality reduction.

[0087] For example, the parallel filter bank of the deep convolutional network may be loaded onto the CPU 110 or GPU 115 of the SOC 105 to achieve high performance and low power consumption. In an alternative embodiment, the parallel filter bank may be loaded onto the DSP 106 or ISP 175 of the SOC 105. In addition, the deep convolutional network 450 may access other processing blocks that may be present on the SOC 105, such as the sensor processor 155 and the navigation module 195 dedicated to sensors and navigation, respectively.

[0088] The deep convolutional network 450 may also include one or more fully connected layers, such as layer 462A (labeled "FC1") and layer 462B (labeled "FC2"). The deep convolutional network 450 may also include a logistic regression (LR) layer 464. Between each layer 456, 458, 460, 462A, 462B, 464 of the deep convolutional network 450 are weights (not shown) to be updated. The output of each of these layers (e.g., 456, 458, 460, 462A, 462B, 464) may serve as the input of a subsequent layer of these layers (e.g., 456, 458, 460, 462A, 462B, 464) of the deep convolutional network 450 to learn a hierarchical feature representation from the input data 452 (e.g., image, audio, video, sensor data, and / or other input data) provided at the initial convolutional block 454A. The output of the deep convolutional network 450 is a classification score 466 for the input data 452. The classification score 466 may be a set of probabilities, where each probability is a probability that the input data includes a feature from a set of features.

[0089] Generally speaking, it is relatively easy to estimate the height of a detected object because the height of an object above ground level rarely changes and is relatively predictable for the class of the object. In some cases, the height of certain objects may even be limited by regulations. The height of relatively flat objects or areas above ground level (such as streets, lane markings, etc.) can be easily estimated to be zero. This relative predictability can be useful when combined with known camera characteristics to estimate the distance of an object based on, for example, how far the object is from the bottom edge.

[0090] Figure 5 A system 500 for processing images for use by a device (eg, a vehicle or a computing system of a vehicle, a mobile device, an extended reality (XR) device, etc.) according to aspects of the present disclosure is illustrated. Figure 5As shown, input camera data 502 (e.g., images captured by a camera such as cameras 122, 136) may be input to a feature extractor 504. Feature extractor 504 may be used to identify certain features in the camera data. In some cases, feature extractor 504 may be ML-based. For example, feature extractor may be implemented as one or more neural networks (NNs), convolutional networks, or deep convolutional networks, such as Figure 4 As shown. As an example, the feature extractor 504 may include one or more layers or transformer blocks, which may include feature maps for identifying certain features. The feature extractor 504 may output the identified features as intermediate camera features 506. Note that although shown as line drawings, the intermediate camera features 506 may be output as a set of feature vectors.

[0091] In some cases, one or more transformation operations 508 may be performed on the intermediate camera features 506. For example, the one or more transformation operations 508 may perform segmentation 510 based on the intermediate camera features 506, perform object detection 512 based on the intermediate camera features 506, or perform a perspective transformation 514 to convert the intermediate camera features 506 from, for example, a front view of the vehicle to a BEV projected camera feature as if the feature were generated based on a camera positioned above the vehicle. It should be understood that the listed transformations are not intended to be limiting and that many other transformation operations may be performed. In some cases, the one or more operations 508 may be ML based. For example, each of the one or more transformation operations 508 may be implemented as one or more neural networks (NNs), convolutional networks, or deep convolutional networks, such as Figure 4 shown.

[0092] In some cases, sensor data from multiple sensors (such as cameras, radar, and lidar sensors) may be fused to generate a semantically segmented BEV. In some examples, this sensor fusion may be performed based on the intermediate camera features 506. Three-dimensional (3D) information may be useful for generating a segmented BEV because information about how far away an object is (e.g., depth information) may help separate the object from the background. However, sensors that directly obtain depth information (such as radar and lidar sensors) may be expensive and may significantly increase the cost of the vehicle. In addition, sensors that directly obtain depth information may draw a lot of power, and techniques that may allow for reducing the use of such sensors may help reduce the power consumption of the vehicle. Techniques for deriving depth information using a single camera that provides a two-dimensional (2D) image may be used to help reduce costs, for example, by reducing or eliminating reliance on sensors that can directly obtain depth information.

[0093] Generally speaking, it is easier to estimate the height of an object than the distance of the object from the viewpoint. For example, for ground-based vehicles that do not need to consider flying objects, the height of the object above the ground plane is image-localization-invariant and scale-variant. That is, no matter where the object is in the image, the height of the object relative to the ground plane does not change, but when the object is farther away from the viewpoint, the size of the object-ground plane relationship changes. Therefore, the roof of the car is about 1.8 meters above the ground, and this has nothing to do with where the roof of the car is in the image. These characteristics can be used to provide accurate pseudo-depth information from captured 2D images (such as those produced by a monocular camera). As an example, for generating predictions, some ML models such as certain types of neural networks may be more able to reliably handle scale changes (e.g., using stride / pooling) than positioning invariance.

[0094] To allow the ML model to predict the distance of features based on height, geometric distance information may be passed into the ML model, for example, during a training process. The geometric distance information may be based on geometric properties of images produced by a camera of the vehicle. The geometric properties of the images may be affected by camera properties, such as the positioning of the camera on the vehicle, the focal length of the camera and lens, distortion from the camera and lens, etc., and the geometric distance information may be provided based on known camera properties. For example, the geometric distance information may be passed into the ML model using distance map images, and these distance map images may be generated based on known geometric properties of images produced by a camera of the vehicle.

[0095] In some cases, additional geometric distance information may be provided based on predicted changes in camera characteristics. For example, the height of the camera may change due to vehicle suspension movement, and during inference, multiple sets of distance map images may be provided as input to the ML model based on the possible movement of the camera. In some cases, the multiple sets of distance map images may also be provided as input to the ML model along with height information from the vehicle suspension to help compensate for this movement. In some cases, the camera calibration matrix may include camera characteristics for a range of cameras, mounting positions, and camera movement to help compensate for different mounting positions in different vehicles, different vehicle pitch angles, etc.

[0096] Figure 6 A set of range map images 600 based on known camera characteristics in accordance with aspects of the present disclosure is illustrated. Figure 6Four distance map images 600 are shown for objects located at four different heights above the ground. A first distance map 602 illustrates the distances of objects detected in the image that are at ground level (e.g., at ground plane) (such as streets, lane markings, curbs, grass, etc.). A second distance map 604 illustrates the distances of objects that are half a meter above the ground in this example (such as headlights, wheels, obstacles, etc.). A third distance map 606 illustrates the distances of objects that are approximately one meter above the ground (such as windshields, bushes, etc.). A fourth distance map 608 illustrates the distances of objects that are approximately two meters above the ground (such as signs, trucks, etc.). Note that while shown as a two-dimensional representation in this example, in some cases the distance map image can be a one-dimensional representation (e.g., an array of values) indicating distances relative to the height dimension.

[0097] In some cases, each distance map indicates how far away objects at a corresponding height above the ground are from the camera based on the relative height of the object in the image (e.g., how far from the bottom or top of the image). As an example of how the distance maps may be used, if an image includes a detected object 610 at ground level, such as a manhole cover, which is approximately one-third of the way from the bottom in the image, as shown superimposed on the first distance map 602, it may be determined that the object is 10 meters away from the camera in this example. Objects located farther from the bottom of the image may be determined to be farther from the camera. Although in Figure 4 Four distance map images 600 are illustrated in FIG. 6 , but it should be understood that more distance map images may be used and that these distance map images may be based on known (and / or predicted) camera characteristics.

[0098] Figure 7 7 illustrates a system 700 for predicting the distance of a detected object according to aspects of the present disclosure. In some cases, system 700 is similar to system 500. Figure 7As shown, received camera data 502 (e.g., images captured by cameras such as cameras 122, 136) may be input to a feature extractor 504, which may be used to identify certain features in the camera data. These feature extractors 504 may be ML-based and may include feature maps for identifying certain features. The feature extractor 504 may output the identified features as intermediate camera features 506. In some cases, the feature extractor 504 may have been trained on the relative heights of the identified features. For example, the feature extractor 504 may recognize that certain features may represent objects such as a windshield or a wheel, and that the combination of these features forms another object such as a car. Similarly, the feature extractor 504 may have been trained to understand that a wheel may generally be centered about 0.25 meters above ground level or that the base of a windshield may generally be about 1.1 meters above ground level. In some cases, the intermediate camera features 506 may include an indication of these estimated heights of one or more features or objects. In some cases, the feature extractor 504 may also perform object detection. The feature extractor 504 may implicitly learn the real-world positioning of objects on a ground level (eg, a plane).

[0099] The intermediate camera features 506 may be combined 702 with a set of distance maps 704 to generate combined feature distance information. In some cases, the set of distance maps 704 may be combined with Figure 6 Representation of a distance map image depicted. In some cases, the intermediate camera features 506 can be combined 702 with a set of distance maps 704 by adding, multiplying, concatenating, applying an attention mechanism, etc., to the distance maps 704 and the intermediate camera features 506.

[0100] In some cases, the combined feature distance information may be input to a feature height mapper 706. The feature height mapper 706 may be ML-based. For example, the feature height mapper may be implemented as one or more neural networks (NNs), convolutional networks, or deep convolutional networks, such as Figure 4 In some cases, the ML model of feature height mapper 706 may associate certain features of intermediate camera features 506 with height information (e.g., distance above ground level such as relative to the ground plane). For example, the transformation operation may learn that certain intermediate features of intermediate camera features 506 (such as features that may describe wheels) are typically located about half a meter above ground level (or ground plane).

[0101] As another example, the feature height mapper 706 may be trained to estimate the distance of the intermediate camera feature 506 based on the set of distance maps 704 and the height information associated with the intermediate camera feature 506. For example, the feature height mapper 706 may select a distance map from the set of distance maps 704 based on the height information associated with the feature or object. Based on the selected distance map, the feature height mapper 706 may then map the feature or object on the selected distance map to estimate the distance of the feature or object. The potential representation 708 illustrates depth map information of the image, as encoded by the feature height mapper 706. Although shown as an image in this example, in some cases, the potential representation 708 may not be an image or directly represented as an image. For example, the depth information may be output as metadata, a bitstream, or the depth information may be output for less than all detected features or objects. In some examples, the potential representation 708 may include depth information as well as information about image features.

[0102] In some cases, one or more convolutional layers (e.g., operations) can be used to generate associations between intermediate features and height information. In some cases, pooling layers can be used with various strides along with downsampling to help improve scale invariance. For example, pooling layers can help summarize features generated by convolutional layers for downsizing and pooling. Different stride settings can vary the amount of feature summarization, which can be useful when downsizing across various amounts.

[0103] The information represented by the latent representation 708 may be input to one or more transformation operations 710. In some cases, the one or more transformation operations 710 may be ML-based. For example, each of the one or more transformation operations 710 may be implemented as one or more neural networks (NNs), convolutional networks, or deep convolutional networks, such as Figure 4 As shown. One or more transformation operations 710 may perform any of a variety of operations on the information represented by the potential representation 708 to generate output information for use by the vehicle. For example, one or more transformation operations 710 may generate a segmented depth map 712 of the scene, generate a segmented BEV 714, or perform 3D object detection 716. In some cases, the information represented by the potential representation 708 may not be generated. For example, the operation of the feature height mapper 706 and the transformation operation 710 may be combined to directly generate (as an example) a segmented BEV without generating intermediate depth information as shown by the potential representation 708. In some cases, the generation of the information represented by the potential representation 708 can be enhanced by adding an auxiliary loss of explicit depth prediction. The auxiliary loss of depth prediction can help improve the quality of depth estimation for use in subsequent operations, such as improving the accuracy of BEV or 3D object detection.

[0104] Figure 8800 is a flow chart illustrating a process 800 for predicting the distance of a detected object according to aspects of the present disclosure. The process 800 may be performed by a computing device (or apparatus) or a component of a computing device (e.g., a chipset, a codec, etc.). The computing device may be a mobile device (e.g., a mobile phone), a network-connected wearable device such as a watch, an extended reality (XR) device such as a virtual reality (VR) device or an augmented reality (AR) device, a vehicle or a component or system of a vehicle, or other type of computing device. In some cases, the computing device may be included in or as part of a system, such as the vehicle management system 200. The operations of the process 800 may be implemented as a software component that executes and runs on one or more processors.

[0105] At block 802, a computing device (or a component thereof) may obtain one or more images of an environment. In some cases, the one or more images are obtained from a monocular camera. In some cases, one or more distance maps are based on camera characteristics of the monocular camera.

[0106] At block 804, the computing device (or a component thereof) may generate a set of features for the one or more images. In some cases, the computing device may use a machine learning model to generate depth information for the environment.

[0107] At block 806, the computing device (or a component thereof) may combine the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level. In some cases, the computing device may combine the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features. In some cases, the computing device may select a distance map based on an estimated height of a feature in the set of features.

[0108] At box 808, the computing device (or a component thereof) may generate depth information for the environment based on the combined feature distance information. At box 810, the network device (or a component thereof) may output the depth information for the environment. In some cases, the computing device may generate at least one of a segmented depth map of the environment, a segmented bird's eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information for the environment. In some cases, the computing device may perform a transformation operation based on the depth information for the environment and the set of features to generate a set of transformed features, and output the set of transformed features, wherein the depth information for the environment is implicitly output in the set of transformed features.

[0109] In some examples, the processes described herein (e.g., process 800 and / or other processes described herein) may be performed by a computing device or apparatus (e.g., SOC 105). In another example, process 800 may be performed by Figure 1A The vehicle 100 is executed.

[0110] Fig. 9 is a diagram illustrating an example of a system for implementing certain aspects of the present technology. Specifically, Fig. 9 An example of a computing system 900 is illustrated, which may be any computing device, for example, constituting an internal computing system, a remote computing system, a camera, or any components thereof, wherein the components of the system communicate with each other using a connection 905. The connection 905 may be a physical connection using a bus, or a direct connection into the processor 910, such as in a chipset architecture. The connection 905 may also be a virtual connection, a networked connection, or a logical connection.

[0111] In some embodiments, computing system 900 is a distributed system, wherein the functionality described in the present disclosure may be distributed within a data center, multiple data centers, a peer-to-peer network, etc. In some embodiments, one or more of the described system components represent a number of such components, each of which performs some or all of the functionality described for that component. In some embodiments, a component may be a physical device or a virtual device.

[0112] The example system 900 includes at least one processing unit (CPU or processor) 910 and connections 905 that communicatively couple various system components including system memory 915, such as read only memory (ROM) 920 and random access memory (RAM) 925, to the processor 910. The computing system 900 may include a cache 912 of high-speed memory directly connected to the processor 910, in close proximity to the processor, or integrated as part of the processor.

[0113] Processor 910 may include any general purpose processor and hardware or software services such as services 932, 934, and 936 stored in storage device 930 configured to control processor 910, as well as special purpose processors where software instructions are incorporated into the actual processor design. Processor 910 may essentially be a completely independent computing system containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0114] To enable user interaction, the computing system 900 includes an input device 945 that can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice, etc. The computing system 900 may also include an output device 935, which may be one or more of a plurality of output mechanisms. In some cases, a multimodal system may enable a user to provide multiple types of input / output to communicate with the computing system 900.

[0115] The computing system 900 may include a communication interface 940, which may generally govern and manage user input and system output. The communication interface may perform or facilitate receiving and / or sending wired or wireless communications using wired and / or wireless transceivers, including using an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple ™ Lightning ™ Ports / plugs, Ethernet ports / plugs, Fiber optic ports / plugs, Dedicated wired ports / plugs, 3G, 4G, 5G and / or other cellular data network wireless signal delivery, Bluetooth ™ Wireless signal transmission, Bluetooth ™ Low energy (BLE) wireless signal transmission, IBEACONTM wireless signal transmission, radio frequency identification (RFID) wireless signal transmission, near field communication (NFC) wireless signal transmission, dedicated short range communication (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, wireless local area network (WLAN) signal transmission, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transmission, public switched telephone network (PSTN) signal transmission, integrated services digital network (ISDN) signal transmission, ad hoc network signal transmission, radio wave signal transmission, microwave signal transmission, infrared signal transmission, visible light signal transmission, ultraviolet light signal transmission, wireless signal transmission along the electromagnetic spectrum, or some combination thereof. The communication interface 940 may also include one or more global navigation satellite system (GNSS) receivers or transceivers, which are used to determine the location of the computing system 900 based on receiving one or more signals from one or more satellites associated with one or more GNSS systems. GNSS systems include, but are not limited to, the United States' Global Positioning System (GPS), Russia's Global Navigation Satellite System (GLONASS), China's BeiDou Navigation Satellite System (BDS), and Europe's Galileo GNSS. There is no restriction to operating on any particular hardware arrangement, and thus the base features herein may be easily substituted for improved hardware or firmware arrangements as they are developed.

[0116] The storage device 930 may be a non-volatile and / or non-transitory and / or computer-readable memory device and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic cassette, a flash memory card, a solid-state memory device, a digital versatile disk, a cassette, a floppy disk, a floppy disk, a hard disk, a magnetic tape, a magnetic stripe / magnetic stripe, any other magnetic storage medium, flash memory, a memristor memory, any other solid-state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disc (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a Memory Stick ® card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (e.g., a layer 1 (L1) cache, a layer 2 (L2) cache, a layer 3 (L3) cache, a layer 4 (L4) cache, a layer 5 (L5) cache, other (L#) cache), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or box, and / or a combination thereof.

[0117] The storage device 930 may include software services, servers, services, etc., which, when the code defining such software is executed by the processor 910, causes the system to perform functions. In some embodiments, hardware services that perform specific functions may include software components for performing functions stored in a computer-readable medium connected to necessary hardware components such as the processor 910, the connector 905, the output device 935, etc. The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, containing or carrying instructions and / or data. Computer-readable media may include non-transitory media in which data may be stored and does not include carrier waves and / or transient electronic signals propagated wirelessly or through wired connections. Examples of non-transitory media may include, but are not limited to, disks or tapes, optical storage media (such as compact discs (CDs) or digital versatile discs (DVDs)), flash memory, memory, or memory devices. Computer readable media may store thereon code and / or machine executable instructions, which may represent a procedure, function, subprogram, program, routine, subroutine, module, software package, category, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, independent variables, parameters, or memory contents. Information, independent variables, parameters, data, etc. may be passed, forwarded, or sent via any suitable means, including memory sharing, message passing, token passing, network sending, etc.

[0118] Specific details are provided in the above description to provide a thorough understanding of each embodiment and each example provided herein, but those skilled in the art will recognize that the application is not limited thereto. Thus, although the exemplary embodiments of the present application have been described in detail herein, it is to be understood that the inventive concept can be embodied and adopted in various other ways, and the appended claims are intended to be interpreted as including such variations, unless limited by the prior art. Various features and aspects of the above-mentioned applications can be used individually or in combination. In addition, without departing from the broader scope of this specification, the embodiments can be used in any number of environments and applications beyond the environment and application described herein. Therefore, the description and the accompanying drawings should be considered as illustrative rather than restrictive. For the purpose of illustration, each method is described in a specific order. It should be understood that in an alternative embodiment, each method can be performed in a different order than described.

[0119] For clarity of explanation, in some instances, the present technology may be presented as including separate functional blocks, which include devices, device components, steps or routines in the method embodied in software or a combination of hardware and software. Additional components other than those components shown in the drawings and / or described herein may be used. For example, circuits, systems, networks, processes and other components may be shown as components in block diagram form to avoid these embodiments becoming obscure in unnecessary details. In other cases, known circuits, processes, algorithms, structures and techniques may be shown without necessary details to avoid making each embodiment obscure.

[0120] In addition, it will be appreciated by those skilled in the art that the various exemplary logic blocks, modules, circuits and algorithmic steps described in conjunction with the various aspects disclosed herein can be implemented as electronic hardware, computer software or a combination of the two. In order to clearly illustrate this interchangeability of hardware and software, various exemplary components, frames, modules, circuits and steps have been generally described above in terms of their functions. Whether such functions are implemented as hardware or software depends on the specific application and the design constraints proposed to the entire system. Those skilled in the art can implement the described functions in different ways for each specific application, but such specific implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0121] Individual embodiments may be described above as processes or methods depicted as flow charts, flow diagrams, data flow diagrams, structure diagrams, or block diagrams. Although a flow chart may describe an operation as a sequential process, many of the operations may be performed in parallel or concurrently. In addition, the order of the operations may be rearranged. The process is terminated when the operations of the process are completed, but the process may have additional steps not included in the accompanying drawings. A process may correspond to a method, function, procedure, subroutine, subprogram, etc. When a process corresponds to a function, the termination of the process may correspond to the function returning to the calling function or main function.

[0122] The processes and methods according to the above examples can be implemented using stored computer executable instructions or otherwise available computer executable instructions from a computer readable medium. Such instructions may include, for example, instructions and data that configure a general-purpose computer, a special-purpose computer, or a processing device to perform a certain function or group of functions. Portions of the computer resources used may be accessed via a network. Computer executable instructions may be, for example, binary, intermediate format instructions such as assembly language, firmware, source code. Examples of computer readable media that can be used to store instructions, information used, and / or information created during the methods according to the described examples include disks or optical disks, flash memories, USB devices with non-volatile memory, networked storage devices, etc.

[0123] In some embodiments, computer-readable storage devices, media, and memories may include wired or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly excludes media such as power consumption, carrier signals, electromagnetic waves, and signals themselves.

[0124] Those skilled in the art will appreciate that information and signals may be represented using any of a variety of different techniques and methods. For example, data, instructions, commands, information, signals, bits, symbols, and chips that may be mentioned throughout the above description may be represented by voltage, current, electromagnetic waves, magnetic fields or particles, optical fields or particles, or any combination thereof in some cases, depending in part on the specific application, in part on the desired design, in part on the corresponding technology, etc.

[0125] The various illustrative logic blocks, modules, and circuits described in conjunction with the various aspects disclosed herein may be implemented or executed using hardware, software, firmware, middleware, microcode, hardware description language, or any combination thereof, and may be implemented in any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing necessary tasks may be stored in a computer-readable or machine-readable medium. The processor may perform the necessary tasks. Examples of form factors include: laptops, smartphones, mobile phones, tablet devices, or other small form factor personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, etc. The functions described herein may also be embodied in peripheral devices or add-in cards. By way of further example, such functions may also be implemented on circuit boards in different chips or different processes executed on a single device.

[0126] Instructions, media for conveying such instructions, computing resources for executing them, and other structures for supporting such computing resources are example components for providing the functionality described in this disclosure.

[0127] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as general-purpose computers, wireless communication devices, or integrated circuit devices with multiple uses, including applications in wireless communication devices, mobile phones, and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately as discrete but interoperable logic devices. If implemented in software, the technology may be implemented at least in part by a computer-readable data storage medium including a program code, which includes instructions for executing one or more of the above methods, algorithms, and / or operations when executed. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM) (such as a synchronous dynamic random access memory (SDRAM)), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a flash memory, a magnetic or optical data storage medium, and the like. Additionally or alternatively, the techniques may be implemented at least in part by a computer-readable communication medium that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer, such as a propagated signal or wave.

[0128] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application-specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuits. Such a processor may be configured to perform any of the techniques described in the present disclosure. A general-purpose processor may be a microprocessor; however, in an alternative, the processor may be any conventional processor, controller, microcontroller, or state machine. The processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors combined with a DSP core, or any other such configuration. Therefore, the term "processor" as used herein may refer to any of the foregoing structures, any combination of the foregoing structures, or any other structure or device suitable for implementing the techniques described herein.

[0129] It should be understood by those skilled in the art that the less than ("<") symbol and greater than (">") symbol or terminology used herein may be replaced by less than or equal to (" ”) symbol and the greater than or equal to (“ ”) symbol instead.

[0130] Where a component is described as being “configured to” perform certain operations, such configuration may be achieved, for example, by designing electronic circuits or other hardware to perform the operations, by programming programmable electronic circuits (e.g., a microprocessor or other suitable electronic circuits) to perform the operations, or any combination thereof.

[0131] The phrases “coupled to” or “communicatively coupled to” refer to any component being physically connected directly or indirectly to another component, and / or any component being in communication with another component directly or indirectly (e.g., connected to the other component via a wired or wireless connection and / or other suitable communication interface).

[0132] Claim language or other language reciting "at least one of" a set and / or "one or more of" a set indicates that one member of the set or multiple members of the set (in any combination) satisfies the claim. For example, claim language reciting "at least one of A and B" or "at least one of A or B" means A, B, or A and B. In another example, claim language reciting "at least one of A, B, and C" or "at least one of A, B, or C" means A, B, C, or A and B, or A and C, or B and C, or any repetition is information or data (e.g., A and A, B and B, C and C, A and A and B, etc.), or any other ordering, repetition, or combination of A, B, and C. The language "at least one of" a set and / or "one or more of" a set does not limit the set to the items listed in the set. For example, claim language stating "at least one of A and B" or "at least one of A or B" may mean A, B, or A and B, and may additionally include items not listed in the set of A and B.

[0133] Illustrative aspects of the present disclosure include:

[0134] Aspect 1. A device for generating depth information, the device comprising: at least one memory, the at least one memory comprising instructions; and at least one processor, the at least one processor coupled to the memory and configured to: obtain one or more images of an environment; generate a set of features for the one or more images; combine the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generate depth information of the environment based on the combined feature distance information; and output the depth information of the environment.

[0135] Aspect 2. An apparatus according to aspect 1, wherein the one or more images are obtained from a monocular camera.

[0136] Aspect 3. An apparatus according to Aspect 2, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

[0137] Aspect 4. An apparatus according to any one of Aspects 1 to 3, wherein the at least one processor is further configured to generate the depth information of the environment using a machine learning model.

[0138] Aspect 5. An apparatus according to any one of Aspects 1 to 4, wherein the at least one processor is further configured to combine the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

[0139] Aspect 6. The apparatus according to any one of aspects 1 to 5, wherein the at least one processor is further configured to select a distance map based on estimated heights of features in the set of features.

[0140] Aspect 7. An apparatus according to any one of Aspects 1 to 6, wherein the at least one processor is further configured to generate at least one of a segmented depth map of the environment, a segmented bird's-eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

[0141] Aspect 8. An apparatus according to any one of Aspects 1 to 7, wherein the at least one processor is further configured to: perform a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and output the set of transformed features, wherein the depth information of the environment is implicitly output in the set of transformed features.

[0142] Aspect 9. A method for generating depth information, the method comprising: obtaining one or more images of an environment; generating a set of features for the one or more images; combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generating depth information of the environment based on the combined feature distance information; and outputting the depth information of the environment.

[0143] Aspect 10. The method according to aspect 9, wherein the one or more images are obtained from a monocular camera.

[0144] Aspect 11. The method according to aspect 10, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

[0145] Aspect 12. The method according to any one of Aspects 9 to 11, further comprising using a machine learning model to generate the depth information of the environment.

[0146] Aspect 13. The method according to any one of aspects 9 to 12, further comprising combining the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

[0147] Aspect 14. The method according to any one of aspects 9 to 13, further comprising selecting a distance map based on estimated heights of features in the set of features.

[0148] Aspect 15. The method according to any one of Aspects 9 to 14, further comprising generating at least one of a segmented depth map of the environment, a segmented bird's-eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

[0149] Aspect 16. The method according to any one of Aspects 9 to 15, further comprising performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and outputting the set of transformed features, wherein the depth information of the environment is implicitly output in the set of transformed features.

[0150] Aspect 17. A non-transitory computer-readable medium having instructions stored thereon, which, when executed by one or more processors, causes the one or more processors to: obtain one or more images of an environment; generate a set of features for the one or more images; combine the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generate depth information for the environment based on the combined feature distance information; and output the depth information for the environment.

[0151] Clause 18. The non-transitory computer-readable medium of Clause 17, wherein the one or more images are obtained from a monocular camera.

[0152] Aspect 19. The non-transitory computer-readable medium of aspect 18, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

[0153] Aspect 20. A non-transitory computer-readable medium according to any one of Aspects 17 to 19, wherein the instructions further cause the at least one processor to use a machine learning model to generate the depth information of the environment.

[0154] Aspect 21. A non-transitory computer-readable medium according to any one of Aspects 17 to 20, wherein the instructions further cause the at least one processor to combine the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

[0155] Aspect 22. The non-transitory computer-readable medium of any one of aspects 17 to 21, wherein the instructions further cause the at least one processor to select a distance map based on estimated heights of features in the set of features.

[0156] Aspect 23. A non-transitory computer-readable medium according to any one of Aspects 17 to 22, wherein the instructions further cause the at least one processor to generate at least one of a segmented depth map of the environment, a segmented bird's-eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

[0157] Aspect 24. A non-transitory computer-readable medium according to any one of Aspects 17 to 23, wherein the instructions further cause the at least one processor to: perform a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and output the set of transformed features, wherein the depth information of the environment is implicitly output in the set of transformed features.

[0158] Aspect 25. A device for generating depth information, the device comprising: a component for obtaining one or more images of an environment; a component for generating a set of features for the one or more images; a component for combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; a component for generating depth information of the environment based on the combined feature distance information; and a component for outputting the depth information of the environment.

[0159] Clause 26. The apparatus according to Clause 25, wherein the one or more images are obtained from a monocular camera.

[0160] Clause 27. An apparatus according to Clause 26, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

[0161] Aspect 28. The apparatus according to any one of Aspects 25 to 27, further comprising a component for generating the depth information of the environment using a machine learning model.

[0162] Aspect 29. An apparatus according to any one of Aspects 25 to 28, further comprising a component for combining the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

[0163] Aspect 30. The apparatus according to any one of aspects 25 to 29, further comprising means for selecting a distance map based on estimated heights of features in the set of features.

[0164] Aspect 31. An apparatus according to any one of Aspects 25 to 30, the apparatus further comprising a component for generating at least one of a segmented depth map of the environment, a segmented bird's-eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

[0165] Aspect 32. An apparatus according to any one of Aspects 25 to 31, the apparatus further comprising: a component for performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and a component for outputting the set of transformed features, wherein the depth information of the environment is implicitly output in the set of transformed features.

Claims

1. A device for generating depth information, the device comprising: at least one memory, the at least one memory comprising instructions; and at least one processor coupled to the at least one memory and configured to: obtaining one or more images of an environment; generating a set of features for the one or more images; combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generating depth information of the environment based on the combined feature distance information; as well as The depth information of the environment is output.

2. The apparatus of claim 1, wherein the one or more images are obtained from a monocular camera.

3. The apparatus of claim 2, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

4. The apparatus of claim 1, wherein the at least one processor is further configured to generate the depth information of the environment using a machine learning model.

5. The apparatus of claim 1, wherein the at least one processor is further configured to combine the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features. 6 . The apparatus of claim 1 , wherein the at least one processor is further configured to select a distance map based on estimated heights of features in the set of features.

7. The apparatus of claim 1, wherein the at least one processor is further configured to generate at least one of a segmented depth map of the environment, a segmented bird's eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

8. The apparatus of claim 1, wherein the at least one processor is further configured to: performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and The transformed set of features is output, wherein the depth information of the environment is implicitly output in the transformed set of features.

9. A method for generating depth information, the method comprising: obtaining one or more images of an environment; generating a set of features for the one or more images; combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generating depth information of the environment based on the combined feature distance information; as well as The depth information of the environment is output.

10. The method of claim 9, wherein the one or more images are obtained from a monocular camera.

11. The method of claim 10, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

12. The method of claim 9, further comprising using a machine learning model to generate the depth information of the environment.

13. The method of claim 9, further comprising combining the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

14. The method of claim 9, further comprising selecting a distance map based on estimated heights of features in the set of features.

15. The method of claim 9, further comprising generating at least one of a segmented depth map of the environment, a segmented bird's eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

16. The method according to claim 9, further comprising: performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; as well as The transformed set of features is output, wherein the depth information of the environment is implicitly output in the transformed set of features.

17. A non-transitory computer readable medium having stored thereon instructions which, when executed by at least one processor, cause the at least one processor to: obtaining one or more images of an environment; generating a set of features for the one or more images; combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; generating depth information of the environment based on the combined feature distance information; as well as The depth information of the environment is output.

18. The non-transitory computer readable medium of claim 17, wherein the one or more images are obtained from a monocular camera.

19. The non-transitory computer readable medium of claim 18, wherein the one or more distance maps are based on camera characteristics of the monocular camera.

20. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the at least one processor to generate the depth information of the environment using a machine learning model.

21. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the at least one processor to combine the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

22. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the at least one processor to select a distance map based on estimated heights of features in the set of features.

23. The non-transitory computer-readable medium of claim 17, wherein the instructions further cause the at least one processor to generate at least one of a segmented depth map of the environment, a segmented bird's-eye view (BEV) of the environment, or three-dimensional position information of one or more objects in the environment based on the depth information of the environment.

24. The non-transitory computer readable medium of claim 17, wherein the instructions further cause the at least one processor to: performing a transformation operation based on the depth information of the environment and the set of features to generate a set of transformed features; and The transformed set of features is output, wherein the depth information of the environment is implicitly output in the transformed set of features.

25. An apparatus for generating depth information, the apparatus comprising: means for obtaining one or more images of an environment; means for generating a set of features for the one or more images; means for combining the set of features with one or more distance maps to generate combined feature distance information, wherein the one or more distance maps indicate distances based on relative heights above ground level; A component for generating depth information of the environment based on the combined feature distance information; and Means for outputting said depth information of said environment.

26. The apparatus of claim 25, wherein the one or more images are obtained from a monocular camera.

27. The apparatus of claim 26, wherein the one or more range maps are based on camera characteristics of the monocular camera.

28. The apparatus of claim 26, further comprising means for generating the depth information of the environment using a machine learning model.

29. The apparatus of claim 26, further comprising means for combining the set of features with the one or more distance maps by concatenating or multiplying one or more values ​​of the one or more distance maps with the set of features.

30. The apparatus of claim 26, further comprising means for selecting a distance map based on estimated heights of features in the set of features.

Images