Method and system for a vehicle and a vehicle

Abstract

The present invention relates to methods and systems for vehicles and vehicles. Embodiments of a ground plane estimation (GPE) using an LSN are disclosed. In an embodiment, the method includes: obtaining a point cloud from a depth sensor of a vehicle operating in an environment; encoding the point cloud; estimating a ground plane in the environment using a deep learning network with the encoded point cloud as input; planning a path through the environment based on a drivable area of ​​the estimated ground plane; and operating the vehicle along the path. The deep learning network includes a two-dimensional (2D) convolutional backbone and a ground plane estimation (GPE) head for estimating the ground plane. In an embodiment, the point cloud is encoded using a point column.

Description

Technical Field

[0001] The following description generally relates to ground estimation for path planning of autonomous vehicles. Background Art

[0002] Light detection and ranging (LiDAR) sensors mounted on autonomous vehicles (AVs) output three-dimensional (3D) point clouds that provide a detailed understanding of the environment surrounding the AV. Ground plane estimation (GPE) is a measure of the height of the ground plane relative to the autonomous vehicle. Accurate GPE facilitates efficient fusion-based autonomous driving. However, existing perception pipelines typically use traditional machine learning techniques to classify LiDAR points as foreground or ground. Summary of the Invention

[0003] A technique for GPE using LiDAR semantic network (LSN) is provided.

[0004] In an embodiment, a method comprises: obtaining, using at least one processor, a point cloud from a depth sensor of a vehicle operating in an environment; encoding, using the at least one processor, the point cloud; estimating a ground plane in the environment using a deep learning network with the encoded point cloud as input; determining a drivable area in the environment using the estimated ground plane and a map of the environment; planning, using the at least one processor, a path through the environment based on the drivable area; and operating the vehicle along the path using control circuitry of the vehicle.

[0005] In an embodiment, the deep learning network includes a two-dimensional convolution backbone, i.e., a 2D convolution backbone, and a ground plane estimation head, i.e., a GPE head, for estimating the ground plane, and the method further includes: using the at least one processor to obtain a feature map according to multiple strides of the 2D convolution backbone; using the at least one processor to upsample each feature map to the original feature map size; and using the at least one processor and weighted summation to combine the upsampled feature maps to generate an output map for estimating the ground plane.

[0006] In an embodiment, encoding the point cloud further comprises: dividing the point cloud into 2D columns in an image embedding; for each column, selecting a plurality of sampling points; adding a plurality of enhancements to the selected sampling points; and processing each column using a point network to obtain point features of the column; mapping the point features into a point feature map based on corresponding column positions to create a 2D point feature map; processing the 2D point feature map using the 2D convolutional backbone to obtain a rich feature map of different amplitudes; and passing the rich feature map to the GPE head to estimate the ground plane.

[0007] In an embodiment, the image embedding is a bird's eye view image, or BEV image.

[0008] In an embodiment, the GPE head comprises one or more convolutional layers.

[0009] In an embodiment, a method includes: obtaining, using at least one processor, a point cloud captured by a depth sensor operating in an environment; encoding, using at least one processor, the point cloud; and estimating a ground plane in the environment using a deep learning network with the encoded point cloud as input; comparing, using the at least one processor, the estimated ground plane to a ground truth map to determine a ground plane estimation loss; and updating, using the at least one processor, parameters of the deep learning network based on the determined ground plane estimation loss.

[0010] In an embodiment, comparing the estimated ground plane to the ground truth map to determine a ground plane estimation loss further comprises minimizing a loss function of the ground plane estimate to the ground truth map.

[0011] In an embodiment, the loss function is a smooth-1 regression loss.

[0012] In an embodiment, the deep learning network includes a 2D convolutional backbone coupled to a ground plane estimation head (GPE) head for estimating the ground plane, and the method further includes: obtaining, using the at least one processor, feature maps according to multiple strides of the 2D convolutional backbone; upsampling, using the at least one processor, each feature map to an original feature map size; and combining, using the at least one processor and weighted summation, the upsampled feature maps to generate an output map for estimating the ground plane.

[0013] In an embodiment, encoding the point cloud further comprises: dividing the point cloud into 2D columns in an image embedding; for each column, selecting a plurality of sampling points; adding a plurality of enhancements to the selected sampling points; and processing each column using a point network to obtain point features of the column; mapping the point features into a point feature map based on corresponding column positions to create a 2D point feature map; processing the 2D point feature map using the 2D convolutional backbone to obtain a rich feature map of different amplitudes; and passing the rich feature map to the GPE head to estimate the ground plane.

[0014] In an embodiment, the GPE head comprises one or more convolutional layers.

[0015] In an embodiment, a vehicle comprises: a depth sensor configured to generate a point cloud of an environment; control circuitry; and at least one processor configured to: obtain the point cloud from the depth sensor; encode the point cloud; estimate a ground plane in the environment using a deep learning network with the encoded point cloud as input; determine a drivable area in the environment using the estimated ground plane and a map of the environment; plan a path through the environment based on the drivable area; and operate the vehicle along the path using the control circuitry.

[0016] In an embodiment, the deep learning network further comprises: a two-dimensional convolutional backbone, i.e., a 2D convolutional backbone, configured to generate feature maps according to a plurality of strides; and a ground plane estimation head, i.e., a GPE head, coupled to the 2D convolutional backbone, the GPE head being configured to: upsample each feature map to the size of the original feature map; and combine the upsampled feature maps by weighted summation to generate an output map for estimating the ground plane.

[0017] In an embodiment, encoding the point cloud also includes: dividing the point cloud into two-dimensional columns, i.e., 2D columns, in an image embedding; selecting multiple sampling points for each column; adding multiple enhancements to the selected sampling points; and processing each column using a point network to obtain point features of the column; mapping the point features into a point feature map based on corresponding column positions to create a 2D point feature map; processing the 2D point feature map using the 2D convolution backbone to obtain a rich feature map of different amplitudes; and passing the rich feature map to the GPE head to estimate the ground plane.

[0018] In an embodiment, the GPE head comprises one or more convolutional layers.

[0019] In an embodiment, a system includes: at least one processor; a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations including: obtaining a point cloud captured by a depth sensor operating in an environment; encoding the point cloud; estimating a ground plane in the environment using a deep learning network with the encoded point cloud as input; comparing the estimated ground plane to a ground truth map to determine a ground plane estimation loss; and updating parameters of the deep learning network based on the determined ground plane estimation loss.

[0020] In an embodiment, comparing the estimated ground plane to the ground truth map to determine a ground plane estimation loss further comprises minimizing a loss function of the ground plane estimate to the ground truth map.

[0021] In an embodiment, the loss function is a smooth-1 regression loss.

[0022] In an embodiment, the operations further include: obtaining feature maps according to a plurality of strides using the deep learning network; upsampling each feature map to an original feature map size using the deep learning network; and combining the upsampled feature maps using a weighted sum to generate an output map for estimating the ground plane.

[0023] In an embodiment, encoding the point cloud also includes: dividing the point cloud into two-dimensional columns, i.e., 2D columns, in image embedding; selecting multiple sampling points for each column; adding multiple enhancements to the selected sampling points; and using a point network to process each column to obtain point features of the column; mapping the point features into a point feature map based on the corresponding column positions to create a 2D point feature map; using the deep learning network to process the 2D point feature map to obtain a rich feature map of different amplitudes; and using the rich feature map to estimate the ground plane.

[0024] In an embodiment, the deep learning network comprises one or more convolutional layers.

[0025] One or more of the disclosed embodiments provide one or more of the following advantages.

[0026] These and other aspects, features, and implementations may be represented as methods, apparatus, systems, components, program products, means or steps, and other means for performing the functions.

[0027] These and other aspects, features and implementations will be apparent from the following description including the claims. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 An example of an autonomous vehicle (AV) having autonomous capabilities is shown in accordance with one or more embodiments.

[0029] Figure 2 An example "cloud" computing environment is illustrated in accordance with one or more embodiments.

[0030] Figure 3 A computer system according to one or more embodiments is illustrated.

[0031] Figure 4 An example architecture of an AV is shown in accordance with one or more embodiments.

[0032] Figure 5 is a block diagram of a system using GPE of LSN according to one or more embodiments.

[0033] Figure 6 is a block diagram of a GPE head coupled to a convolutional backbone in accordance with one or more embodiments.

[0034] Figure 7 is a flow chart of a route planning process based on GPE using LSN, according to one or more embodiments.

[0035] Figure 8 is a flow diagram of a process for training an LSN for a GPE according to one or more embodiments. DETAILED DESCRIPTION

[0036] In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be apparent that the present invention can be practiced without these specific details. In other examples, well-known configurations and devices are shown in block diagram form to avoid unnecessarily obscuring the present invention.

[0037] In the accompanying drawings, for ease of description, a specific arrangement or order of schematic elements (such as those representing devices, modules, instruction blocks, and data elements) is shown. However, it will be understood by those skilled in the art that the specific order or arrangement of schematic elements in the accompanying drawings is not intended to require a specific processing order or sequence, or separation of processing processes. In addition, the inclusion of schematic elements in the accompanying drawings is not intended to mean that such elements are required in all embodiments, nor is it intended to mean that the features represented by such elements cannot be included in some embodiments or cannot be combined with other elements in some embodiments.

[0038] In addition, in the accompanying drawings, connecting elements, such as solid or dotted lines or arrows, are used to illustrate the connection, relationship or association between two or more other schematic elements, and without any such connecting elements it is not intended to mean that there can be no connection, relationship or association. In other words, the connection, relationship or association between some elements are not shown in the accompanying drawings, so as not to obscure the present disclosure. In addition, for ease of illustration, a single connecting element is used to represent multiple connections, relationships or associations between elements. For example, if the communication of connecting element representation signal, data or instruction, it will be understood by those skilled in the art that this element representation affects one or more signal paths (for example, bus) that may be needed for communication.

[0039] Reference will now be made in detail to the embodiments, examples of which are illustrated in the accompanying drawings. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various embodiments described. However, it will be apparent to one of ordinary skill in the art that the various embodiments described may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

[0040] The several features described below can each be used independently of one another or in combination with any other features. However, any individual feature may not solve any of the problems discussed above, or may only solve one of the problems discussed above. Some of the problems discussed above may not be fully solved by any one of the features described herein. Although headings are provided, information related to a specific heading but not found in the section with that heading may also be found elsewhere in this specification. This article describes an embodiment according to the following summary:

[0041] 1. General Overview

[0042] 2. System Overview

[0043] 3. Autonomous Vehicle Architecture

[0044] 4. GPE using LSN

[0045] General Overview

[0046] A GPE technique using LSN is provided. In an embodiment, the LSN inputs a 3D LiDAR point cloud by first dividing the 3D point cloud into two-dimensional (2D) columns in an image embedding (e.g., a bird's-eye view (BEV) image). For each column, multiple point cloud samples are selected and processed by a neural network to obtain point features. The point features are used to generate a 2D point feature map based on their corresponding column positions. The 2D point feature map is passed to a 2D convolutional backbone (e.g., a convolutional neural network (CNN)) that generates rich feature maps of different amplitudes. These rich feature maps are input to a detection head and a GPE head coupled to the 2D convolutional backbone, and the detection head and the GPE head are configured to estimate objects and a ground plane, respectively. These estimates are compared with the ground truth map to calculate the GPE loss. The GPE loss is backpropagated through the LSN to update the parameters of the LSN (e.g., weights, biases).

[0047] In an embodiment, the GPE head is a fully convolutional neural network. Convolutional feature maps of different strides (e.g., stride 8, 16, and 32) are obtained based on the 2D convolution backbone and input into the GPE head. The feature map is upsampled by the deconvolution layer to obtain the original feature map size. The upsampled maps are combined by weighted summation to obtain the final output map of the estimated ground plane. To train the LSN, the GPE loss is minimized using a loss function (e.g., smooth-l1 regression loss) between the output map predicted by the network and the ground truth map. In an embodiment, if the annotated ground data is not available for the entire ground truth map, the four corners of the bottom surface of the annotated 3D bounding box in the output map of the detection head can be used as ground truth data to fill the ground truth map.

[0048] System Overview

[0049] Figure 1 An example of an autonomous vehicle 100 having autonomous capabilities is shown.

[0050] As used herein, the term "autonomous capability" refers to a function, feature, or facility that enables a vehicle to operate partially or fully without real-time human intervention, including but not limited to fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles.

[0051] As used herein, an autonomous vehicle (AV) is a vehicle with autonomous capabilities.

[0052] As used herein, "vehicle" includes any mode of transport for goods or people, such as a car, bus, train, airplane, drone, truck, boat, ship, submersible, or spacecraft. An unmanned car is an example of a vehicle.

[0053] As used herein, a "trajectory" refers to a path or route that an AV takes from a first spatiotemporal location to a second spatiotemporal location. In embodiments, the first spatiotemporal location is referred to as an initial location or starting location, and the second spatiotemporal location is referred to as a destination, final location, target, target position, or target location. In some examples, a trajectory traverses one or more map segments (e.g., sections of a road), and each segment is composed of one or more blocks (e.g., a lane or portion of an intersection). In embodiments, a spatiotemporal location corresponds to a real-world location. For example, a spatiotemporal location is a pickup or drop-off location to get people or cargo on or off the vehicle.

[0054] As used herein, "sensor(s)" includes one or more hardware components for detecting information related to the sensor's surroundings. Some hardware components may include sensing components (e.g., image sensors, biometric sensors), transmitting and / or receiving components (e.g., laser or radio frequency wave transmitters and receivers), electronic components (such as analog-to-digital converters), data storage devices (such as RAM and / or non-volatile memory), software or firmware components and data processing components (such as application specific integrated circuits), microprocessors, and / or microcontrollers.

[0055] As used herein, a “scene description” is a data structure (e.g., a list) or data stream that includes one or more classified or labeled objects detected by one or more sensors on an AV vehicle, or one or more classified or labeled objects provided by a source external to the AV.

[0056] As used herein, a "road" is a physical area that can be traversed by a vehicle and can correspond to a named thoroughfare (e.g., a city street, an interstate highway, etc.) or can correspond to an unnamed thoroughfare (e.g., a driveway within a house or office building, a section of a parking lot, a section of a vacant parking lot, a dirt road in a rural area, etc.). Because some vehicles (e.g., four-wheel drive pickup trucks, off-road vehicles (SUVs), etc.) can traverse a variety of physical areas that are not particularly suitable for vehicle travel, a "road" can be any physical area that is not formally defined as a thoroughfare by a municipality or other governmental or administrative agency.

[0057] As used herein, a "lane" is a portion of a road that can be traversed by vehicles and may correspond to most or all of the space between lane markings, or to only some of the space between lane markings (e.g., less than 50%). For example, a road with lane markings that are far apart may accommodate two or more vehicles between the markings, such that one vehicle can pass another vehicle without crossing the lane markings, and thus may be interpreted as having a lane that is narrower than the space between the lane markings, or as having two lanes between the lanes. Lanes may also be interpreted in the absence of lane markings. For example, lanes may be defined based on physical features of the environment, such as rocks and trees along a pathway in a rural area.

[0058] As used herein, a "machine learning model" includes any model trained on data to make predictions, including but not limited to: artificial neural networks, support vector machines, decision trees, regression analysis, Bayesian networks, and genetic algorithms.

[0059] As used herein, a "ground plane" is a set of points that are separate from other points in a 3D point cloud and represent pulsed light waves reflected from the ground.

[0060] As used herein, "estimated ground level" is a measure of the height of the ground relative to the vehicle.

[0061] As used herein, a "drivable area" is a portion of an estimated ground plane that is determined to be drivable by a vehicle using, for example, map data, road conditions, and traffic laws.

[0062] “One or more” includes a function performed by one element, a function performed by multiple elements, such as in a distributed manner, several functions performed by one element, several functions performed by several elements, or any combination of the foregoing.

[0063] It will also be understood that although in some cases, the terms "first," "second," etc. are used herein to describe various elements, these elements should not be limited by these terms. These terms are only used to distinguish one element from another. For example, a first contact may be referred to as a second contact, and similarly, a second contact may be referred to as a first contact, without departing from the scope of the various described embodiments. Both the first contact and the second contact are contacts, but they are not the same contact.

[0064] The terms used in the specification of the various embodiments described herein are for the purpose of describing specific embodiments only and are not intended to be limiting. As used in the specification of the various embodiments described and the appended claims, the singular forms "a", "an", and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise. It will also be understood that "and / or" as used herein refers to and includes any and all possible combinations of one or more related list items. It will also be understood that when the terms "comprises", "comprising", "having" and / or "having" are used in this specification, the presence of the stated features, integers, steps, operations, elements and / or components is specified, but the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof is not excluded.

[0065] As used herein, the term "if" is alternatively understood to mean "when" or "at the time" or "in response to being determined as" or "in response to being detected," depending on the context. Similarly, the phrases "if it has been determined" or "if [stated condition or event] has been detected" are alternatively understood to mean "upon the determination" or "in response to being determined as" or "upon detecting [stated condition or event]" or "in response to detecting [stated condition or event]," depending on the context.

[0066] As used herein, an AV system refers to an AV and the array of hardware, software, stored data, and real-time generated data that supports AV operation. In an embodiment, the AV system is incorporated into the AV. In an embodiment, the AV system is distributed across several locations. For example, some software for the AV system is in a system similar to the following about Figure 2 The described cloud computing environment 200 is implemented in a cloud computing environment.

[0067] In general, this document describes technologies applicable to any vehicle having one or more autonomous capabilities, including fully autonomous vehicles, highly autonomous vehicles, and conditionally autonomous vehicles, such as so-called Level 5, Level 4, and Level 3 vehicles, respectively (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for Road-Based Motor Vehicles, which is incorporated by reference in its entirety into this document for more detailed information on vehicle autonomy levels). The technologies described in this document are also applicable to partially autonomous vehicles and driver-assisted vehicles, such as so-called Level 2 and Level 1 vehicles (see SAE International Standard J3016: Classification and Definitions of Terms Relating to Automated Driving Systems for Road-Based Motor Vehicles). In embodiments, one or more Level 1, Level 2, Level 3, Level 4, and Level 5 vehicle systems may automatically perform certain vehicle operations (e.g., steering, braking, and using maps) under certain operating conditions based on processing of sensor inputs. The technologies described in this document can benefit vehicles of any level, ranging from fully autonomous vehicles to human-operated vehicles.

[0068] refer to Figure 1 , the AV system 120 causes the AV 100 to operate along a trajectory 198 through the environment 190 to a destination 199 (sometimes referred to as a final location) while avoiding objects (e.g., natural obstacles 191, vehicles 193, pedestrians 192, cyclists, and other obstacles) and obeying road rules (e.g., operating rules or driving preferences).

[0069] In an embodiment, the AV system 120 includes a device 101 equipped to receive and operate operating commands from a computer processor 146. In an embodiment, the computer processor 146 is connected to the computer 101 with reference to Figure 3 The processor 304 is similarly described. Examples of devices 101 include steering controls 102, brakes 103, gears, an accelerator pedal or other acceleration control mechanism, windshield wipers, side door locks, window controls, and turn indicators.

[0070] In an embodiment, the AV system 120 includes sensors 121 for measuring or inferring attributes of the state or condition of the AV 100, such as the AV's position, linear and angular velocity and acceleration, and heading (e.g., the direction of the front end of the AV 100). Examples of sensors 121 are GNSS, an inertial measurement unit (IMU) that measures both linear acceleration and angular rate of the vehicle, wheel rate sensors for measuring or estimating wheel slip, wheel brake pressure or brake torque sensors, engine torque or wheel torque sensors, and steering angle and angular rate sensors.

[0071] In an embodiment, the sensors 121 also include sensors for sensing or measuring properties of the AV's environment, such as a monocular or stereo camera 122 in the visible, infrared, or thermal (or both) spectrum, a LiDAR 123, a RADAR, an ultrasonic sensor, a time-of-flight (TOF) depth sensor, a velocity sensor, a temperature sensor, a humidity sensor, and a precipitation sensor.

[0072] In an embodiment, the AV system 120 includes a data storage unit 142 and a memory 144 for storing machine instructions associated with the computer processor 146 or data collected by the sensor 121. In an embodiment, the data storage unit 142 is associated with the following Figure 3 ROM 308 or storage device 310 described above. In an embodiment, memory 144 is similar to main memory 306 described below. In an embodiment, data storage unit 142 and memory 144 store historical, real-time, and / or predictive information about environment 190. In an embodiment, the stored information includes maps, driving performance, traffic congestion updates, or weather conditions. In an embodiment, data related to environment 190 is transmitted from remote database 134 to AV 100 via a communication channel.

[0073] In an embodiment, the AV system 120 includes communication devices 140 for transmitting measured or inferred attributes of the state and condition of other vehicles (such as position, linear and angular velocity, linear and angular acceleration, and linear and angular heading) to the AV 100. These devices include vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communication devices, as well as devices for wireless communication via point-to-point or ad hoc networks, or both. In an embodiment, the communication devices 140 communicate across the electromagnetic spectrum (including radio and optical communications) or other media (e.g., air and acoustic media). The combination of vehicle-to-vehicle (V2V) and vehicle-to-infrastructure (V2I) communications (and in some embodiments, one or more other types of communications) is sometimes referred to as vehicle-to-everything (V2X) communication. V2X communications typically conform to one or more communication standards for communication with and between autonomous vehicles.

[0074] In an embodiment, the communication device 140 includes a communication interface. For example, a wired, wireless, WiMAX, Wi-Fi, Bluetooth, satellite, cellular, optical, near-field, infrared, or radio interface. The communication interface transmits data from the remote database 134 to the AV system 120. In an embodiment, the remote database 134 is embedded in a computer such as Figure 2The communication interface 140 transmits data collected from the sensors 121 or other data related to the operation of the AV 100 to the remote database 134. In an embodiment, the communication interface 140 transmits information related to teleoperation to the AV 100. In some embodiments, the AV 100 communicates with other remote (e.g., "cloud") servers 136.

[0075] In an embodiment, the remote database 134 also stores and transmits digital data (e.g., data such as roads and street locations). This data is stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0076] In an embodiment, the remote database 134 stores and transmits historical information regarding driving attributes (e.g., speed and acceleration profiles) of vehicles that have previously traveled along the trajectory 198 at similar times of day. In one implementation, such data may be stored in the memory 144 on the AV 100 or transmitted from the remote database 134 to the AV 100 via a communication channel.

[0077] The computing device 146 located on the AV 100 algorithmically generates control actions based on both real-time sensor data and a priori information, allowing the AV system 120 to perform its autonomous driving capabilities.

[0078] In an embodiment, the AV system 120 includes a computer peripheral device 132 coupled to a computing device 146 for providing information and alerts to a user of the AV 100 (e.g., an occupant or a remote user) and receiving input from the user. In an embodiment, the peripheral device 132 is similar to the one described below with reference to Figure 3 The display 312, input device 314 and cursor control 316 are discussed. The coupling may be wireless or wired. Any two or more of the interface devices may be integrated into a single device.

[0079] Sample cloud computing environment

[0080] Figure 2 Illustrate an example "cloud" computing environment. Cloud computing is a service delivery model for enabling convenient, on-demand access to a shared pool of configurable computing resources (e.g., networks, network bandwidth, servers, processing, memory, storage, applications, virtual machines, and services) over a network. In a typical cloud computing system, one or more large cloud data centers house the machines used to deliver the services provided by the cloud. Now refer to Figure 2, cloud computing environment 200 includes cloud data centers 204a, 204b, and 204c interconnected by a cloud 202. Data centers 204a, 204b, and 204c provide cloud computing services to computer systems 206a, 206b, 206c, 206d, 206e, and 206f connected to cloud 202.

[0081] The cloud computing environment 200 includes one or more cloud data centers. Generally speaking, a cloud data center (e.g. Figure 2 The cloud data center 204a shown in FIG refers to a cloud (eg Figure 2 The physical arrangement of servers in a cloud 202 (or a specific portion of a cloud) as shown in FIG. For example, servers are physically arranged into rooms, groups, rows, and racks in a cloud data center. A cloud data center has one or more zones, which include one or more server rooms. Each room has one or more rows of servers, and each row includes one or more racks. Each rack includes one or more individual server nodes. In some implementations, servers in zones, rooms, racks, and / or rows are arranged into groups based on the physical infrastructure requirements of the data center facility, including power, energy, heat, heat sources, and / or other requirements. In an embodiment, the server nodes are similar to Figure 3 The data center 204a has many computing systems distributed across multiple racks.

[0082] Cloud 202 includes cloud data centers 204a, 204b, and 204c, as well as networks and network resources (e.g., network devices, nodes, routers, switches, and network cables) used to connect cloud data centers 204a, 204b, and 204c and facilitate access to cloud computing services by computing systems 206a-f. In embodiments, the network represents any combination of one or more local networks, wide area networks, or internetworks coupled via wired or wireless links deployed using terrestrial or satellite connections. Data exchanged over the network is transmitted using a variety of network layer protocols, such as Internet Protocol (IP), Multiprotocol Label Switching (MPLS), Asynchronous Transfer Mode (ATM), Frame Relay, and the like. Furthermore, in embodiments where the network represents a combination of multiple subnetworks, a different network layer protocol is used on each underlying subnetwork. In some embodiments, the network represents one or more interconnected internetworks, such as the public Internet.

[0083] Computing systems 206a-f, or cloud computing service consumers, are connected to the cloud 202 via network links and network adapters. In embodiments, computing systems 206a-f are implemented as various computing devices, such as servers, desktops, laptops, tablets, smartphones, Internet of Things (IoT) devices, autonomous vehicles (including cars, drones, shuttles, trains, buses, etc.), and consumer electronics. In embodiments, computing systems 206a-f are implemented in other systems or as part of other systems.

[0084] Example Computer System

[0085] Figure 3 An example computer system 300 is shown. In an implementation, the computer system 300 is a dedicated computing device. The dedicated computing device may be hardwired to perform these techniques, or may include digital electronic devices such as one or more application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs) that are permanently programmed to perform the above-mentioned techniques, or may include one or more general-purpose hardware processors that are programmed to perform these techniques according to program instructions in firmware, memory, other storage, or a combination thereof. Such a dedicated computing device may also combine customized hardwired logic, ASICs, or FPGAs with customized programming to accomplish these techniques. In various embodiments, the dedicated computing device is a desktop computer system, a portable computer system, a handheld device, a network device, or any other device that includes hardwired and / or program logic to implement these techniques.

[0086] In an embodiment, computer system 300 includes a bus 302 or other communication mechanism for communicating information, and a hardware processor 304 coupled to bus 302 for processing information. Hardware processor 304 is, for example, a general-purpose microprocessor. Computer system 300 also includes a main memory 306, such as a random access memory (RAM) or other dynamic storage device, coupled to bus 302 to store information and instructions for execution by processor 304. In one implementation, main memory 306 is used to store temporary variables or other intermediate information during the execution of instructions to be executed by processor 304. When these instructions are stored in a non-transitory storage medium accessible to processor 304, computer system 300 becomes a special-purpose machine customized to perform the operations specified in the instructions.

[0087] In embodiments, computer system 300 also includes a read-only memory (ROM) 308 or other static storage device coupled to bus 302 for storing static information and instructions for processor 304. A storage device 310, such as a magnetic disk, optical disk, solid-state drive, or three-dimensional cross-point memory, is provided and coupled to bus 302 to store information and instructions.

[0088] In an embodiment, the computer system 300 is coupled via a bus 302 to a display 312, such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, a light emitting diode (LED) display, or an organic light emitting diode (OLED) display for displaying information to a computer user. An input device 314, including alphanumeric and other keys, is coupled to the bus 302 for communicating information and command selections to the processor 304. Another type of user input device is a cursor controller 316, such as a mouse, a trackball, a touch-sensitive display, or cursor direction keys, for communicating direction information and command selections to the processor 304 and for controlling movement of a cursor on the display 312. Such input devices typically have two degrees of freedom along two axes, a first axis (e.g., an x-axis) and a second axis (e.g., a y-axis), which allow the device to specify a position on a plane.

[0089] According to one embodiment, the techniques herein are performed by computer system 300 in response to processor 304 executing one or more sequences of one or more instructions contained in main memory 306. These instructions are read into main memory 306 from another storage medium, such as storage device 310. Execution of the sequences of instructions contained in main memory 306 causes processor 304 to perform the process steps described herein. In alternative embodiments, hard-wired circuitry is used in place of or in combination with software instructions.

[0090] As used herein, the term "storage medium" refers to any non-transitory medium that stores data and / or instructions that cause a machine to operate in a particular manner. Such storage media include non-volatile media and / or volatile media. Non-volatile media include, for example, optical disks, magnetic disks, solid-state drives, or three-dimensional cross-point memory, such as storage device 310. Volatile media include dynamic memory, such as main memory 306. Common forms of storage media include, for example, floppy disks, disks, hard disks, solid-state drives, magnetic tape or any other magnetic data storage medium, CD-ROMs, any other optical data storage medium, any physical medium with a hole pattern, RAM, PROM and EPROM, FLASH-EPROM, NV-RAM, or any other memory chip or storage cartridge.

[0091] Storage media are distinct from transmission media, but can be used in conjunction with them. Transmission media participate in the transmission of information between storage media. Examples of transmission media include coaxial cables, copper wire, and optical fiber, including the wires that comprise bus 302. Transmission media can also take the form of acoustic or optical waves, such as those generated during radio wave and infrared data communications.

[0092] In one embodiment, various forms of media are involved in carrying one or more sequences of one or more instructions to processor 304 for execution. For example, the instructions may initially be executed on a disk or solid-state drive of a remote computer. The remote computer loads the instructions into its dynamic memory and sends the instructions over a telephone line using a modem. The modem local to computer system 300 receives the data on the telephone line and uses an infrared transmitter to convert the data to an infrared signal. An infrared detector receives the data carried in the infrared signal, and appropriate circuitry places the data on bus 302. Bus 302 carries the data to main memory 306, from which processor 304 retrieves and executes the instructions. The instructions received by main memory 306 may optionally be stored on storage device 310 before or after execution by processor 304.

[0093] Computer system 300 also includes a communication interface 318 coupled to bus 302. Communication interface 318 provides a two-way data communication coupled to a network link 320 that is connected to a local network 322. For example, communication interface 318 is an integrated services digital network (ISDN) card, a cable modem, a satellite modem, or a modem for providing a data communication connection with a corresponding type of telephone line. As another example, communication interface 318 is a local area network (LAN) card for providing a data communication connection with a compatible LAN. In some implementations, a wireless link is also implemented. In any such implementation, communication interface 318 sends and receives electrical, electromagnetic, or optical signals that carry digital data streams representing various types of information.

[0094] Network link 320 typically provides data communication to other data devices through one or more networks. For example, network link 320 provides a connection to a host computer 324 or to a cloud data center or facility operated by an Internet Service Provider (ISP) 326 through a local network 322. ISP 326, in turn, provides data communication services through the worldwide packet data communication network now commonly referred to as the "Internet" 328. Both local network 322 and Internet 328 use electrical, electromagnetic, or optical signals that carry digital data streams. The signals through the various networks and the signals on network link 320 and through communication interface 318 are example forms of transmission media, where these signals carry digital data to and from computer system 300. In an embodiment, network 320 includes cloud 202 or a portion of cloud 202 as described above.

[0095] Computer system 300 sends messages and receives data, including program code, through network(s), network link 320, and communication interface 318. In an embodiment, computer system 300 receives code for processing. The received code is executed by processor 304 upon receipt and / or stored in storage device 310 or other non-volatile storage for later execution.

[0096] Example Autonomous Vehicle Architecture

[0097] Figure 4 Shown for autonomous vehicles (e.g., Figure 1 100). The architecture 400 includes a sensing module 402 (sometimes referred to as sensing circuitry), a planning module 404 (sometimes referred to as planning circuitry), a control module 406 (sometimes referred to as control circuitry), a positioning module 408 (sometimes referred to as positioning circuitry), and a database module 410 (sometimes referred to as database circuitry). Each module plays a role in the operation of the AV 100. Collectively, the modules 402, 404, 406, 408, and 410 may be Figure 1 4 and 5. In some embodiments, any of modules 402, 404, 406, 408, and 410 is a combination of computer software (e.g., executable code stored on a computer-readable medium) and computer hardware (e.g., one or more microprocessors, microcontrollers, application-specific integrated circuits [ASICs], hardware memory devices, other types of integrated circuits, other types of computer hardware, or a combination of any or all of these).

[0098] In use, the planning module 404 receives data representing a destination 412 and determines data representing a trajectory 414 (sometimes referred to as a route) that the AV 100 may travel in order to reach (e.g., arrive at) the destination 412. In order for the planning module 404 to determine the data representing the trajectory 414, the planning module 404 receives data from the perception module 402, the positioning module 408, and the database module 410.

[0099] The perception module 402 uses, for example, Figure 1 One or more sensors 121 are shown to identify nearby physical objects, classify the objects (e.g., into types such as pedestrians, bicycles, cars, traffic signs, etc.), and provide a scene description including the classified objects 416 to the planning module 404.

[0100] The planning module 404 also receives data representing the AV's position 418 from the positioning module 408. The positioning module 408 determines the AV's position by using data from the sensor 121 and data from the database module 410 (e.g., geographic data) to calculate the position. For example, the positioning module 408 uses data from a GNSS (Global Navigation Satellite System) sensor and geographic data to calculate the longitude and latitude of the AV. In an embodiment, the data used by the positioning module 408 includes a high-precision map with lane geometry, a map describing the road network connectivity attributes, a map describing the physical attributes of the lanes (such as traffic speed, traffic volume, the number of vehicle and bicycle lanes, lane width, lane traffic direction, or lane marking type and location, or a combination thereof), and a map describing the spatial location of road features (such as intersections, traffic signs, or various types of other driving signals).

[0101] The control module 406 receives data representing the trajectory 414 and data representing the AV's position 418 and operates the AV's control functions 420 a - 420 c (e.g., steering, throttle, brakes, ignition) in a manner that will cause the AV 100 to travel the trajectory 414 to reach the destination 412. For example, if the trajectory 414 includes a left turn, the control module 406 will operate the control functions 420 a - 420 c in such a manner that the steering angle of the steering function will cause the AV 100 to turn left, and the throttle and brakes will cause the AV 100 to pause and wait for a passing pedestrian or vehicle before executing the turn.

[0102] Example GPE using LSN

[0103] Figure 5 5 is a block diagram of a system 500 for GPE using LSN according to one or more embodiments. The system 500 includes an encoder 502, a 2D convolutional backbone 503, a detection head 504, and a GPE head 505. A point cloud 501 (e.g., output by a LiDAR) is input to the encoder 502. In an embodiment, the point cloud is converted into a stacked column tensor 507 and a column index tensor. Sample points 508 from each column are represented by distance and offset measurements x. c 、y c 、z c 、x p and y pThe GPE is augmented / decorated 509, where the subscript c represents the distance of the sample point to the arithmetic mean of all points in the column, and the subscript p represents the offset of the sample point relative to the x, y center of the column. The augmented / decorated points are input to the point network 510, which uses the augmented points to learn features that can be used to generate a sparse two-dimensional (2D) pseudo image (point feature map) 511. The point feature map 511 is input to the 2D convolution backbone 503. The features output by the 2D convolution backbone 503 are used by the detection head 504 (e.g., a single shot detector (SSD)) to detect and retract a three-dimensional (3D) bounding box, which is projected into the output image 506. The camera view 513 is a visualization of the projected GPE. More details about encoder 502 can be found in “PointPillars: Fast Encoders for Object Detection from Point Clouds” (Lang Alex H. et al., arXiv:1812.05784[cs.LG], 2019), retrieved from https: / / arxiv.org / abs / 1812.05784.

[0104] In an embodiment, the 2D convolution backbone 503 provides a point feature map 511 to the GPE head 505, which outputs an estimated ground plane 512, as shown in FIG. Figure 6 More fully described.

[0105] Figure 6 6 is a block diagram of a GPE head 505 coupled to a 2D convolution backbone 503 according to one or more embodiments. In an embodiment, the GPE head 505 is a CNN. Convolutional feature maps of different strides (e.g., strides 8, 16, and 32) are obtained from the 2D convolution backbone 503 and input into the GPE head 505. The feature map is upsampled by an upsampler 602 through a deconvolution layer to obtain an upsampled feature map with the original size of the point feature map 511. The upsampled maps 603 are combined by a weighted sum 604 to obtain a final output map 605 as the GPE. During AV operation, the output map 605 is used by the AV's perception pipeline or route planner to determine a drivable area based on a ground plane estimate and other data (e.g., a map), and to calculate the AV's path through the drivable area.

[0106] To train the LSN 500, a loss function (e.g., smooth-l1 regression loss) is used between the output image 605 (i.e., GPE) and the ground truth map to minimize the GPE loss 606. The GPE loss 606 is then back-propagated to update the parameters (e.g., weights, biases) of the LSN 500. In an embodiment, if annotated ground truth data is not available for the entire ground truth map, the four corners of the bottom surface of the annotated 3D bounding box in the output image 506 are used as ground truth data to fill in the missing annotated ground truth data in the ground truth map.

[0107] Example Processing

[0108] Figure 7 is a flow chart of a route planning process based on GPE using LSN according to one or more embodiments. Process 700 can be performed, for example, by referring to Figure 3 The computer system 300 described above is implemented.

[0109] Process 700 includes the following steps: obtaining a point cloud from a depth sensor of a vehicle operating in an environment (701); encoding the point cloud (702); using a deep learning network to estimate a ground plane in the environment with the encoded point cloud as input (703); using the estimated ground plane and a map of the environment to determine a drivable area in the environment (704); planning a path through the environment based on the drivable area (705); and operating the vehicle along the path using control circuitry of the vehicle (706).

[0110] In the examples, as referenced Figure 5 As described, the point cloud is a LiDAR point cloud and the encoding uses a point column. In an embodiment, the deep learning network includes a 2D convolutional backbone (e.g., VGG, residual neural network) coupled to a detection head for estimating object detection and a GPE head for GPE.

[0111] Figure 8 is a flow chart of a process for training an LSN for a GPE according to one or more embodiments. Process 800 may be performed, for example, by referring to Figure 3 The computer system 300 described above is implemented.

[0112] Process 800 includes the following steps: obtaining point cloud data captured by a depth sensor operating in an environment (801); encoding the point cloud data (802); using a deep learning network to estimate a ground plane in the environment using the encoded point cloud data as input (803); comparing the estimated ground plane to a ground truth map to determine a GPE loss (804); and updating parameters (e.g., weights, biases) of the deep learning network based on the GPE loss (805).

[0113] For example, to train a deep learning network, a loss function (e.g., smooth-l1 regression loss) between the GPE and the ground truth map is used to minimize the GPE loss. The GPE loss 606 is then back-propagated to update the parameters (e.g., weights, biases) of the LSN 500.

[0114] In an embodiment, if annotated ground data is not available for the entire ground truth map, the four corners of the bottom surface of the annotated 3D bounding box are used as ground truth data to fill in the missing annotated ground data in the ground truth map.

[0115] In the previous description, embodiments of the present invention have been described with reference to many specific details, which may vary from implementation to implementation. Therefore, the description and drawings should be regarded as illustrative, not restrictive. The only and exclusive indication of the scope of the invention, and what the applicants expect to be the scope of the invention, is the literal and equivalent scope of the claims issued from this application in the specific form in which the claims are issued, including any subsequent amendments. Any definition of terms explicitly set forth herein for being included in such claims should be based on the meaning of such terms as used in the claims. In addition, when the term "also includes" is used in the previous description or the appended claims, the phrase may be followed by additional steps or entities, or sub-steps / sub-entities of the steps or entities previously described.

Claims

1. A method for a vehicle, comprising: obtaining, using at least one processor, a point cloud from a depth sensor of a vehicle operating in an environment; Encoding the point cloud using a two-dimensional convolutional backbone network (2D convolutional backbone network) executed by the at least one processor, the encoding comprising: Dividing the point cloud into two-dimensional columns in the image embedding, i.e., 2D columns; Processing the plurality of selected sampling points for each column using a point network to obtain point features of the corresponding column; and Mapping the point features of the plurality of pillars into a point feature map based on corresponding pillar positions to create a 2D point feature map; Using the 2D point feature map as input, obtaining a convolutional feature map according to multiple strides of the 2D convolutional backbone network; providing the convolutional feature map to a ground plane estimation head (GPE head) executed by the at least one processor; Using the convolutional feature map and using the ground plane estimation head (GPE head) to estimate a ground plane in the environment, wherein the estimating includes: Upsampling one or more convolutional feature maps to the original feature map size of the 2D point feature map to obtain an upsampled feature map; and combining the upsampled feature maps using a weighted sum to generate an output map for estimating the ground plane; and A drivable area in the vehicle's environment is determined using the estimated ground plane and a map of the environment.

2. The method according to claim 1, wherein The method further comprises: planning a path through the environment based on the drivable area using the at least one processor; and The vehicle is operated along the path using control circuitry of the vehicle.

3. The method according to claim 1, wherein Using a point network to process a plurality of selected sampling points for each column to obtain point features of the corresponding column includes: Select multiple sampling points; adding one or more enhancements to the selected sampling points; and The enhanced sampling points are processed using the point network to obtain point features of the corresponding columns.

4. The method according to claim 1, wherein The image embedded is a bird's-eye view image, namely a BEV image.

5. The method according to claim 1, wherein The GPE head includes one or more convolutional layers.

6. A method for processing a deep learning network, comprising: obtaining, using at least one processor, a point cloud captured by a depth sensor operating in an environment; Encoding the point cloud using a two-dimensional convolutional backbone network (2D convolutional backbone network) of the deep learning network executed by at least one processor, the encoding comprising: Dividing the point cloud into two-dimensional columns in the image embedding, i.e., 2D columns; Processing the plurality of selected sampling points for each column using a point network to obtain point features of the corresponding column; and Mapping the point features of the plurality of pillars into a point feature map based on corresponding pillar positions to create a 2D point feature map; Using the 2D point feature map as input, obtaining a convolutional feature map according to multiple strides of the 2D convolutional backbone network; providing the convolutional feature map to a ground plane estimation head (GPE head) of the deep learning network executed by the at least one processor; estimating a ground plane in the environment using the ground plane estimation head (GPE head), wherein the estimating comprises: Upsampling one or more convolutional feature maps to the original feature map size of the 2D point feature map to obtain an upsampled feature map; and combining the upsampled feature maps using a weighted sum to generate an output map for estimating the ground plane; comparing, using the at least one processor, the estimated ground plane to a ground truth map to determine a ground plane estimation loss; and Parameters of the deep learning network are updated based on the determined ground plane estimation loss using the at least one processor.

7. The method according to claim 6, wherein: Comparing the estimated ground plane to the ground truth map to determine the ground plane estimation loss further includes minimizing a loss function of the ground plane estimate to the ground truth map.

8. The method according to claim 7, wherein: The loss function is a smooth-Il regression loss.

9. The method according to claim 6, wherein: Using a point network to process a plurality of selected sampling points for each column to obtain point features of the corresponding column includes: Select multiple sampling points; adding one or more enhancements to the selected sampling points; and The enhanced sampling points are processed using the point network to obtain point features of the corresponding columns.

10. The method according to claim 6, wherein: The GPE head includes one or more convolutional layers.

11. A vehicle comprising: a depth sensor configured to generate a point cloud of the environment; control circuit; as well as At least one processor configured to: obtaining the point cloud from the depth sensor; Encoding the point cloud using a two-dimensional convolutional backbone network (2D convolutional backbone network) executed by the at least one processor, the encoding comprising: Dividing the point cloud into two-dimensional columns in the image embedding, i.e., 2D columns; Processing the plurality of selected sampling points for each column using a point network to obtain point features of the corresponding column; and Mapping the point features of the plurality of pillars into a point feature map based on corresponding pillar positions to create a 2D point feature map; Using the 2D point feature map as input, obtaining a convolutional feature map according to multiple strides of the 2D convolutional backbone network; providing the convolutional feature map to a ground plane estimation head (GPE head) executed by the at least one processor; Using the convolutional feature map and using the ground plane estimation head (GPE head) to estimate a ground plane in the environment, wherein the estimating includes: Upsampling one or more convolutional feature maps to the original feature map size of the 2D point feature map to obtain an upsampled feature map; and combining the upsampled feature maps using a weighted sum to generate an output map for estimating the ground plane; and A drivable area in the vehicle's environment is determined using the estimated ground plane and a map of the environment.

12. The vehicle according to claim 11, wherein: The at least one processor is further configured to: planning a path through the environment based on the drivable area; and The vehicle is operated along the path using the control circuit.

13. The vehicle according to claim 11, wherein: Using a point network to process a plurality of selected sampling points for each column to obtain point features of the corresponding column includes: Select multiple sampling points; adding one or more enhancements to the selected sampling points; and The enhanced sampling points are processed using the point network to obtain point features of the corresponding columns.

14. The vehicle according to claim 11, wherein: The GPE head includes one or more convolutional layers.

15. A system for processing a deep learning network, comprising: at least one processor; a memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations comprising: Obtaining a point cloud captured by a depth sensor operating in the environment; Encoding the point cloud using a two-dimensional convolutional backbone network (2D convolutional backbone network) of the deep learning network executed by the at least one processor, the encoding comprising: Dividing the point cloud into two-dimensional columns in the image embedding, i.e., 2D columns; Processing the plurality of selected sampling points for each column using a point network to obtain point features of the corresponding column; and Mapping the point features of the plurality of pillars into a point feature map based on corresponding pillar positions to create a 2D point feature map; Using the 2D point feature map as input, obtaining a convolutional feature map according to multiple strides of the 2D convolutional backbone network; providing the convolutional feature map to a ground plane estimation head (GPE head) of the deep learning network executed by the at least one processor; estimating a ground plane in the environment using the ground plane estimation head (GPE head), wherein the estimating comprises: Upsampling one or more convolutional feature maps to the original feature map size of the 2D point feature map to obtain an upsampled feature map; and combining the upsampled feature maps using a weighted sum to generate an output map for estimating the ground plane; comparing the estimated ground plane to a ground truth map to determine a ground plane estimation loss; and Parameters of the deep learning network are updated based on the determined ground plane estimation loss.

16. The system according to claim 15, wherein: Comparing the estimated ground plane to the ground truth map to determine the ground plane estimation loss further includes minimizing a loss function of the ground plane estimate to the ground truth map.

17. The system according to claim 16, wherein: The loss function is a smooth-Il regression loss.

18. The system according to claim 15, wherein: Using a point network to process a plurality of selected sampling points for each column to obtain point features of the corresponding column includes: Select multiple sampling points; adding one or more enhancements to the selected sampling points; and The enhanced sampling points are processed using the point network to obtain point features of the corresponding columns.

19. The system of claim 15, wherein: The deep learning network includes one or more convolutional layers.

20. A method for a vehicle, comprising: obtaining, using at least one processor, a point cloud from a depth sensor of a vehicle operating in an environment; Encoding the point cloud using the at least one processor, the encoding comprising: Divide the point cloud into a plurality of subsets in the image embedding, wherein the subsets are two-dimensional columns, i.e., 2D columns, For each subset, selecting a plurality of sampling points, adding one or more enhancements to the selected sampling points, and processing the enhanced sampling points using a point network to obtain point features of the corresponding column, and Mapping the point features of the plurality of columns into a feature map based on corresponding column positions to create a 2D point feature map; estimating a ground plane in the environment using a deep learning network, wherein the estimation is based on using the encoded point cloud including the 2D point feature map as input to the deep learning network, the deep learning network including a two-dimensional convolutional backbone network (i.e., a 2D convolutional backbone network) as a first part and a ground plane estimation head network (i.e., a GPE head network) for estimating the ground plane as a second part, the estimation comprising: processing the 2D point feature maps using the first portion of the deep learning network to obtain enriched feature maps of different amplitudes, Obtaining an upsampled feature map by upsampling one or more enriched feature maps to an original feature map size of the 2D point feature map using the second portion of the deep learning network; and estimating the ground plane using a second portion of the deep learning network by weighting and combining the upsampled feature maps to generate an output map estimating the ground plane, determining a drivable area in the environment using the estimated ground plane and a map of the environment; planning a path through the environment based on the drivable area using the at least one processor; and The vehicle is operated along the path using control circuitry of the vehicle.

21. The method according to claim 20, wherein Estimating a ground plane in the environment using the deep learning network includes: Obtaining the enriched feature map using the 2D point feature map according to multiple strides of the 2D convolutional backbone network; The enriched feature map is passed to the GPE head network to generate the output map for estimating the ground plane.

22. A computer program product comprising instructions, characterized in that When the instruction is executed by a processor, the steps of the method described in any one of claims 1-10 and claims 20-21 are implemented.

Images