Boundary estimation device and method

Abstract

Certain aspects are directed to an apparatus for lane estimation that generally includes at least one memory and at least one processor coupled to the at least one memory that is configured to receive a first input associated with a three-dimensional (3D) space, extract from the first input a first set of points associated with a ground plane of the 3D space, map each of the first set of points to a region of a plurality of regions of a two-dimensional (2D) frame, determine one or more attributes associated with each region of the plurality of regions based on one or more of the first set of points mapped to the region, and identify one or more road lanes based on the one or more attributes.

Description

[Technical field]

[0001]

[0001] The present disclosure generally relates to performing boundary estimation. Some aspects of the present disclosure include systems and techniques for sensor-based (e.g., light detection and ranging (LiDAR)-based) three-dimensional (3D) boundary estimation (e.g., lane boundary estimation or boundaries of other objects). [Background technology]

[0002]

[0002] Object detection techniques can be used to detect one or more objects in a scene. For example, object detection can be used to detect or identify one or more positions and / or locations of markings in an environment. Examples of areas in which a device may determine the positions and / or locations of markings include autonomous driving by an autonomous driving system (e.g., of an autonomous vehicle), autonomous navigation by a robotic system (e.g., an automatic vacuum cleaner, an automatic surgical device, etc.), among others. For example, a three-dimensional (3D) environment may include markings to facilitate navigation within the environment, such as road lanes. It may be important for an autonomous device to detect such markings and accurately navigate the space relative to such markings.

[0003]

[0003] Using an autonomous driving system as an example, an important feature for autonomous driving is the ability of an autonomous vehicle to detect lanes on a road and accurately determine the extent of the drivable space on the road. For example, some lanes on a road may be faded or partially obscured, making such lanes difficult to identify. Lane detection techniques can be used to detect or identify lanes on a road to facilitate various autonomous tasks. In some fields, the location of lanes in an environment can be determined so that an autonomous device can accurately navigate the environment (e.g., to determine accurate motion and trajectory planning). There is a need for systems and techniques to reliably track markings (e.g., road lanes) to facilitate navigation by an autonomous device. Summary of the Invention

[0004]

[0004] Certain aspects of the present application are directed to systems and techniques for boundary estimation. For example, certain aspects are directed to lane boundary estimation. The apparatus generally includes at least one memory and at least one processor coupled to the at least one memory, the at least one processor configured to receive a first input associated with a three-dimensional (3D) space, extract from the first input a first set of points associated with a ground plane of the 3D space, map each of the first set of points to a region of a plurality of regions of a two-dimensional (2D) frame, determine one or more attributes associated with each region of the plurality of regions based on one or more of the first set of points mapped to the region, and identify one or more road lanes based on the one or more attributes.

[0005]

[0005] Certain aspects are directed to a method for lane estimation. The method generally includes receiving a first input associated with a three-dimensional (3D) space, extracting from the first input a first set of points associated with a ground plane of the 3D space, mapping each of the first set of points to a region of a plurality of regions of a two-dimensional (2D) frame, determining one or more attributes associated with each region of the plurality of regions based on one or more of the first set of points mapped to the region, and identifying one or more road lanes based on the one or more attributes.

[0006]

[0006] Certain aspects are directed to an apparatus for lane estimation that generally includes means for receiving a first input associated with a three-dimensional (3D) space, means for extracting from the first input a first set of points associated with a ground plane of the 3D space, means for mapping each of the first set of points to a region of a plurality of regions of a two-dimensional (2D) frame, means for determining one or more attributes associated with each region of the plurality of regions based on one or more of the first set of points mapped to the region, and means for identifying one or more road lanes based on the one or more attributes.

[0007]

[0007] Certain aspects are directed to a non-transitory computer-readable medium having stored thereon instructions that, when executed by a processor, cause the processor to receive a first input associated with three-dimensional (3D) space, extract from the first input a first set of points associated with a ground plane of the 3D space, map each of the first set of points to one of a plurality of regions of a two-dimensional (2D) frame, determine one or more attributes associated with each of the plurality of regions based on one or more of the first set of points mapped to the region, and identify one or more road lanes based on the one or more attributes.

[0008] In some aspects, the computing device and / or the apparatus is, is a part of, and / or includes a vehicle or a computing device or component of a vehicle (e.g., an autonomous vehicle), a robotic device or system, a computing device or component of a robotic device or system, a personal computer, a laptop computer, a server computer, a camera, or other device. In some aspects, the computing device, the apparatus, and / or the vehicle includes a camera or cameras for capturing one or more images, or any other sensor for capturing environmental data. In some aspects, the computing device, the apparatus, and / or the vehicle further includes a display for displaying one or more images, notifications, and / or other displayable data. In some aspects, the computing device, the apparatus, and / or the vehicle described above may include one or more sensors (e.g., one or more inertial measurement units (IMUs), such as one or more gyrometers, one or more accelerometers, any combination thereof, and / or other sensors).

[0009]

[0009] This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used independently to determine the scope of the claimed subject matter, which subject matter should be understood by reference to appropriate portions of the entire specification of this patent, any or all of the drawings, and the claims.

[0010]

[0010] The foregoing summary, together with other features and embodiments, will become more apparent from a review of the following specification, claims, and accompanying drawings. [Brief description of the drawings]

[0011]

[0011] With reference to the following figures, exemplary embodiments of the present application are described in detail below. [Figure 1A]

[0012] 1 illustrates a computing device having a lane estimation system in accordance with certain aspects of the present disclosure. [Figure 1B]

[0013] FIG. 2 is a flow diagram illustrating an example process for lane estimation in accordance with certain aspects of the present disclosure. [Diagram 2]

[0014] 1 illustrates a scene including road lanes to be identified in accordance with certain aspects of the present disclosure. [Diagram 3]

[0015] 1 illustrates a bird's-eye view frame partitioned into regions, according to certain aspects of the present disclosure. [Figure 4]

[0016] FIG. 2 is a flow diagram illustrating an example process for lane estimation in accordance with certain aspects of the present disclosure. [Diagram 5]

[0017] FIG. 1 is a block diagram illustrating an embodiment of a deep learning network, in accordance with some embodiments. [Figure 6]

[0018] FIG. 1 is a block diagram illustrating an embodiment of a convolutional neural network, in accordance with some embodiments. [Figure 7]

[0019] FIG. 1 is a block diagram of an example computing device that can be used to implement some aspects of the techniques described herein, in accordance with some embodiments. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012]

[0020] Specific aspects of the present disclosure are provided below. As will be apparent to those skilled in the art, some of these aspects can be applied independently, and some of them can be applied in combination. In the following description, for the purpose of explanation, specific details are set forth in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent that various embodiments can be practiced without these specific details. The figures and descriptions are not intended to be limiting.

[0013]

[0021] The following description merely provides exemplary embodiments and is not intended to limit the scope, applicability, or configuration of the present disclosure. Rather, the following description of exemplary embodiments will provide those skilled in the art with an enabling description for implementing the exemplary embodiments. It should be understood that various changes can be made in the function and arrangement of elements without departing from the spirit and scope of the present application, as set forth in the appended claims.

[0014]

[0022] As discussed above, a system or device may determine the position and / or location of markings within an environment. In some illustrative examples, an autonomous system or device (e.g., an autonomous driving system for an autonomous vehicle, an autonomous drive system for a robotic system, among others) may facilitate navigation within a three-dimensional (3D) environment by detecting markings (e.g., road lanes or other navigation markings) within the environment. It may be important for the autonomous device or system to detect such markings and accurately navigate the space relative to such markings.

[0015]

[0023] Using an autonomous driving system as an exemplary embodiment, an important goal of such a system is the ability of an autonomous vehicle to detect lanes on a driving surface and accurately determine the extent of the drivable space on the road in relation to the detected lanes. The ability to detect and track lanes may be particularly important for higher levels of autonomy, such as autonomy level 3 and above. For example, autonomy level 0 is one in which the vehicle does not have an autonomous driving system and thus requires full control from the driver, and autonomy level 1 involves basic assistance functions such as cruise control, in which the driver of the vehicle has full control over the vehicle. Autonomy level 2 refers to semi-autonomous driving, in which the vehicle can perform functions such as driving a straight path, staying within a specific lane, controlling the distance from other vehicles in front of the vehicle, or other functions. Autonomy levels 3, 4, and 5 include more autonomy than levels 1 and 2. For example, autonomy level 3 refers to an on-board autonomous driving system capable of assuming all driving functions in certain situations, in which the driver remains ready to assume at any time if necessary. Autonomy level 4 refers to a fully autonomous experience that does not rely on user assistance in complex driving situations (e.g., on highways and in busy urban areas). Autonomy level 4 also allows a human to remain in the driver's seat behind the wheel. A vehicle operating at autonomy level 4 can communicate with other vehicles to inform them of upcoming maneuvers (e.g., the vehicle is changing lanes, turning, stopping, etc.). Autonomy level 5 vehicles are fully autonomous self-driving vehicles that operate autonomously in all conditions. A human operator is not required for the vehicle to take any action.

[0016]

[0024] Lane markers are important landmarks for the orientation of autonomous vehicles in traffic situations. They separate roads from non-drivable environments and partly determine driving rules. Automatic generation of street maps is another important application for lane marker detection. Lane detection techniques, applied either in generating highly automated driving (HAD) map data or in providing location information to moving vehicles, have attracted increasing interest in recent years. Lane detection techniques are widely considered in several vehicle safety systems, such as lane departure warning systems (LDWS) and advanced driver assistance systems (ADAS). Alternatively, prior map localization, commonly known as map-based localization or map matching, has attracted interest because the map can be accurately pre-constructed. To this end, a key step in assessing traffic situations is the proper detection and mapping of lanes and intersections. However, poor quality lines, sharp curves, irregular lane shapes, emerging and merging lanes, writing, other markings on the road (e.g., crosswalks), and various pavement materials make lane marking detection difficult.

[0017]

[0025] In some implementations, a lane estimation system can facilitate autonomous driving by using information from a light detection and ranging (LiDAR) sensor to identify various attributes of a road. For example, LiDAR sensor data can be used to identify lanes on a road, allowing an autonomous vehicle to stay within the boundaries set by the road lanes. LiDAR is a method for determining distance by targeting an object with a light source (e.g., a laser) and measuring the time it takes for the reflected light to return to a receiver. The output of a LiDAR sensor can be a LiDAR point cloud, which is a set of points each associated with information such as 3D coordinates (x, y, and z dimensions) and reflectivity (e.g., an indication of the intensity of the reflected light).

[0018]

[0026] Examples of lane estimation implementations using LiDAR point clouds include filtering, thresholding, and systematic fitting using techniques such as Hough or random sample consensus (RANSAC), which assume a fixed number of lanes in the sensor's field of view. Current lane estimation techniques using LiDAR may not identify attributes such as lane type (e.g., dashed or solid) or lane identifiers. Some current LiDAR lane estimation techniques are designed based on prior information (e.g., preset information), such as parallel lane markings or a fixed number of lanes, which may not always be true (e.g., in the case of merging and new appearances), which may adversely affect lane estimation accuracy. Furthermore, these techniques require manual adjustment of thresholding and filtering parameters to improve performance in various scenarios. Therefore, current LiDAR lane estimation techniques may be unreliable and unable to handle complexities such as faded lane markings, merging lanes, and new appearance lanes.

[0019]

[0027] Described herein are systems, apparatus, processes (or processes), and computer-readable media (collectively "systems and techniques") for performing lane estimation (e.g., using deep neural networks) based on light-based inputs, such as LiDAR inputs or other time-of-flight (ToF)-based inputs, structured light inputs, any combination thereof, and / or other light-based inputs. Although particular examples are described herein using LiDAR as an example of such light-based inputs, the systems and techniques may also be performed using any other type of light-based input, including, but not limited to, time-of-flight (ToF)-based inputs, structured light inputs, and / or other light-based inputs, any combination thereof, and / or other light-based inputs.

[0020]

[0028] Aspects of the present disclosure can be implemented without prior assumptions about lane geometry and can address faded lane markings without manual tuning of parameters. For example, a lane estimation system can receive a LiDAR point cloud from a LiDAR sensor for a three-dimensional (3D) environment (also referred to as 3D space). From the LiDAR point cloud, a subset of points can be extracted to enable a more efficient lane estimation process. Various techniques are described herein for extracting ground points that are important for lane estimation. As an example, the subset of points can include points that are on or near the ground plane.

[0021]

[0029] The lane estimation system can then map the extracted ground points to corresponding regions of a two-dimensional (2D) frame, such as a bird's-eye view (BEV) frame from a BEV perspective. For example, one or more of the extracted ground points can be mapped to each region of the BEV frame or other 2D frame. For each region of the BEV frame or other 2D frame, a set of features (or attributes) associated with the points mapped to that region can be calculated and used to perform lane estimation. In some examples, a trained machine learning model can receive the set of calculated features and identify road lanes in the 3D environment based on the set of calculated features.

[0022]

[0030] Although certain examples provided herein are described with respect to autonomous driving for ease of understanding, other types of systems may also benefit from lane detection or tracking. For example, robotic systems performing various operations may detect and track lanes for navigation purposes. In one exemplary embodiment, a robotic device used for cleaning (e.g., an autonomous vacuum cleaner) may detect the presence and location of lanes in an environment that are marked to avoid objects as the robotic device moves through the environment. In another exemplary embodiment, a robotic device used in manufacturing may need to know the exact location of lanes in an environment to avoid accidents. There are many other examples of systems that need to be able to identify lanes or various markings in an environment to facilitate autonomous tasks.

[0023]

[0031] FIG. 1A illustrates a computing device 102 having a lane estimation system 108 according to certain aspects of the disclosure. As illustrated, the computing device 102 may include one or more light-based sensors 105 that may provide a light-based output that may be used as an input to the lane estimation system 108. In some aspects, the one or more light-based sensors 105 may include a LiDAR sensor configured to generate a LiDAR point cloud. In some aspects, the one or more light-based sensors 105 may additionally or alternatively include one or more other types of time-of-flight (ToF)-based sensors, one or more structured light sensors, any combination thereof, and / or other light-based sensors. Using LiDAR as an illustrative example, the LiDAR point cloud output by the LiDAR sensor may be used as a LiDAR input to the lane estimation system 108.

[0024]

[0032] In some aspects, the computing device 102 may also include a communication interface 122 that facilitates communication with the network 104. The communication interface 122 may receive information such as a two-dimensional (2D) frame (e.g., a bird's eye view (BEV) frame from a BEV perspective), which may be provided to the lane estimation system 108. In some examples, the computing device 102 may also generate information such as a 2D frame (e.g., a BEV frame).

[0025]

[0033] Using light-based input (e.g., LiDAR input) and a 2D frame (e.g., BEV frame), the lane estimation system 108 can detect one or more lanes (e.g., on a road). In some aspects, the lane estimation system 108 can detect the type of lane, such as whether the lane is a broken line or a solid line. In some aspects, the computing device 102 may also include a processing system 124. The processing system 124 can receive the lane detection output from the lane estimation system 108 for further processing. For example, the processing system 124 can process the lane detection output to control an autonomous task, such as to perform an autonomous driving operation for an autonomous vehicle. In some aspects, the lane estimation system 108 and the processing system 124 can be implemented as part of the same processor (e.g., a central processing unit (CPU), a digital signal processor (DSP), a neural processing unit (NPU), a general-purpose microprocessor, an application specific integrated circuit (ASIC), a field programmable logic array (FPGA), any combination thereof, or other processor).

[0026]

[0034] 1B is a flow diagram illustrating an example process for lane estimation according to certain aspects of the present disclosure. The operations of process 100 may be performed by a lane estimation system (e.g., lane estimation system 108) or components of the lane estimation system, such as a processor 710 and, in some aspects, a storage device 730.

[0027]

[0035] At block 105, the lane estimation system may acquire LiDAR input. LiDAR is a remote sensing technology that uses light in the form of a pulsed laser to measure distances (variable distances) to various points in a 3D environment. The LiDAR input may provide information about an area, including roads and surrounding road areas. The LiDAR output may include a LiDAR point cloud, with each point associated with a 3D coordinate and reflectance (r), as described in more detail with respect to FIG. 2. As mentioned above, light-based inputs such as ToF data, structured light data, etc., other than or in addition to LiDAR data, may also be acquired and used by the process 100.

[0028]

[0036] 2 illustrates a scene 200 including a road 202 having lanes (e.g., lane 260), stationary objects (e.g., stationary object 204 or stationary object 212), and a vehicle 206 traveling on the road 202. In some cases, the scene 200 may be captured by a tracking vehicle (not shown). A LiDAR sensor 230 may capture information about the scene 200 (e.g., a LiDAR point cloud) and provide the information to the lane estimation system 108 for processing. The LiDAR sensor may be provided from any suitable source, such as a vehicle.

[0029]

[0037] The LiDAR sensor can generate a LiDAR point cloud that provides information about points on the scene 200. Several example points 208, 210, 211, 214, 216, 218 of the LiDAR point cloud are shown in FIG. 2, although the LiDAR point cloud may include many more points that are not shown in the scene 200. Each point can be associated with x, y, and z dimensions (e.g., indicating the 3D coordinates of the point) and a reflectance (r). Reflectance indicates the intensity associated with light reflecting from the LiDAR point. Reflectance can indicate various attributes about the LiDAR point, such as surface properties. Some points (e.g., point 214) may be on road lane 260, some points (e.g., point 208) may be on an area of ​​the road outside of any lane markings, some points (e.g., points 210, 211) may be outside the road 202, some points (e.g., point 216) may be on a stationary object (e.g., object 204), and some points (e.g., point 218) may be on a vehicle (e.g., vehicle 206).

[0030]

[0038] Referring back to FIG. 1B, in block 110, the lane estimation system can extract ground points from the LiDAR input (e.g., extract a subset of the LiDAR point cloud). Since lane markers are expected to be present on the road surface or at the boundary of the road surface, various points associated with the laser from the LiDAR sensor are deemed important for lane estimation and are extracted for further processing. For example, a subset of the LiDAR point cloud (also referred to as ground points) is extracted to reduce the number of points that must be processed (e.g., aggregated) during the detection process. Extracting the LiDAR ground points also facilitates avoiding any occlusions that may occur after the aggregation process by moving objects (e.g., vehicles) on the road. The extracted ground points can be points of the LiDAR input at or near the ground plane. Extracting the ground points can be performed using any plane segmentation technique.

[0031]

[0039] In some cases, ground points can be extracted to exclude points that represent objects (e.g., moving or stationary objects) on the road, but not all points on the object can be excluded when extracting ground points. For example, LiDAR point 210 can be on object 212 (e.g., a building) and can be excluded from extraction as a ground point. In some cases, the extracted ground points can be points that are on one or more flat surfaces or within a threshold distance (e.g., 0.1 meters, 0.25 meters, or other suitable distance) from one or more flat surfaces. For example, LiDAR point 211 can be an extracted ground point because it can be on a flat surface outside of road 202. In some aspects, the LiDAR point cloud can be processed to remove any points on the vehicle (e.g., vehicle 206) that are being captured by the LiDAR sensor, and any remaining points can be extracted ground points. For example, rather than using all points in the LiDAR point cloud, at least a portion of the LiDAR points representing objects that obstruct the view of the road can be removed, and the remaining points can be used as ground points for lane estimation. By extracting the ground points, subsequent processing steps can be performed more efficiently because any occlusions can be avoided by selecting a subset of the LiDAR point cloud.

[0032]

[0040] At block 120, the lane estimation system may aggregate ground points over time. For example, ground points from different LiDAR inputs captured at different times may be aggregated. Aggregating ground points over time improves the accuracy of the lane estimation technique by providing more information (e.g., denser images) to be used for lane estimation.

[0033]

[0041] The lane boundary detection techniques provided herein use Global Positioning System (GPS) and Inertial Measurement Unit (IMU) navigation data. The GPS and IMU data may be of sufficient quality to provide an Earth-relative 6-degree-of-freedom (DOF) pose estimation. 6D pose estimation is the task of detecting the 6D pose of an object, including its location and orientation. In some aspects, the Earth-relative 6-DOF pose estimation is used to aggregate extracted ground points (e.g., from LiDAR inputs captured at different times). For example, the orientation and location information from the 6-DOF pose estimation can be used to aggregate successive LiDAR frames. In some aspects, to perform the aggregation of extracted ground points over time, each LiDAR frame, initially in LiDAR sensor coordinates, is transformed to world coordinates (e.g., latitude and longitude) using the LiDAR sensor orientation and location information as provided by the 6-DOF pose estimation. In some cases, after the extracted ground points are transformed to world coordinates, they are concatenated and transformed back to LiDAR coordinates.

[0034]

[0042] The aggregation of extracted points can be causal for on-board (e.g., live or online) implementations, or non-causal for off-board (e.g., offline) implementations. For example, for on-board implementations, lane estimation can be performed at or near time t and can aggregate ground points acquired before time t. For off-board implementations, lane estimation can be performed with respect to time t based on an aggregation of ground points acquired before and after time t.

[0035]

[0043] At block 130, the ground points (e.g., aggregated ground points) can be projected onto a bird's eye view (BEV) frame. This projection is used to reduce the dimensionality of the point cloud (e.g., extracted ground points) since objects of interest may be on the same ground plane in the context of autonomous driving. Furthermore, the projection of the ground points onto the BEV frame allows the network to leverage prior information about the physical dimensions of the objects by preserving the distance space (e.g., a function of the distance between points).

[0036]

[0044] To perform the projection, the BEV can be divided into regions (e.g., regions of 30 cm length and 4 cm width). The ground points can then be quantized. For example, the extracted points can be mapped to regions on the BEV frame, and a particular feature associated with the points mapped to each region can be calculated. The features extracted from the BEV frame can be a representative intensity (e.g., average intensity), a maximum intensity, a representative height (e.g., average height), a maximum height, a normalized density of points falling within each region of the BEV frame, any combination thereof, and / or other features, as described in more detail herein.

[0037]

[0045] 3 illustrates a BEV frame that is divided into regions 300 (also referred to as cells) in accordance with certain aspects of the disclosure. As shown, a vehicle 206 may be present within region 345 of the BEV frame. A lane estimation system may determine which of the regions 300 include lane markings, and in some aspects, may determine the type of lane markings.

[0038]

[0046] From the LiDAR point cloud 350, a subset of points may be extracted as described above to generate extracted points 352. Each of the extracted ground points 352 may be processed and mapped to a region of the BEV frame such that one or more of the regions 300 each include one or more extracted points. In some cases, some of the regions 300 may map to extracted ground points, while some (e.g., at least one region) may not map to any ground points. For example, extracted points 306, 308, 310, 312 may map to region 304 as shown. Various features may be calculated for each of the regions 300. In one exemplary embodiment, such features may include a representative height (e.g., an average height or other representative height) associated with the points in the region, a representative intensity, such as a maximum height, an average intensity (e.g., an average reflectance of the points mapped to the region) or other representative intensity, a maximum intensity (e.g., a maximum reflectance of the points mapped to the region), a number of points in the region (e.g., a normalized density of the points dropped in each region), or any combination thereof. For example, for region 304, the average height associated with points 306, 308, 310, 312 may be a feature. The maximum height associated with points 306, 308, 310, 312 may be an additional or alternative feature (e.g., the height of one of points 306, 308, 310, 312 has the greatest height). The average intensity associated with points 306, 308, 310, 312 may also be an additional or alternative feature. The maximum intensity associated with points 306, 308, 310, 312 may be an additional or alternative feature (e.g., the intensity of one of points 306, 308, 310, 312 has the greatest intensity). The amount of points that are mapped to region 304 may be another example feature.

[0039]

[0047] 1B , the features of the regions of the BEV frame are provided to a machine learning model to perform lane boundary estimation at block 140. For example, the features of the regions may represent five channels (e.g., one channel each for average height, maximum height, average intensity, maximum intensity, and density) for the machine learning model to perform lane estimation.

[0040]

[0048] Given the features from the BEV frame, any suitable neural network, such as a row-wise classification network, can be used to estimate the dense lane marking vertices along with their attributes. Various attributes associated with the lane markings can be estimated via a machine learning model. Such attributes can include lane ID and type of lane marking (e.g., long-chain lane, solid lane, short-chain lane). For example, the machine learning model may indicate that a lane is present for region 304 and region 340 in FIG. 3. In some aspects, a classification-based network can be used to estimate the dense vertices of the lane markings. However, any suitable regression network can be employed to perform the lane estimation. The third dimension (e.g., z-dimension or height value) of the estimated lane marking vertex can be directly extracted from the average height feature channel in the associated input data at the estimated xy location output from the machine learning model. In some aspects, the neural network of the machine learning model can be implemented with an additional output layer to estimate the height associated with the lane marking vertex from the given input data.

[0041]

[0049] In some cases, a regression network employed for camera-based lane boundary estimation can be used. For example, a regression network for camera-based lane boundary estimation can be used to process two-dimensional data. The regression network can be implemented using a modified input layer to process data associated with additional feature channels (e.g., average height and maximum height).

[0042]

[0050] In some aspects, the machine learning model can be trained using an objective function. For example, candidate solutions can be provided to the model and evaluated against a training dataset. An error score (also referred to as the loss of the model) can be calculated by comparing the solutions to the training dataset. The machine learning model can be trained to minimize the error score. For example, the machine learning model can be trained by receiving a training dataset such as a representative height (e.g., average height), a maximum height, a representative intensity (e.g., average intensity), a maximum intensity, and a density associated with each of the regions of the BEV frame along with the candidate solutions (e.g., lane identification information such as lane boundaries and types), and the machine learning model can be trained based on this by adjusting the weights of the model. The process of training the machine learning model is described in more detail herein with respect to FIG. 5, FIG. 6.

[0043]

[0051] Once one or more road lanes are identified, the identified road lanes can be used to facilitate various operations associated with the autonomous task. For example, the identified lanes can be used by the device to locate the device (e.g., the vehicle's location) on a map. As another example, the vehicle can use the identified lanes to perform route planning operations. In some illustrative examples, route planning operations can include, among other things, planning for the vehicle to merge from one traffic lane or flow to another, staying in a lane while traveling, changing lanes, automatically braking or stopping (e.g., based on a determination that a lane is ending and one or more vehicles or other objects are occupying other nearby lanes), and considering vehicles in oncoming or other lanes.

[0044]

[0052] 4 is a flow diagram illustrating an example process 400 for lane estimation. The operations of process 400 may be performed by a lane estimation system, such as a processor 710 and, in some aspects, a storage device 730. In some cases, the lane estimation system may be part of the vehicle.

[0045]

[0053] At block 402, the lane estimation system receives a first input associated with a 3D space (e.g., any light-based input, such as a LiDAR point cloud or frames received from a sensor). For example, the first input can be received from one or more sensors of the vehicle. In some examples, the first input includes a LiDAR point cloud received from one or more LiDAR sensors of the vehicle.

[0046]

[0054] At block 404, the lane estimation system extracts a first set of points from the first input (e.g., from a first LiDAR point cloud, such as LiDAR point cloud 350) that are associated with a ground plane in the 3D space. For example, extracting the first set of points associated with the ground plane may involve selecting a subset of the points of the first input (e.g., a subset of points from the first LiDAR point cloud), which includes points that reside on a flat surface or within a threshold distance (e.g., 0.1 meters, 0.25 meters, 0.5 meters, or other suitable distance) from the flat surface. In some aspects, the first set of points includes one or more points of the first input (e.g., one or more points of the first LiDAR point cloud) that reside on the ground plane. In some aspects, extracting the first set of points associated with the ground plane may involve excluding one or more points of the first input (e.g., one or more points of the first LiDAR point cloud) that lie on one or more objects (e.g., another vehicle blocking the view of the road). In some aspects, the lane estimation system may extract the first set of points based on GPS or IMU data.

[0047]

[0055] In block 406, the lane estimation system maps each of the first set of points to one of a plurality of regions (e.g., region 300) of a two-dimensional frame, such as a BEV frame. In some aspects, the lane estimation system may also receive a second input (e.g., a second LiDAR point cloud) associated with a 3D space, extract from the second input (e.g., the second LiDAR point cloud) a second set of points associated with a ground plane of the 3D space, and aggregate the first set of points and the second set of points to generate aggregated points. In this case, mapping each of the first set of points in block 406 may involve mapping each of the aggregated points to one of a plurality of regions.

[0048]

[0056] At block 408, the lane estimation system determines (e.g., calculates) one or more attributes associated with each of the multiple regions based on one or more of the first set of points mapped to the region. The one or more attributes may include at least one of a representative height (e.g., average height) associated with one or more of the first set of points mapped to the region, a maximum height associated with one or more of the first set of points mapped to the region, a representative intensity (e.g., average intensity) associated with one or more of the first set of points mapped to the region, a maximum intensity associated with one or more of the first set of points mapped to the region, and a quantity of one or more of the first set of points mapped to the region.

[0049]

[0057] At block 410, the lane estimation system identifies one or more road lanes based on the one or more attributes. For example, identifying the one or more road lanes may involve analyzing the one or more attributes using a machine learning model. The machine learning model may include a row-wise machine learning model or a classification neural network. In some aspects, identifying the one or more road lanes may involve identifying a type of each of the one or more road lanes (e.g., whether each lane is a solid or dashed line).

[0050]

[0058] In some aspects, the lane estimation system locates the device on a map based on one or more road lanes. In some aspects, the lane estimation system can perform path planning operations based on one or more road lanes.

[0051]

[0059] FIG. 5 is an exemplary embodiment of a deep learning neural network 500 that may be used by a lane estimation system. The input layer 520 includes input data. In one exemplary embodiment, the input layer 520 may include data representing pixels of an input image or video frame. The neural network 500 includes multiple hidden layers 522a, 522b, . . . The hidden layers 522a, 522b, . . . include "n" number of hidden layers, where "n" is an integer equal to or greater than 1. The number of hidden layers may include as many layers as are required for a given application. The neural network 500 further includes an output layer 524 that provides an output resulting from the processing performed by the hidden layers 522a, 522b, . . . In one exemplary embodiment, the output layer 524 may provide a classification for objects in the input image or video frame. The classification may include classes that identify the type of lane (eg, stationary object, vehicle, person, dog, cat, or other object).

[0052]

[0060] Neural network 500 is a multi-layered neural network of interconnected nodes. Each node can represent one piece of information. The information associated with the nodes is shared between different layers, and each layer retains the information as it is processed. In some cases, neural network 500 may include a feed-back network, where there are no feedback connections, where the output of the network is fed back into itself. In some cases, neural network 500 may include a recurrent neural network, which may have loops that allow information to be conveyed across nodes while reading at the input.

[0053]

[0061] Information can be exchanged between nodes via node-to-node interconnections between the various layers. The nodes of the input layer 520 can activate a set of nodes in the first hidden layer 522a. For example, as shown, each of the input nodes of the input layer 520 is connected to each of the nodes of the first hidden layer 522a. The nodes of the hidden layers 522a, 522b, . . . 522n can transform the information of each input node by applying an activation function to the information. The information derived from that transformation can then be passed to the nodes of the next hidden layer 522b, which can be activated, and which can perform their own specified function. Exemplary functions include convolution, upsampling, data transformation, and / or any other suitable function. The output of the hidden layer 522b can then activate the nodes of the next hidden layer, and so on. The output of the last hidden layer 522n can activate one or more nodes of the output layer 524, which provide the output. In some cases, a node in neural network 500 (e.g., node 526) is shown as having multiple output lines; however, the node has a single output, and all lines shown as outputting from the node represent the same output value.

[0054]

[0062] In some cases, each node, or the interconnections between nodes, may have weights, which are a set of parameters derived from training the neural network 500. Once the neural network 500 is trained, it may be referred to as a trained neural network and may be used to classify one or more objects. For example, the interconnections between nodes may represent a piece of information that has been learned about the nodes that are interconnected. The interconnections may have adjustable numerical weights that may be adjusted (e.g., based on a training data set) to allow the neural network 500 to be adaptive to the input and to learn as more data is processed.

[0055]

[0063] Neural network 500 is pre-trained to process features from data in input layer 520 using various hidden layers 522a, 522b-522n to provide output via output layer 524. In one embodiment where neural network 500 is used to identify objects in images, neural network 500 can be trained using training data that includes both images and labels. For example, training images can be input into the network, where each training image has a label that indicates the class of one or more objects in each image (essentially telling the network what the objects are and what characteristics those objects have). In one exemplary embodiment, the training images can include images of the number 2, where the label for that image can be [0 0 1 0 0 0 0 0 0 0].

[0056]

[0064] In some cases, the neural network 500 can adjust the node weights using a training process called backpropagation. Backpropagation can include a forward pass, a loss function, a backward pass, and a weight update. The forward pass, loss function, backward pass, and parameter update are performed for one training iteration. This process can be repeated for each set of training images for a certain number of iterations until the neural network 500 is trained well enough that the layer weights are accurately adjusted.

[0057]

[0065] For the example of identifying lanes, the forward pass may include passing training features (e.g., average height, maximum height, average intensity, maximum intensity, and density) through neural network 500. Before neural network 500 is trained, the weights are first randomized. The training features may include representative height (e.g., average height), maximum height, representative intensity (e.g., average intensity), maximum intensity, and density, as described above.

[0058]

[0066] For the first training iteration for the neural network 500, due to the random weights selected during initialization, the output is likely to contain values ​​that do not favor any particular class. For example, if the output is a vector with the probabilities of the lane containing different types, the probability values ​​for each of the different types may be equal or at least very similar (e.g., for 10 possible classes, each class may have a probability value of 0.1). With these initial weights, the neural network 500 is unable to determine low-level features and therefore cannot accurately determine what the classification (e.g., type) of the lane is. A loss function can be used to analyze the error in the output. Any suitable loss function definition can be used. One example of a loss function is the mean squared error (MSE). MSE is calculated by summing the squared actual response minus the predicted (output) response times 1 / 2.

[0059]

number

[0060] The loss is defined as E total can be set equal to the value of

[0061]

[0067] For the first training images, the loss (or error) will be large because the actual values ​​will be very different from the predicted outputs. The goal of training is to minimize the amount of loss so that the predicted outputs are the same as the training labels. The neural network 500 can perform a backward pass by determining which inputs (weights) contributed most to the network's loss, and adjust the weights so that the loss is reduced and eventually minimized.

[0062]

[0068] To determine which weights contributed most to the network's loss, the derivative of the loss with respect to the weights (denoted as dL / dW, where W is the weight in a particular layer) can be calculated. After calculating the derivative, a weight update can be performed by updating all the weights of the filter. For example, the weights can be updated so that they change in the opposite direction of the gradient. A weight update is

[0063]

number

[0064] where w represents the weight and w i represents the initial weights and η represents the learning rate, which can be set to any suitable value, with a high learning rate involving larger weight updates and a lower value indicating smaller weight updates.

[0065]

[0069] Neural network 500 may include any suitable deep network. One example is a convolutional neural network (CNN), which includes an input layer and an output layer with multiple hidden layers between the input layer and the output layer. One example of a CNN is described below with respect to FIG. 6. The hidden layers of a CNN include a series of convolutional layers, nonlinear layers, pooling layers (for downsampling), and fully connected layers. Neural network 500 may include any other deep network other than a CNN, such as autoencoders, deep belief nets (DBNs), recurrent neural networks (RNNs), among others.

[0066]

[0070] FIG. 6 is an exemplary embodiment of a convolutional neural network 600 (CNN 600). The input layer 620 of the CNN 600 includes data representing an image. For example, this data may include an array of numbers representing pixels of the image, with each number in the array including a value between 0 and 255 that describes the pixel intensity at that location in the array. Using the previous example from above, the array may include a 28×28×3 array of numbers, with 28 rows and 28 columns of pixels and three color components (e.g., a red component, a green component, and a blue component, or a luma component and two chroma components, etc.). The image may be passed through a convolutional hidden layer 622a, an optional nonlinear activation layer, a pooling hidden layer 622b, and a fully connected hidden layer 622c to obtain an output at the output layer 624. 6, only one of each hidden layer is shown, but one skilled in the art will appreciate that multiple convolutional hidden layers, non-linear layers, pooling hidden layers, and / or fully connected layers may be included within CNN 600. As previously mentioned, the output may indicate a single class of object (e.g., a lane) or may include a probability of the class that best describes the object in the image.

[0067]

[0071] The first layer of the CNN 600 is the convolutional hidden layer 622a. The convolutional hidden layer 622a analyzes the image data of the input layer 620. Each node of the convolutional hidden layer 622a is connected to a region of nodes (pixels) of the input image, called the receptive field. The convolutional hidden layer 622a can be viewed as one or more filters (each filter corresponding to a different activation map or feature map), and each convolutional iteration of a filter is a node or neuron of the convolutional hidden layer 622a. For example, the region of the input image that a filter covers in each convolutional iteration becomes the receptive field for that filter. In one exemplary embodiment, if the input image includes a 28×28 array and each filter (and corresponding receptive field) is a 5×5 array, there will be 24×24 nodes in the convolutional hidden layer 622a. Each node learns to analyze its particular local receptive field in the input image by learning a weight and, in some cases, a global bias for each connection between the node and the receptive field associated with that node. Each node in the hidden layer 622a will have the same weights and biases (called shared weights and shared biases). For example, the filter will have an array of weights (numbers) and the same depth as the input. For an example of an image or video frame, the filter will have a depth of 3 (according to the three color components of the input image). The size of an exemplary example of the filter array is 5×5×3, corresponding to the size of the receptive field of the node.

[0068]

[0072] The convolutional nature of the convolutional hidden layer 622a results from each node of the convolutional layer being applied to its corresponding receptive field. For example, the filter of the convolutional hidden layer 622a may start at the top left corner of the input image array and convolve around the input image. As mentioned above, each convolutional iteration of the filter may be considered as a node or neuron of the convolutional hidden layer 622a. In each convolutional iteration, the value of the filter is multiplied with the corresponding numerical value of the original pixel value of the image (e.g., a 5×5 filter array is multiplied with a 5×5 array of input pixel values ​​in the top left corner of the input image array). The multiplications from each convolutional iteration may be summed to obtain a total sum for that iteration or node. This process is then continued at the next location in the input image according to the receptive field of the next node in the convolutional hidden layer 622a. For example, the filter may be moved to the next receptive field in a step amount. The step amount may be set to 1 or other suitable amount. For example, if the step amount is set to 1, the filter will be moved one pixel to the right in each convolution iteration. Processing the filter at each unique location in the input volume produces a numerical value representing the filter result for that location, which in turn determines a summation value for each node in the convolution hidden layer 622a.

[0069]

[0073] The mapping from the input layer to the convolutional hidden layer 622a is called an activation map (or feature map). The activation map contains a value for each node that represents the filter result at each location of the input volume. The activation map may include an array that contains the various sum values ​​resulting from each iteration of the filter on the input volume. For example, if a 5×5 filter is applied to each pixel of a 28×28 input image (with a step size of 1), the activation map will include a 24×24 array. The convolutional hidden layer 622a may include several activation maps to identify multiple features in an image. The example shown in FIG. 6 includes three activation maps. Using the three activation maps, the convolutional hidden layer 622a can detect three different types of features, each detectable throughout the image.

[0070]

[0074] In some embodiments, a nonlinear hidden layer can be applied after the convolutional hidden layer 622a. A nonlinear layer can be used to introduce nonlinearity into a system that was previously computing linear operations. One illustrative embodiment of a nonlinear layer is a rectified linear unit (ReLU) layer. The ReLU layer can apply a function f(x)=max(0,x) to all of the values ​​in the input volume, which changes all negative activations to 0. Therefore, the ReLU can increase the nonlinear properties of the CNN 600 without affecting the receptive field of the convolutional hidden layer 622a.

[0071]

[0075] A pooling hidden layer 622b may be applied after the convolutional hidden layer 622a (and after the non-linear hidden layer, if used). The pooling hidden layer 622b is used to simplify the information in the output from the convolutional hidden layer 622a. For example, the pooling hidden layer 622b may take each activation map output from the convolutional hidden layer 622a and use a pooling function to generate a condensed activation map (or feature map). Max pooling is an example of a function performed by the pooling hidden layer. Other forms of pooling functions, such as average pooling, L2 norm pooling, or other suitable pooling functions, may be used by the pooling hidden layer 622b. A pooling function (e.g., a max pooling filter, an L2 norm filter, or other suitable pooling filter) is applied to each activation map included in the convolutional hidden layer 622a. In the example shown in FIG. 6, three pooling filters are used for the three activation maps in the convolutional hidden layer 622a.

[0072]

[0076] In some implementations, max pooling can be used by applying a max pooling filter (e.g., having a size of 2×2) to the activation map output from the convolutional hidden layer 622a in a step amount (e.g., equal to the dimensions of the filter, such as a step amount of 2). The output from the max pooling filter contains the maximum numerical value in every sub-region that the filter convolves around. Using a 2×2 filter as an example, each unit in the pooling layer can summarize a region of 2×2 nodes (each node is a value in the activation map) in the previous layer. For example, four values ​​(nodes) in the activation map would be analyzed by a 2×2 max pooling filter at each iteration of the filter, and the maximum value from those four values ​​would be output as the “max” value. If such a max pooling filter is applied to an activation filter from the convolutional hidden layer 622a, which has dimensions of 24×24 nodes, the output from the pooling hidden layer 622b would be an array of 12×12 nodes.

[0073]

[0077] In some embodiments, an L2 norm pooling filter can also be used, which involves computing the square root of the sum of the squares of the values ​​in a 2×2 region (or other suitable region) of the activation map (rather than computing the maximum value as is done in max pooling) and using the computed value as the output.

[0074]

[0078] Intuitively, a pooling function (e.g., max pooling function, L2 norm pooling function, or other pooling function) determines whether a given feature is found anywhere in a region of the image, and the exact location information is discarded. This can be done without affecting the outcome of the feature detection, because once a feature is found, the exact location of the feature is not as important as its approximate location relative to other features. Max pooling (as well as other pooling methods) offers the advantage that far fewer features are pooled, thus reducing the number of parameters required in subsequent layers of the CNN 600.

[0075]

[0079] The final layer of connections in the network is a fully connected layer that connects every node from the pooling hidden layer 622b to every single output node in the output layer 624. Using the above example, the input layer includes 28×28 nodes that encode pixel intensities of the input image, the convolutional hidden layer 622a includes 3×24×24 hidden feature nodes based on application of 5×5 local receptive fields (for the filters) to three activation maps, and the pooling hidden layer 622b includes a layer of 3×12×12 hidden feature nodes based on application of max pooling filters to 2×2 regions across each of the three feature maps. Extending this example, the output layer 624 may include 10 output nodes. In such an example, every node of the 3×12×12 pooling hidden layer 622b is connected to every node of the output layer 624.

[0076]

[0080] The fully connected layer 622c can take the output of the previous pooling layer 622b (which should represent an activation map of high level features) and determine which features are most correlated to a particular class. For example, the fully connected layer 622c can determine which high level features are most strongly correlated to a particular class and can include weights (nodes) for the high level features. By calculating the product between the weights of the fully connected layer 622c and the weights of the pooling hidden layer 622b, probabilities for various classes can be obtained. For example, if the CNN 600 is being used to predict that an object in an image or video frame is a vehicle, there will be high values ​​in the activation map that represents the high level features of a vehicle (e.g., two or four tires, windshield, side mirrors, etc.).

[0077]

[0081] In some embodiments, the output from the output layer 624 may include an M-dimensional vector (in the preceding embodiment, M=10), where M may include the number of classes the program must choose from when classifying objects in an image. Other exemplary outputs may also be provided. Each number in the N-dimensional vector may represent the probability that the object is of a particular class. In one exemplary embodiment, a 10-dimensional output vector representing 10 different classes of objects may be [0 0 0.05 0.8 0 0.15 0 0 0 0], which indicates that there is a 5% probability that the image is a third class of object (e.g., a person), an 80% probability that the image is a fourth class of object (e.g., a stationary object on a road or other driving surface), and a 15% probability that the image is a sixth class of object (e.g., a vehicle). The probabilities for a class may be considered as a level of confidence that the object is part of that class.

[0078]

[0082] Figure 7 illustrates an example of a system for implementing certain aspects of the present technology. In particular, Figure 7 illustrates an example of a computing system 700, which may be any computing device, such as an internal computing system, a remote computing system, a camera, or any of the components thereof, in which the components communicate with each other using a connection 705. The connection 705 may be a physical connection using a bus or a direct connection to a processor 710, such as in a chipset architecture. The connection 705 may also be a virtual connection, a network connection, or a logical connection.

[0079]

[0083] In some aspects, computing system 700 is a distributed system, allowing the functionality described in this disclosure to be distributed across a data center, multiple data centers, within a peer network, etc. In some aspects, one or more of the system components described represent many such components, each performing some or all of the functionality described with respect to that component. In some aspects, these components may be physical or virtual devices.

[0080]

[0084] The exemplary system 700 includes at least one processing unit (CPU or processor) 710 and connections 705 coupling various system components to the processor 710, including system memory 715, such as read only memory (ROM) 720 and random access memory (RAM) 725. The computing system 700 may include a cache 712 of high-speed memory, either directly connected to the processor 710, connected in close proximity to the processor 710, or integrated as part of the processor 710.

[0081]

[0085] The processor 710 may include any general-purpose processor and hardware or software services. In some aspects, code stored in the storage device 730 may be configured to control the processor 710 to perform the operations described herein. In some aspects, the processor 710 may be a special-purpose processor in which instructions or circuits are built into the actual processor design to perform the operations described herein. The processor 710 may essentially be a fully self-contained computing system including multiple cores or processors, buses, memory controllers, caches, etc. The multi-core processor may be symmetric or asymmetric. For example, the processor 710 may include a circuit 760 for receiving (e.g., receiving a LiDAR point cloud), a circuit 762 for extracting (e.g., extracting points from the LiDAR point cloud), a circuit 764 for mapping (e.g., mapping the extracted points to regions of the BEV frame), a circuit 766 for calculating (e.g., calculating attributes associated with regions of the BEV frame), and a circuit 768 for identifying (e.g., identifying road lanes). In some aspects, the processor 710 also optionally includes a circuit 769 for aggregating.

[0082]

[0086] The storage device 730 may store code that, when executed by the processor 710, performs operations described herein. For example, the storage device 730 may include code 770 for receiving (e.g., receiving a LiDAR point cloud), code 772 for extracting (e.g., extracting points from the LiDAR point cloud), code 774 for mapping (e.g., mapping the extracted points to regions of the BEV frame), code 776 for calculating (e.g., calculating attributes associated with regions of the BEV frame), and code 778 for identifying (e.g., identifying road lanes). In some aspects, the storage device 730 may include code 780 for aggregating.

[0083]

[0087] To enable user interaction, computing system 700 includes input devices 745, which may represent any number of input mechanisms, such as a microphone for speaking, a camera for generating images or video, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, speech, etc. Computing system 700 may also include output devices 735, which may be one or more of a number of output mechanisms. In some instances, a multimodal system may enable a user to provide multiple types of input / output to communicate with computing system 700. Computing system 700 may include a communication interface 740, which may generally govern and manage user input and system output.The communications interface may be an audio jack / plug, a microphone jack / plug, a Universal Serial Bus (USB) port / plug, an Apple® Lightning® port / plug, an Ethernet port / plug, an optical fiber port / plug, a proprietary wired port / plug, BLUETOOTH® wireless signal transmission, BLUETOOTH® Low Energy (BLE) wireless signal transmission, IBEACON® wireless signal transmission, Radio Frequency Identification (RFID) wireless signal transmission, Near Field Communication (NFC) wireless signal transmission, Dedicated Short Range Communications (DSRC) wireless signal transmission, 802.11 Wi-Fi wireless signal transmission, a Wireless Local Area Network (WLAN) ) signal transfer, visible light communication (VLC), Worldwide Interoperability for Microwave Access (WiMAX), infrared (IR) communication wireless signal transfer, public switched telephone network (PSTN) signal transfer, integrated services digital network (ISDN) signal transfer, 3G / 4G / 5G / LTE cellular data network wireless signal transfer, ad-hoc network signal transfer, radio wave signal transfer, microwave signal transfer, infrared signal transfer, visible light signal transfer, ultraviolet light signal transfer, wireless signal transfer along the electromagnetic spectrum, or any combination thereof. Communications interface 740 may also include one or more Global Navigation Satellite System (GNSS) receivers or transceivers used to determine a location of computing system 700 based on reception of one or more signals from one or more satellites associated with one or more Global Navigation Satellite System (GNSS) systems. GNSS systems include, but are not limited to, the United States-based Global Positioning System (GPS), the Russia-based Global Navigation Satellite System (GLONASS), the China-based Beidou Navigation Satellite System (BDS), and the Europe-based Galileo GNSS.There are no constraints to operating on any particular hardware configuration, and therefore the basic features herein can be readily substituted for improved hardware or firmware configurations as they are developed.

[0084]

[0088] The storage device 730 can be a non-volatile and / or non-transitory and / or computer readable memory device, and can be a magnetic cassette, a flash memory card, a solid state memory device, a digital versatile disk, a cartridge, a floppy disk, a flexible disk, a hard disk, a magnetic tape, a magnetic strip / stripe, any other magnetic storage medium, a flash memory, a memristor memory, any other solid state memory, a compact disk-read only memory (CD-ROM) optical disk, a rewritable compact disk (CD) optical disk, a digital video disk (DVD) optical disk, a Blu-ray disk (BDD) optical disk, a holographic optical disk, another optical medium, a secure digital (SD) card, a micro secure digital (microSD) card, a memory stick card, a smart card chip, an EMV chip, a subscriber identity module (SIM) card, The SIM card may be a hard disk or other type of computer readable medium capable of storing data that is accessible by a computer, such as a SIM card, a mini / micro / nano / pico SIM card, another integrated circuit (IC) chip / card, a random access memory (RAM), a static RAM (SRAM), a dynamic RAM (DRAM), a read only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (EPROM), an electrically erasable programmable read only memory (EEPROM), a flash EPROM (FLASHEPROM), a cache memory (L1 / L2 / L3 / L4 / L5 / L#), a resistive random access memory (RRAM / ReRAM), a phase change memory (PCM), a spin transfer torque RAM (STT-RAM), another memory chip or cartridge, and / or a combination thereof.

[0085]

[0089] Storage device 730 may include software services, servers, services, etc., where code defining such software, when executed by processor 710, causes the processor to perform functions in the system. In some aspects, hardware services that perform a particular function may include software components stored in a computer readable medium in association with the necessary hardware components, such as processor 710, connections 705, output devices 735, etc., to perform that function.

[0086]

[0090] The term "computer-readable medium" includes, but is not limited to, portable or non-portable storage devices, optical storage devices, and various other media capable of storing, storing, or transporting instructions and / or data. Computer-readable media may include non-transitory media capable of storing data, including carrier waves and / or ephemeral electronic signals propagating wirelessly or over a wired connection. Examples of non-transitory media include, but are not limited to, magnetic disks or tapes, optical storage media such as compact disks (CDs) or digital versatile disks (DVDs), flash memory, memories, or memory devices. Code and / or machine-executable instructions may be stored on a computer-readable medium, which may represent a procedure, a function, a subprogram, a program, a routine, a subroutine, a module, a software package, a class, or any combination of instructions, data structures, or program statements. A code segment may be coupled to another code segment or a hardware circuit by passing and / or receiving information, data, arguments, parameters, or memory contents. Information, arguments, parameters, data, etc. can be passed, forwarded, or transmitted via any suitable means including memory sharing, message passing, token passing, network transmission, etc.

[0087]

[0091] In some embodiments, computer-readable storage devices, media, and memories may include cable or wireless signals, including bitstreams, etc. However, when mentioned, non-transitory computer-readable storage media explicitly excludes media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0088]

[0092] Specific details are provided in the above description to provide a thorough understanding of the embodiments and examples provided herein. However, it will be understood by those skilled in the art that the embodiments can be practiced without these specific details. For clarity of explanation, in some cases, the present technology may be presented as including individual functional blocks, including functional blocks including devices, device components, steps or routines in a method embodied in software or a combination of hardware and software. Additional components other than those shown in the figures and / or described herein may be used. For example, circuits, systems, networks, processes, and other components may be shown as components in block diagram form so as not to obscure the embodiments in unnecessary detail. In other cases, well-known circuits, processes, algorithms, structures, and techniques may be shown without unnecessary detail so as to avoid obscuring the embodiments.

[0089]

[0093] Particular embodiments may be described above as a process or method that is depicted as a flowchart, a flow diagram, a data flow diagram, a structure diagram, or a block diagram. Although the flowcharts may describe operations as a sequential process, many of the operations may be performed in parallel or simultaneously. In addition, the order of operations may be rearranged. A process is terminated when the operations are completed, but may have additional steps not included in the figures. A process may correspond to a method, a function, a procedure, a subroutine, a subprogram, etc. When a process corresponds to a function, its termination may correspond to a return of the function to the calling function or to the main function.

[0090]

[0094] The processes and methods according to the above-described embodiments may be implemented using computer-executable instructions stored on or otherwise available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions, or that otherwise configure a general-purpose computer, a special-purpose computer, or a processing device to perform a particular function or group of functions. Portions of the computer resources used may be accessible over a network. The computer-executable instructions may be, for example, binary or intermediate format instructions such as assembly language, firmware, source code, etc. Examples of computer-readable media that may be used to store instructions, information used, and / or information created during the methods according to the described embodiments include magnetic or optical disks, flash memory, USB devices with non-volatile memory, networked storage devices, etc.

[0091]

[0095] Devices implementing the processes and methods according to these disclosures may include hardware, software, firmware, middleware, microcode, hardware description languages, or any combination thereof, and may take any of a variety of form factors. When implemented in software, firmware, middleware, or microcode, program code or code segments (e.g., computer program products) for performing the necessary tasks may be stored in a computer-readable or machine-readable medium. A processor may perform the necessary tasks. Typical examples of form factors include laptops, smartphones, mobile phones, tablet devices or other small-footprint personal computers, personal digital assistants, rack-mounted devices, stand-alone devices, and the like. The functionality described herein may also be embodied in peripherals or add-in cards. Such functionality may also be implemented on a circuit board, among various chips or various processes executed within a single device, as further examples.

[0092]

[0096] The instructions, media for carrying such instructions, computing resources for executing those instructions, and other structures for supporting such computing resources are exemplary means for providing the functionality described in this disclosure.

[0093]

[0097] In the foregoing description, aspects of the present application have been described with reference to specific embodiments thereof, but those skilled in the art will recognize that the present application is not limited thereto. Thus, while exemplary embodiments of the present application have been described in detail herein, it should be understood that the concepts of the present invention can be variously embodied and employed in other manners, and that the appended claims are intended to be construed to include such variations, except insofar as limited by the prior art. The various features and aspects of the above-described application can be used individually or jointly. Moreover, the embodiments can be utilized in any number of environments and applications beyond those described herein without departing from the broader spirit and scope of the present specification. Thus, the present specification and drawings should be regarded as illustrative and not restrictive. For purposes of illustration, methods have been described in a particular order. It should be understood that in alternative embodiments, methods can be performed in an order different from that described.

[0094]

[0098] Those skilled in the art will understand that the less than ("<") and greater than (">") symbols or terms used herein may be replaced with the less than or equal to ("≦") and greater than or equal to ("≧") symbols, respectively, without departing from the scope of this description.

[0095]

[0099] When a component is described as being "configured to" perform a particular operation, such configuration may be achieved, for example, by designing electronic circuitry or other hardware to perform that operation, by programming a programmable electronic circuitry (e.g., a microprocessor or other suitable electronic circuitry) to perform that operation, or any combination thereof.

[0096]

[0100] The phrase "coupled to" refers to any component that is physically connected, either directly or indirectly, to another component and / or any component that is in communication, either directly or indirectly, with another component (e.g., connected to the other component via a wired or wireless connection, and / or other suitable communication interface).

[0097]

[0101] Claim language or other language reciting "at least one of" a set and / or "one or more" of a set indicates that one member of the set or multiple members of the set (in any combination) satisfy the claim. For example, a claim language reciting "at least one of A and B" means A, B, or A and B. In another example, a claim language reciting "at least one of A, B, and C" means A, B, C, or A and B, or A and C, or B and C, or A, B, and C. The language "at least one of" a set and / or "one or more" of a set does not limit the set to the items listed in the set. For example, a claim language reciting "at least one of A and B" can mean A, B, or A and B, and can also include items not listed in the set of A and B.

[0098]

[0102] The various exemplary logic blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, firmware, or a combination thereof. To clearly illustrate this interchangeability of hardware and software, the various exemplary components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends on the specific application and design constraints imposed on the overall system. Those skilled in the art may implement the described functionality in various ways for each specific application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0099]

[0103] The techniques described herein may also be implemented in electronic hardware, computer software, firmware, or any combination thereof. Such techniques may be implemented in any of a variety of devices, such as a general purpose computer, a wireless communication device handset, or an integrated circuit device having multiple uses, including applications in wireless communication device handsets and other devices. Any features described as modules or components may be implemented together in an integrated logic device or separately in separate but interoperable logic devices. If implemented in software, these techniques may be realized at least in part by a computer-readable data storage medium comprising program code including instructions that, when executed, perform one or more of the methods described above. The computer-readable data storage medium may form part of a computer program product, which may include packaging materials. The computer-readable medium may include a memory or data storage medium, such as a random access memory (RAM), such as a synchronous dynamic random access memory (SDRAM), a read-only memory (ROM), a non-volatile random access memory (NVRAM), an electrically erasable programmable read-only memory (EEPROM), a FLASH memory, a magnetic or optical data storage medium, or the like. These techniques may also, or alternatively, be realized at least in part by a computer-readable communications medium, such as a propagated signal or wave, that carries or communicates program code in the form of instructions or data structures and that can be accessed, read, and / or executed by a computer.

[0100]

[0104] The program code may be executed by a processor, which may include one or more processors, such as one or more digital signal processors (DSPs), general-purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry. Such a processor may be configured to perform any of the techniques described in this disclosure. A general-purpose processor may be a microprocessor, but alternatively, the processor may be any conventional processor, controller, microcontroller, or state machine. A processor may also be implemented as a combination of computing devices, such as a combination of a DSP and a microprocessor, multiple microprocessors, one or more microprocessors in conjunction with a DSP core, or any other such configuration. Thus, the term "processor" as used herein may refer to any of the above structures, any combination of the above structures, or any other structure or apparatus suitable for implementing the techniques described herein.

[0101]

[0105] Exemplary aspects of the present disclosure include:

[0106] Aspect 1: An apparatus for lane estimation comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor configured to receive a first input associated with a three-dimensional (3D) space; extract from the first input a first set of points associated with a ground plane in the 3D space; map each of the first set of points to a region of a plurality of regions of a two-dimensional (2D) frame; determine one or more attributes associated with each region of the plurality of regions based on one or more of the first set of points mapped to the region; and identify one or more road lanes based on the one or more attributes.

[0102]

[0107] Example 2: The device of example 1, wherein the first input includes a light-based input.

[0103]

[0108] Example 3: The device of example 2, wherein the light-based input includes a Light Detection and Ranging (LiDAR) point cloud.

[0104]

[0109] Embodiment 4: The device of any of embodiments 1 to 3, wherein the 2D frame includes a bird's-eye view (BEV) frame.

[0105]

[0110] Aspect 5: The apparatus of any of aspects 1-4, wherein at least one processor is configured to analyze one or more attributes using a machine learning model to identify one or more road lanes.

[0106]

[0111] Example 6: The apparatus of example 5, wherein the machine learning model comprises a line-wise classification machine learning model.

[0107]

[0112] Example 7: The apparatus of any of examples 5 or 6, wherein the machine learning model comprises a classification neural network.

[0108]

[0113] Aspect 8: The apparatus of any of aspects 1-7, wherein to identify one or more road lanes, at least one processor is configured to identify a respective type of the one or more road lanes.

[0109]

[0114] Aspect 9: The apparatus of any of aspects 1-8, wherein the at least one processor is further configured to receive a second input associated with a 3D space, extract from the second input a second set of points associated with a ground plane in the 3D space, and aggregate the first set of points and the second set of points to generate aggregated points, and wherein to map each of the first set of points, the at least one processor is configured to map each of the aggregated points to one of a plurality of regions.

[0110]

[0115] Example 10: The apparatus of example 9, wherein the first input includes a first Light Detection and Ranging (LiDAR) point cloud and the second input includes a second LiDAR point cloud.

[0111]

[0116] Aspect 11: The apparatus of any of aspects 1-10, wherein the one or more attributes include at least one of: a representative height associated with one or more of the first set of points mapped to the region; a maximum height associated with one or more of the first set of points mapped to the region; a representative intensity associated with one or more of the first set of points mapped to the region; a maximum intensity associated with one or more of the first set of points mapped to the region; and a quantity of one or more of the first set of points mapped to the region.

[0112]

[0117] Aspect 12: The apparatus of any of aspects 1-11, wherein to extract the first set of points associated with the ground plane, at least one processor is configured to select a subset of points of the first input that are on the planar surface or within a threshold distance of the planar surface.

[0113]

[0118] Aspect 13: The apparatus of any of aspects 1-12, wherein the first set of points includes one or more points of a first input that are on a ground plane or within a threshold distance of the ground plane.

[0114]

[0119] Aspect 14: The device of any of aspects 1 to 13, wherein to extract a first set of points associated with a ground plane, at least one processor is configured to exclude one or more points of the first input that are on one or more objects.

[0115]

[0120] Example 15: The apparatus of example 14, wherein the one or more objects include a vehicle.

[0116]

[0121] Aspect 16: The device of any of aspects 1-15, wherein at least one processor is configured to extract the first set of points based on at least one of Global Positioning System (GPS) or Inertial Measurement Unit (IMU) data.

[0117]

[0122] Embodiment 17: The device of any of embodiments 1-16, wherein the device is included as part of a vehicle.

[0118]

[0123] Example 18: The device of any of Examples 1-17, wherein at least one processor is further configured to determine a location of the device on a map based on one or more road lanes.

[0119]

[0124] Example 19: The apparatus of any of Examples 1-18, wherein the at least one processor is further configured to perform a route planning operation based on one or more road lanes.

[0120]

[0125] Aspect 20: A method for lane estimation comprising: receiving a first input associated with a three-dimensional (3D) space; extracting from the first input a first set of points associated with a ground plane in the 3D space; mapping each of the first set of points to one of a plurality of regions in a two-dimensional (2D) frame; determining one or more attributes associated with each of the plurality of regions based on one or more of the first set of points mapped to the region; and identifying one or more road lanes based on the one or more attributes.

[0121]

[0126] Example 21: The method of example 20, wherein the first input includes a light-based input.

[0122]

[0127] Aspect 22: The method of aspect 21, wherein the light-based input includes a light detection and ranging (LiDAR) point cloud.

[0123]

[0128] Embodiment 23: Any of the methods of embodiments 20 to 22, wherein the 2D frame includes a bird's-eye view (BEV) frame.

[0124]

[0129] Example 24: The method of any of examples 20-23, wherein identifying one or more road lanes includes analyzing one or more attributes using a machine learning model.

[0125]

[0130] Example 25: The method of example 24, wherein the machine learning model comprises a line-wise classification machine learning model.

[0126]

[0131] Example 26: The method of any of examples 24 or 25, wherein the machine learning model comprises a classification neural network.

[0127]

[0132] Example 27: The method of any of Examples 20-26, wherein identifying one or more road lanes includes identifying a type of each of the one or more road lanes.

[0128]

[0133] Aspect 28: The method of any of aspects 20-27, further including receiving a second input associated with a 3D space, extracting from the second input a second set of points associated with a ground plane in the 3D space, and aggregating the first set of points and the second set of points to generate aggregated points, wherein mapping each of the first set of points includes mapping each of the aggregated points to one of a plurality of regions.

[0129]

[0134] Example 29: The apparatus of example 28, wherein the first input includes a first Light Detection and Ranging (LiDAR) point cloud and the second input includes a second LiDAR point cloud.

[0130]

[0135] Aspect 30: The method of any of aspects 20-29, wherein the one or more attributes include at least one of: a representative height associated with one or more of the first set of points mapped to the region; a maximum height associated with one or more of the first set of points mapped to the region; a representative intensity associated with one or more of the first set of points mapped to the region; a maximum intensity associated with one or more of the first set of points mapped to the region; and a quantity of one or more of the first set of points mapped to the region.

[0131]

[0136] Aspect 31: Any of the methods of aspects 20-30, wherein extracting the first set of points associated with the ground plane includes selecting a subset of points of the first input that are on the planar surface or within a threshold distance of the planar surface.

[0132]

[0137] Aspect 32: The method of any of aspects 20-31, wherein the first set of points includes one or more points of a first input that are on a ground plane or within a threshold distance of the ground plane.

[0133]

[0138] Aspect 33: Any of the methods of aspects 20 to 32, wherein extracting the first set of points associated with the ground plane includes excluding one or more points of the first input that are on one or more objects.

[0134]

[0139] Aspect 34: The method of aspect 33, wherein the one or more objects include a vehicle.

[0135]

[0140] Aspect 35: The method of any of aspects 20 to 34, wherein extracting the first set of points is based on at least one of Global Positioning System (GPS) or Inertial Measurement Unit (IMU) data.

[0136]

[0141] Embodiment 36: The method of any of embodiments 20-35, wherein the method is performed by an apparatus included as part of a vehicle.

[0137]

[0142] Example 37: The method of any of Examples 20-36, further comprising determining a location of the device on a map based on one or more road lanes.

[0138]

[0143] Example 38: The method of any of Examples 20-37, further comprising performing a route planning operation based on one or more road lanes.

[0139]

[0144] Aspect 39. A non-transitory computer-readable medium having stored thereon instructions that, when executed by one or more processors, cause the one or more processors to perform operations according to any of aspects 1-38.

[0140]

[0145] Example 40: An apparatus for image processing, comprising one or more means for performing the operations according to any of Examples 1 to 38.

Claims

1. 1. An apparatus for lane estimation, comprising: at least one memory; at least one processor coupled to the at least one memory, Receiving a first input associated with a three-dimensional (3D) space; Extracting a first set of points from the first input, the first set of points being associated with a ground plane in the 3D space; For a two-dimensional (2D) bird's-eye view frame that is divided into a plurality of regions, mapping each point of the first set of points to a region; for each region of the 2D bird's-eye view frame to which one or more points of the first set of points are mapped, determining one or more attributes associated with the region based on the one or more points of the first set of points mapped to the region; identifying one or more road lanes based on the determined attributes of the regions, wherein identifying the one or more road lanes comprises identifying which regions of the 2D bird's eye view frame include lane markings by analyzing the determined attributes of the regions using a machine learning model. at least one processor configured to: An apparatus comprising:

2. The device of claim 1 , wherein the first input comprises a light-based input.

3. The apparatus of claim 2 , wherein the light-based input comprises a Light Detection and Ranging (LiDAR) point cloud.

4. The apparatus of claim 1 , wherein the machine learning model comprises a row-wise classification machine learning model or a classification neural network.

5. The apparatus of claim 1 , wherein to identify the one or more road lanes, the at least one processor is configured to identify a type of each of the one or more road lanes.

6. the at least one processor: receiving a second input associated with the 3D space; extracting from the second input a second set of points associated with the ground plane in the 3D space; further configured to aggregate the first set of points and the second set of points to generate aggregated points, wherein the at least one processor is configured to map the aggregated points to the region to map the first set of points; 2. The apparatus of claim 1, wherein the first input comprises a first Light Detection and Ranging (LiDAR) point cloud and the second input comprises a second LiDAR point cloud.

7. The one or more attributes determined for each region include: a representative height associated with the one or more of the first set of points mapped to the region; and a maximum height associated with the one or more of the first set of points mapped to the region; and a representative intensity associated with the one or more of the first set of points mapped to the region; and a maximum intensity associated with the one or more of the first set of points mapped to the region; and and the quantity of the one or more of the first set of points mapped to the region.

8. 2. The apparatus of claim 1, wherein to extract the first set of points associated with the ground plane, the at least one processor is configured to select a subset of points of the first input that lie on a flat surface or within a threshold distance of a flat surface.

9. The apparatus of claim 1 , wherein the first set of points includes one or more points of the first input that lie on the ground plane or within a threshold distance of the ground plane.

10. 2. The apparatus of claim 1, wherein to extract the first set of points associated with the ground plane, the at least one processor is configured to exclude one or more points of the first input that lie on one or more objects, and optionally the one or more objects include a vehicle.

11. 10. The apparatus of claim 1, wherein the at least one processor is configured to extract the first set of points based on at least one of Global Positioning System (GPS) or Inertial Measurement Unit (IMU) data.

12. The device of claim 1 , wherein the at least one processor is further configured to determine a location of the device on a map based on the one or more road lanes.

13. The apparatus of claim 1 , wherein the at least one processor is further configured to perform a route planning operation based on the one or more road lanes.

14. A vehicle equipped with the device described in claim 1.

15. A method for lane estimation performed by a computing device, comprising: Receiving a first input associated with a three-dimensional (3D) space; Extracting a first set of points from the first input, the first set of points being associated with a ground plane in the 3D space; For a two-dimensional (2D) bird's-eye view frame that is divided into a plurality of regions, mapping each point of the first set of points to a region; for each region of the 2D bird's-eye view frame to which one or more points of the first set of points are mapped, determining one or more attributes associated with the region based on the one or more of the points of the first set mapped to the region; identifying one or more road lanes based on the determined attributes of the regions, wherein identifying the one or more road lanes comprises identifying which regions of the 2D bird's eye view frame include lane markings based on the determined attributes of the regions by analyzing the determined attributes of the regions using a machine learning model. A method comprising:

Images