Height gap detection based on stereoscopic vision

Abstract

Aspects of the present disclosure relate to stereoscopic vision-based height gap detection. An apparatus includes a processor that acquires first data and second data from different cameras of a vehicle. The processor generates a depth map based on a disparity between the first data and the second data. The processor determines a road segmentation based at least in part on the depth map. The processor determines road surface height information for corresponding portions of the road segmentation. The processor determines a height estimate of a non-road object corresponding to at least a portion of the road segmentation. The processor provides an indication of a height gap estimate for the at least a portion of the road segmentation based on the road surface height information and the height estimate of the non-road object.

Description

[0001] introduction

[0002] Vehicles (including electric vehicles) may include camera systems. For example, a vehicle camera system may include cameras strategically placed on the vehicle to detect obstacles in the vehicle's surroundings. Summary of the invention

[0003] Camera systems can be used to detect obstacles or objects in front of (and / or behind) the vehicle to help prevent collisions or damage to the vehicle. These camera systems can use a combination of sensors and cameras to detect objects in the vehicle's path and provide visual and / or audible warnings to the driver. However, some conventional front height clearance warning systems may have limited accuracy and / or robustness. For example, the accuracy of the front height clearance estimate may be limited by the quality of the sensor data and / or the robustness of the algorithm applied, which may result in incorrect front height clearance values. In addition, some conventional front height clearance warning systems may not be able to detect overhead obstacles that are obscured, partially visible, and / or have complex shapes, and / or some conventional front height clearance warning systems may generate false alarms due to noise, lighting conditions, and / or sensor failures.

[0004] The subject technology provides a front (and / or rear) height clearance detection system based on stereoscopic vision, wherein stereoscopic images can be used for a height clearance detection system on a vehicle. The system uses two or more cameras to capture images (e.g., stereo images) of the same scene from slightly different perspectives, which allows the creation of a three-dimensional (3D) depth map of the environment. In a stereoscopic image system, two cameras are mounted in front of the vehicle (or facing any position selected as a direction suitable for the need for height clearance detection), and their images are processed by, for example, an on-board processing unit to determine whether the vehicle can enter a specific opening along the path of the vehicle. This can be done by analyzing the parallax between the two images, which is the difference in position of corresponding points (or image features) in the two images. The on-board processing unit can then use this information to create a 3D depth map of the environment, which can be used to detect any overhead obstacles and calculate the height clearance of the vehicle and provide visual and / or audible warnings to the driver.

[0005] According to one or more aspects of the subject technology, a method includes: acquiring, by a processor, first data and second data from different cameras of a vehicle, respectively; generating a depth map based on a disparity between the first data and the second data; determining a road segmentation based at least in part on the depth map; determining road surface height information for a corresponding portion of the road segmentation; determining a height estimate of a non-road object corresponding to at least a portion of the road segmentation; and providing an indication of a height gap estimate for the at least a portion of the road segmentation based at least in part on the road surface height information and the height estimate of the non-road object.

[0006] According to one or more aspects of the subject technology, a system is provided, the system comprising a memory and at least one processor, the at least one processor coupled to the memory and configured to acquire a first stereo image and a second stereo image from different cameras of a vehicle, respectively; generate a depth map based on a disparity between the first stereo image and the second stereo image; determine a road segmentation based at least in part on the depth map; generate a road surface height grid, each cell of the road surface height grid indicating road surface height information for a corresponding portion of the road segmentation; detect non-road objects in the depth map; segment the depth map into a plurality of stixel representations; classify the plurality of stixel representations into road stixel representations and non-road stixel representations; identify the non-road object as at least one of the non-road stixel representations in the depth map; determine a height estimate of the at least one of the non-road stixel representations corresponding to at least a portion of the road segmentation; and provide an indication of a height gap estimate for the at least one portion of the road segmentation based at least in part on the road surface height information and the height estimate of the at least one of the non-road stixel representations.

[0007] According to one or more aspects of the subject technology, a vehicle includes a first camera and a second camera, the vehicle also including a processor configured to acquire stereo images from the first camera and the second camera; generate a depth map based on disparity between the stereo images; generate an initial road segmentation by performing a gradient-based graph search on the depth map; generate a stereo depth error model using a region growing process driven by the stereo depth error model to fit a refined secondary road model to the initial road segmentation; determine a road surface height grid, each cell of the road surface height grid indicating road surface height information for a corresponding portion of the road segmentation; segment the depth map into stick pixel representations; and determine a height gap based on the road surface height information using a probabilistic occupancy grid indicating one or more stick pixel representations representing overhead obstacles from the stick pixel representations. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] Certain features of the subject technology are set forth in the appended claims. However, for purposes of explanation, several embodiments of the subject technology are set forth in the following figures.

[0009] Figure 1 A schematic perspective side view of one example implementation of a vehicle having a stereo vision-based camera system according to aspects of the present disclosure is shown.

[0010] Figure 2A schematic perspective side view of another example implementation of a vehicle having a stereo vision-based camera system according to aspects of the present disclosure is shown.

[0011] Figure 3 A schematic perspective side view of yet another example implementation of a vehicle having a stereo vision-based camera system according to aspects of the present disclosure is shown.

[0012] Figure 4 A diagram of an example vehicle including a system for stereo vision-based height gap detection is shown in accordance with one or more implementations of the subject technology.

[0013] Figure 5 A process flow diagram illustrating an example of a stereo vision-based height gap detection system on a vehicle in accordance with one or more implementations of the subject technology is shown.

[0014] Figure 6 A flow chart illustrating an example process for stereo vision-based height gap detection on a vehicle in accordance with one or more implementations of the subject technology.

[0015] Figure 7 An electronic system is shown that can be used to implement one or more implementations of the subject technology. DETAILED DESCRIPTION

[0016] The specific embodiments set forth below are intended to be described as various configurations of the subject technology, and are not intended to represent the only configurations that can practice the subject technology. The accompanying drawings are incorporated herein and constitute a part of the specific embodiments. The specific embodiments include specific details in order to provide a thorough understanding of the subject technology. However, the subject technology is not limited to the specific details set forth herein, and one or more other specific implementations can be used to practice. In one or more specific implementations, well-known structures and components are shown in block diagram form to avoid confusion to the concepts of the subject technology.

[0017] The height clearance detection feature is designed to alert the driver with both visual and auditory signals when the available height clearance in front of their vehicle is below a pre-configured passing height (such as the height of their vehicle). To achieve this goal, the height clearance detection system can warn the driver if the height of the open space in front of the vehicle drops below the height of the vehicle itself. The warning can be triggered before reaching a specified distance from an obstacle that has been detected. By complying with these requirements, the height clearance detection feature enhances the driver's awareness and ensures that there is sufficient clearance when approaching overhead obstacles (such as garages, tunnels, and overhead signs). Several obstacles may need to be overcome to ensure accurate and reliable detection of overhead obstacles, such as sensor limitations, environmental factors, occlusions, image misalignment, and / or noise filtering techniques. For example, appropriately removing signal noise from stereo images can facilitate the implementation of an object height clearance detection system.

[0018] The embodiment of the subject technology provides a height gap detection system that uses stereo image data captured by a vehicle camera to predict the road surface height and the overhead obstacle height, and calculates the height gap between the two predicted heights. The vehicle camera system applies data indicating the height gap to an occupancy grid to inform the user of the vehicle whether there is enough height gap for the vehicle to continue to project along the current route. Because the stereo image data may include noise, an intermediate step may be taken to filter out this noise. Therefore, the embodiment of the subject technology related to the height gap detection based on stereo vision may include: 1) generating a depth map from a stereo image; 2) generating an initial road segmentation by performing a gradient-based graph search on the depth map; 3) fitting a refined secondary road model to the initial road segmentation and determining the road surface height with a region growing process driven by a stereo depth error model; 4) fusing and filtering the road surface height across multiple frames; 5) representing non-road features including features that represent overhead obstacles in the depth map as stick pixels; and / or 6) estimating the front (or rear) height gap with a probability occupancy grid indicating stick pixels representing overhead obstacles.

[0019] Figure 1 is a diagram illustrating an example implementation of a mobile device as described herein. Figure 1In the example of , the vehicle 100 is implemented as a truck (e.g., a pickup truck) with a camera. As shown, the camera may include stereo vision cameras 110, 111. In one or more specific implementations, the cameras 110-111 may be mounted on the front of the vehicle 100, and their images are processed by a computing device (e.g., an electronic control unit 150) to detect and locate obstacles in the path of the vehicle 100. For illustrative purposes, the cameras 110-111 are depicted as being positioned substantially adjacent to each other in a forward direction on the roof of the vehicle 100 above the front windshield window; however, the position, spacing and / or orientation of the cameras 110-111 and / or the distance between the cameras 110-111 may vary depending on the specific implementation without departing from the scope of the present disclosure. For example, the cameras 110-111 may be positioned at other locations of the vehicle 100, such as the front bumper, side mirrors, hood, tailgate, rear bumper, etc.

[0020] In one or more implementations, the vehicle 100 may be an electric vehicle having one or more electric motors that drive the vehicle's wheels 102 using electricity from a battery pack (not shown). In one or more implementations, the vehicle 100 may also or alternatively include one or more chemically powered engines, such as a gasoline powered engine or a fuel cell powered motor. For example, the electric vehicle may be fully electric or partially electric (e.g., a hybrid or plug-in hybrid). In various implementations, the vehicle 100 may be a fully autonomous vehicle capable of traveling on roads without a human operator or driver, a partially autonomous vehicle capable of traveling on some roads without a human operator or driver or capable of traveling on roads under the supervision of a human operator, an unmanned vehicle capable of traveling on roads or other paths without any human occupants, or a human-operated (non-autonomous) vehicle configured for a human operator.

[0021] In one or more implementations, one or more of the cameras 110-111, one or more of the position sensors, and / or other sensors of the vehicle 100 may periodically capture position data to determine a surround view of the vehicle 100. In one or more implementations, one or more of the cameras 110-111 of the vehicle 100 may periodically capture one or more images, and the vehicle 100 may analyze the images (e.g., via object recognition) to determine whether any obstacles are detected approaching the vehicle 100 along the path trajectory. In the event that the position data is captured as one or more images (e.g., by the cameras 110-111), the vehicle 100 may analyze the images to determine whether such obstacles are visible around the vicinity of the vehicle 100 in the images. In the event that the position data is captured as global positioning system (GPS) data (e.g., by Figure 44 ), the vehicle 100 may analyze the location data relative to the known route trajectory of the vehicle 100 to determine if any detected objects are located along the route trajectory of the vehicle 100. In other aspects, the vehicle 100 may analyze the images to determine an omnidirectional visualization of the surroundings of the vehicle 100 and provide a surround view of the vehicle 100. For example, the subject system may process data from all cameras (e.g., cameras 110-111) and create a cohesive image that accurately reflects the position and orientation of the vehicle 100 relative to its surroundings.

[0022] In some implementations, the vehicle 100 may include an electronic control unit (ECU) 150. Because stereoscopic image processing may be computationally intensive, the ECU 150 may include a powerful processing unit, such as a dedicated graphics processing unit (GPU) or a field programmable gate array (FPGA), to perform the necessary image processing in real time.

[0023] The subject system can use a combination of computer vision techniques and advanced algorithms to accurately track the position and orientation of the vehicle 100. The subject system can receive information about the geometry of the vehicle 100 and the surrounding environment as input parameters. The system can also detect obstacles and other vehicles in the environment and display them in a front (or rear) view image, for example, via the display 160 of the vehicle 100.

[0024] In one or more implementations, the vehicle camera system may provide a clear and intuitive user interface for displaying the height gap detection image. This may involve integrating the height gap detection image display with an existing dashboard display or providing a separate display dedicated to the height gap detection image. The display 160 may potentially include additional features such as object detection or distance estimation to further enhance driver awareness and safety. In one or more other implementations, the vehicle camera system may provide other versions of warnings, such as audible warnings, head-up display (HUD) warnings, etc.

[0025] In some implementations, at least one of the cameras 110-111 may be wireless. In this regard, a mechanism for wireless communication between the cameras 110-111 and the ECU 150 may be implemented. This may involve the use of wireless protocols such as Bluetooth or Wi-Fi, or the use of dedicated wireless modules such as ZigBee or LoRa for longer distance communications. In order to ensure reliable transmission of images over wireless communications, data transmission protocols such as the Transmission Control Protocol (TCP) or the User Datagram Protocol (UDP) may be used. These protocols enable error detection and correction, packet retransmission, and other mechanisms to ensure reliable data transmission over unreliable wireless links.

[0026] In one or more specific implementations, stereo calibration can be performed while the vehicle 100 is moving to compute accurate depth perception and 3D reconstruction to support processing of stereo images from multiple vehicle cameras. For example, stereo calibration can include several steps, such as calibration image acquisition, feature extraction from calibration images, correspondence estimation between calibration images, calibration parameter estimation, stereo camera distortion correction, and / or calibration verification. In order to provide the most comprehensive and accurate height gap detection, in addition to the image sensors 110-111, the vehicle camera system can potentially combine data from multiple types of sensors. This can include sensors such as lidar or radar to provide additional depth and distance information, as well as sensors to detect the orientation and movement of the vehicle 100.

[0027] like Figure 1 As shown, stereo vision cameras 110 - 111 are positioned at the front of vehicle 100 in a forward-facing orientation. Figure 1 The example of is merely illustrative, in which the vehicle 100 is implemented as a pickup truck having a cabin at the rear thereof. For example, Figure 2 Another implementation is shown in which the vehicle 100 including the stereo vision cameras 210-211 is implemented as a sport utility vehicle (SUV), such as an electric sport utility vehicle. Figure 2 In the example of FIG. 1 , a vehicle 100 including stereo vision cameras 210 - 211 may include a cargo storage area enclosed within the vehicle 100 (eg, behind a row of seats within the vehicle cabin). Figure 2 As shown, the stereo vision cameras 210, 211 are positioned at the rear of the vehicle 100 in a rearwardly disposed manner. In another example, Figure 3 Yet another implementation is shown in which the vehicle 100 including the stereo vision cameras 110-111 and 310-311 is implemented as a commercial truck, such as an electric delivery truck. Figure 3 In the example of FIG. 1 , a vehicle 100 including stereo vision cameras 110 - 111 and 310 - 311 may include a cargo storage area enclosed within the vehicle 100 (eg, within a vehicle cabin). Figure 3 As shown, stereo vision cameras 110-111 are positioned at the front of vehicle 100 in a forward-facing manner, and stereo vision cameras 310-311 are positioned at the rear of vehicle 100 in a rear-facing manner. In other specific implementations, vehicle 100 may be implemented as another type of electric truck, electric car, electric car, electric motorcycle, electric scooter, electric bicycle, electric passenger vehicle, electric passenger or commercial truck, hybrid vehicle, aircraft, ship and / or any other movable device (e.g., the battery pack is a battery pack or other battery device that powers the propulsion or drive components of the movable device). Figures 1 to 3The stereo vision camera placement shown in is for illustrative purposes, and their placement on vehicle 100 may vary depending on the specific implementation without departing from the scope of the present disclosure.

[0028] Figure 4 A diagram of an example vehicle including a system for stereoscopic vision-based height gap detection according to one or more implementations of the subject technology is shown. However, not all of the depicted components may be used in all embodiments, and one or more implementations may include additional or different components in addition to those shown in the figures. Changes to the arrangement and type of components may be made without departing from the spirit or scope of the claims set forth herein. Additional components, different components, or fewer components may be provided.

[0029] The vehicle 100 may include one or more ECUs 150, one or more cameras 110, one or more geolocation sensors 430, and radio frequency (RF) circuitry 440. The ECU 150 may include a processor 402 and a memory 404. In one or more implementations, the vehicle 100 may include a processor 402 and / or a memory 404 separate from the ECU 150. For example, the vehicle 100 may not include an ECU 150 and may include a processor 402 that is part or all of a separate semiconductor device. In one or more embodiments, the vehicle 100 may include multiple ECUs 150, each ECU 150 controlling a specific function of the vehicle 100.

[0030] The processor 402 may include suitable logic, circuitry, and / or code to enable processing of data and / or control of the operation of the vehicle 100. In this regard, the processor 402 may be enabled to provide control signals to various other components of the vehicle 100. The processor 402 may also control the transmission of data between various parts of the vehicle 100. The processor 402 may also implement an operating system, such as a real-time operating system, or may otherwise execute code to manage the operation of the vehicle 100. The processor 402 may perform all or substantially all of the embodiments of the subject technology.

[0031] The memory 404 may include appropriate logic, circuitry, and / or code that enables storage of various types of information, such as received data, machine learning model data (such as for computer vision and / or other user / object detection algorithms), user authentication data, and / or configuration information. The memory 404 may include, for example, random access memory (RAM), read-only memory (ROM), flash memory, and / or magnetic storage. In one or more embodiments, the memory 404 may store identifiers and / or authentication information of one or more users to determine authorized users and / or authorized authentication devices of the vehicle 100. The memory 404 may also store account information corresponding to authorized users for exchanging information between the vehicle 100 and a remote server. The memory 404 may also store location data, including the geographic location of the charging station and the frequency with which one or more charging stations are used to charge the battery. The memory 404 may also store battery data, including the amount of time that has passed since the battery was last charged.

[0032] Cameras 110-111 may be, or at least partially included in, an onboard camera, a dash cam, an event camera, an infrared camera, a video camera, or any other type of device that captures a digital image representation of a physical environment. Cameras 110-111 may be used to capture images to detect and / or identify objects. For example, an image captured by at least one of cameras 110-111 may be input into a trained obstacle detection model for identifying road or non-road obstacles, which may be compared to a database of road data stored, for example, in memory 404.

[0033] The RF circuit 440 may include appropriate logic, circuitry, and / or code to enable wired or wireless communications, such as local wired or wireless communications within the vehicle 100 and / or between the vehicle 100 and one or more remote servers or devices. The RF circuit 440 may include, for example, one or more of an ultra-wideband (UWB) interface, a Bluetooth communication interface, a near field communication (NFC) interface, a Zigbee communication interface, a near wireless local area network (WLAN) communication interface, a universal serial bus (USB) communication interface, a cellular interface, or any interface generally used to send and / or receive electronic communications. The RF circuit 440 may communicate with other cameras located on the vehicle 100 or otherwise detect other cameras, such as by detecting nearby cameras with UWB ranging.

[0034] In one or more embodiments, one or more of the processor 402, memory 404, cameras 110-111, geolocation sensor 430, RF circuit 440, and / or one or more portions thereof may be implemented in software (e.g., subroutines and code), in hardware (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gating logic, discrete hardware components, or any other suitable device), and / or a combination of both.

[0035] Figure 5 A process flow diagram 500 is shown of an example of a stereo vision-based height gap detection system on a vehicle according to one or more implementations of the subject technology. The process flow diagram 500 includes a disparity map component 520 that receives stereo images (e.g., one or more left images 510 and one or more right images 511) from different cameras and feeds a disparity image 522 to a ground segmentation component 530. The ground segmentation component 530 feeds a road segmentation image 532 to a stick pixel fitting component 540 and a road modeling component 550. In one or more implementations, the ground segmentation component 530 sends its road segmentation image 532 to the stick pixel fitting component 540 and the road modeling component 550 at different times (e.g., first to the stick pixel fitting component 540 and then subsequently to the road modeling component 550, or vice versa). In one or more other implementations, the ground segmentation component 530 sends its road segmentation image 532 to the stick pixel fitting component 540 and the road modeling component 550 at the same time. Figure 5 As shown, the disparity map component 520, the ground segmentation component 530, and the stick pixel fitting component 540 can operate in a 2D disparity space, while the road modeling component 550, the drivable road surface fusion component 560, and the overhead clearance map component 570 can operate in a 2.5D grid space. In some aspects, the 2.5D grid space can refer to a 3D world represented in a 2D grid format. It can be used to capture depth information of a scene or object captured by a stereo camera system. The 2.5D grid space can be a top-down view. Figure 2In some aspects, a 2D disparity space may refer to an image in which intensity values ​​represent the disparity between corresponding points in a left stereo image and a right stereo image. The 2D disparity space may be used to visualize and analyze the estimated disparity or depth information of a scene captured by a stereo camera system.

[0036] The road modeling component 550 may send the secondary road surface model fitting image 552 to the drivable road surface fusion component 560. In one or more implementations, the drivable road surface fusion component 560 may receive a vehicle state, such as vehicle state information 580. In one or more implementations, the drivable road surface fusion component 560 may send a ground elevation map with a road boundary grid image 562 to an overhead clearance map component 570. In one or more implementations, the overhead clearance map component 570 sends an overhead clearance occupancy grid image 572 to the display 160. In one or more implementations, the overhead clearance occupancy grid image 572 may be a static image provided for display via the display 160, or at least partially include the static image. In one or more other implementations, the overhead clearance occupancy grid image 572 may be a video stream provided for display via the display 160, or include one or more images as part of the video stream. In one or more specific implementations, each of the disparity map component 520 , the ground segmentation component 530 , the stick pixel fitting component 540 , the road modeling component 550 , the drivable road surface fusion component 560 , and the head clearance map component 570 is at least partially implemented by the processor 402 in the ECU 150 .

[0037] The vehicle camera system uses at least two cameras (e.g. Figures 1 to 4The camera 110-111 of the vehicle 100 can be used to capture one or more left images 510 and one or more right images 511 of the same scene (e.g., the surrounding environment in front of the vehicle 100) from slightly different perspectives, which allows the creation of a 3D depth map of the surrounding environment. For example, one or more left images 510 can be captured by camera 110, and one or more right images 511 can be captured by camera 111. The disparity map component 520 can analyze the disparity between the one or more left images 510 and the one or more right images 511, which is the position difference of corresponding points in the two images. For example, the disparity value can represent the horizontal shift required to align the corresponding points between the left image 510 and the right image 511. The disparity value can indicate how far the corresponding point is on the left or right in the stereoscopic image pair. A positive disparity value can represent a point that is moved to the right in the left image 510 compared to the right image 511, and a negative disparity value represents a point that is moved to the left. The disparity map component 520 can then use this information to create a disparity image 522 representing a 3D depth map of the surrounding environment. In some implementations, disparity image 522 can be a grayscale image in which each pixel value corresponds to the disparity or depth of a corresponding point in the scene.

[0038] In one or more specific implementations, the disparity map component 520 can perform multiple steps to generate the disparity image 522. For example, the disparity map component 520 can first perform a correction step by transforming the stereo images (e.g., one or more left images 510 and one or more right images 511) so that corresponding points in the two stereo images are on the same horizontal scan line. For example, the correction transformation can be applied to each stereo image (e.g., 510-611) of the distorted stereo images so that corresponding points are on the same horizontal scan line. The correction transformation can be based on the stereo camera (e.g., Figure 1 The correction transform is calculated by using intrinsic and extrinsic parameters of cameras 110-111 of the correction image. In some aspects, a horizontal scan line may refer to a row of pixels in the corrected stereo image corresponding to a specific horizontal position in the original stereo image (e.g., 510-611). Once the stereo images have been corrected, the disparity map component 520 may perform stereo matching by comparing the pixel intensities along corresponding horizontal scan lines in the two corrected images. In one or more specific implementations, the disparity map component 520 may identify corresponding features between stereo matched images using one or more feature detection and matching algorithms, such as a scale-invariant feature transform (SIFT) algorithm or a speeded up robust feature (SURF) algorithm. Once corresponding features in the two images are identified, the disparity map component 520 may calculate the disparity between the images.

[0039] In one or more implementations, the ground segmentation component 530 performs an initial gradient-based ground segmentation of the disparity image 522 and generates a road segmentation image 532. In one or more implementations, the ground segmentation component 530 may estimate the depth value of the pixel based on the disparity value calculated in the disparity image 522. In one or more implementations, the ground segmentation component 530 may perform multiple steps to generate the road segmentation image 532. For example, the ground segmentation component 530 may generate a gradient map containing gradient values ​​of the depth value of the pixel from the disparity image 522 by calculating the magnitude and direction of the gradient at each pixel. In some aspects, the gradient value may refer to the vertical gradient and / or horizontal gradient of the neighboring pixels in the disparity image 522. In one or more implementations, the ground segmentation component 530 may perform local smoothing on the gradient value to remove sharp noise points.

[0040] The ground segmentation component 530 may identify pixels or regions corresponding to the road surface in the disparity image 522 to separate road pixels from non-road pixels in the disparity image 522. In one or more other specific implementations, the ground segmentation component 530 may identify road pixels and non-road pixels in the gradient map. The ground segmentation component 530 may use color segmentation, texture analysis, edge detection and / or machine learning algorithms to identify road pixels and non-road pixels in the disparity image 522. In one or more embodiments, the ground segmentation component 530 may generate a binary mask or a set of marked areas that indicate which pixels in the disparity image 522 belong to the road surface and which pixels belong to other objects or backgrounds. For example, the ground segmentation component 530 generates a road segmentation image 532 indicating road surface pixels. In one or more specific implementations, the road segmentation image 532 includes a mask covering that identifies the segmented road surface in the road segmentation image 532.

[0041] In one or more implementations, the ground segmentation component 530 may generate a gradient-based graph from the gradient map by starting a region growing process with an initial seed pixel or region of interest, and then growing the region by iteratively adding neighboring pixels or regions that meet a certain criterion (such as the gradient difference between adjacent pixels in the gradient map). In one or more implementations, the ground segmentation component 530 uses a predefined region on the gradient map to select seed ground points. In some aspects, the ground segmentation component 530 may generate a road segmentation image 532 by assigning a binary value to each pixel in the gradient-based graph based on whether the pixel belongs to a road (by comparing the pixel value to a predefined gradient change threshold or using other segmentation algorithms). In one or more implementations, the ground segmentation component 530 determines whether the gradient value in the predefined region is less than a first gradient change threshold (which may correspond to an angle value in degrees). For example, if the ground segmentation component 530 determines that the gradient value in the predefined region is less than the first gradient change threshold, the ground segmentation component 530 may select the corresponding pixel in the predefined region as a seed ground point. Subsequently, the ground segmentation component 530 may propagate these seed points to the entire disparity image 522 using a gradient-based graph search, such as a depth-first search (DFS) graph search or a breadth-first search (BFS) graph search: if the gradient difference is less than a second gradient change threshold, then adjacent neighboring pixels may be added as seed points. In one or more specific implementations, the second gradient change threshold corresponds to an angle value in degrees that is less than the first gradient change threshold. The DFS graph search may be an example of a graph search algorithm that is configured to search all neighboring vertices along a branch of a current vertex and backtrack to a previous vertex to resume the search of unsearched neighboring vertices along a branch of the previous vertex for all vertices in the gradient-based graph. The BFS graph search may be another example of a graph search algorithm that is configured to search each vertex in a gradient-based graph by searching all vertices in one level before traversing to the next level for each level of the gradient-based graph.

[0042] In one or more specific implementations, the road modeling component 550 may model the road surface as a quadratic surface. For example, the road modeling component 550 may assume that the road surface can be approximated by a quadratic function. In one or more specific implementations, the road modeling component 550 may generate a quadratic road surface model fitting image 552 by fitting a quadratic function to an area corresponding to the segmented road surface in the road segmentation image 532 using a least squares optimization method. In one or more specific implementations, the quadratic function may be defined as Y=a·X+b·X 2 +c·Z+d·Z 2+e, where X and Y are the coordinates of the 3D point in 3D camera coordinates (the origin at the center of the camera), and a, b, c, d, and e are the parameters of the quadratic function. In one or more specific implementations, the values ​​of the parameters a, b, c, d, and e can be determined by minimizing a least squares objective function as defined below:

[0043] In one or more implementations, the road modeling component 550 can refine the quadratic function using a region growing process and a stereo depth error model associated with the disparity image 522. In one or more implementations, the stereo depth error model is generated from a stereo camera specification and calibration. In some aspects, the stereo camera specification can define the geometry and characteristics of a stereo camera setup and can include information such as a stereo camera (e.g., Figure 1 110, 111), the focal length of the lens, the image sensor size, and the resolution of the image captured by each of the stereo cameras. Stereo camera calibration may be performed to determine the intrinsic and extrinsic parameters of each stereo camera in the stereo camera pair. In some aspects, each stereo camera may capture images of a calibration pattern with a known 3D geometry from different viewing angles and estimate the stereo camera parameters using computer vision techniques (or machine learning techniques). In this regard, for example, once the stereo cameras 110, 111 are calibrated, the stereo cameras may be used to generate disparity images 522. In one or more specific implementations, the road modeling component 550 may use a tunable disparity uncertainty value to calculate stereo depth error data. In one or more specific implementations, the road modeling component 550 may obtain a stereo depth error model based on the stereo depth error data.

[0044] The stereo depth error model can characterize the error in the depth values ​​of the disparity image 522 by capturing the statistical properties of the depth error or uncertainty present in the disparity image 522 and providing a mathematical representation of these properties. The road modeling component 550 can use the stereo depth error model to determine inliers and / or outliers in the road segmentation image 532. In some aspects, inliers can refer to pixels or regions that are consistent with the modeled depth information of the road surface and have low stereo depth errors according to the stereo depth error model, while outliers can refer to pixels or regions that deviate significantly from the expected depth value and have higher stereo depth errors according to the stereo depth error model.

[0045] The region growing process may be driven by a stereo depth error model to refine the initial depth estimate and improve the accuracy of the secondary road surface model fit image 552. For example, the road modeling component 550 may first select a pixel in the road segmentation image 532 covered with a road segmentation mask as an initial seed point corresponding to a portion of the road surface in the scene. The road modeling component 550 may perform a region growing process on the road segmentation image 532 to iteratively expand the initial seed point into the growing region by adding neighboring pixels around the initial seed point. The road modeling component 550 may use the stereo depth error model to refine one or more depth values ​​in the growing region, and thereafter fit the secondary model to the growing region with the refined one or more depth values ​​to generate a refined secondary road model. In one or more specific implementations, the refined secondary road model may be defined as Y=aX 2 +bZ 2 +cXZ+dX+eZ+f, where Y is the estimated road surface height, X and Z are the coordinates of the 3D point in 3D camera coordinates (origin at the center of the camera), and a, b, c, d, e, and f are coefficients of the model. In one or more specific implementations, the XZ term is optional.

[0046] In one or more other specific implementations, the ground segmentation component 530 performs an intermediate step between the initial gradient-based ground segmentation and the quadratic model fitting by analyzing the gradient-based graph and identifying the clusters of nodes corresponding to the road or driving surface using techniques such as clustering (e.g., K-means, mean shift or spectral clustering) or segmentation algorithms. In some aspects, each node in the gradient-based graph can be assigned to the cluster to which it belongs based on its proximity to the cluster center or other predefined criteria. The ground segmentation component 530 can then filter the identified clusters to remove noise or non-road nodes, such as nodes corresponding to obstacles or other objects in the scene, using techniques such as morphological operations, regional growth or machine learning algorithms. For example, the ground segmentation component 530 can utilize regional growth by grouping adjacent pixels or regions with similar characteristics into larger regions or objects. In this regard, the road modeling component 550 can model the road surface as a quadratic surface based on the filtered clusters from the ground segmentation component 530.

[0047] The rod pixel fitting component 540 may perform a rod pixel calculation to generate a rod pixel representation overlay 542 containing a rod pixel representation representing non-road data in the disparity image 522. In one or more specific implementations, the rod pixel fitting component 540 may segment the disparity image 522 into vertical columns (or referred to as rod pixel columns) each containing one or more rod pixels, wherein each rod pixel may include a group of pixels with similar disparity values, but the rod pixels within the rod pixel column may include different disparity values. For each rod pixel column, the rod pixel group may be assumed to be a vertical plane and may be evaluated based on different criteria (such as geometric consistency, occlusion relationship with the road surface and other rod pixels, and / or appearance characteristics). Rod pixels within each rod pixel column that do not belong to the road surface may be classified as non-road rod pixels. In one or more specific implementations, rod pixels connected to the ground may be referred to as ground rod pixels. Ground rod pixels may represent non-road rod pixels. In one or more other specific implementations, rod pixels that can be connected to the ground via other rod pixels connected to the ground may be referred to as ground rod pixels. In one or more other implementations, rod pixels with gaps between the rod pixels and the ground can be classified as overhead rod pixels. In one or more other implementations, rod pixels within each rod pixel column belonging to the road surface can be classified as road rod pixels. Non-road stick pixels can be represented as non-road stick pixel representations having a rectangular shape, whose size corresponds to their estimated height and position in the scene. In one or more specific implementations, the secondary road surface model fitting image 552 includes stick pixel representations from the stick pixel representation overlay 542 provided by the stick pixel fitting component 540.

[0048] In one or more specific implementations, the drivable road surface fusion component 560 may perform road surface height filtering and fusion. In some aspects, the drivable road surface fusion component 560 may map the drivable road surface height estimate by averaging the height estimate values ​​between consecutive frames. The drivable road surface fusion component 560 may receive vehicle state information 580 indicating the state, condition and / or parameters (including vehicle height and / or stereo to vehicle calibration) related to the vehicle 100, so that the drivable road surface fusion component 560 maps the drivable road surface height estimate to the segmented road surface according to the vehicle state information 580. In one or more specific implementations, the drivable road surface fusion component 560 may determine the road surface height information of the corresponding portion of the road segmentation. In one or more specific implementations, the drivable road surface fusion component 560 may determine a road surface height grid for the road segmentation image 532, wherein each cell of the road surface height grid provides a road surface height value. In some aspects, the road surface height grid is mapped to the segmented road surface. The road surface height grid may be constructed by calculating the road surface height grid using the refined secondary road model (which is fitted to the initial road segmentation) at all points in the scene corresponding to the road segmentation image 532. For example, to calculate the road surface height value for a given grid cell, the drivable road surface fusion component 560 may substitute the coordinates of the point (X, Z) into the refined secondary road model formula, and the resulting value of Y represents the estimated height of the road surface at that point of the corresponding grid cell. In some aspects, the road surface height may refer to the height of the drivable road surface at one or more cells in the drivable road surface occupancy grid.

[0049] In one or more other specific implementations, the drivable road surface fusion component 560 may perform drivable road surface occupancy grid generation and filtering. For example, the drivable road surface fusion component 560 may map the drivable road surface occupancy grid to the secondary road surface model fitting image 552 through multi-frame probability updates, resulting in the generation of a ground elevation map with a road boundary grid image 562. The drivable road surface occupancy grid may remove false alarm overhead obstacles in non-drivable areas to improve the clarity of the grid. In one or more specific implementations, the drivable road surface occupancy grid may have ground dynamic obstacles removed by multi-frame probability updates to help distinguish ground static obstacles from ground dynamic obstacles. In some aspects, ground static obstacles may refer to buildings, trees, curbs, parked vehicles, etc., while ground dynamic obstacles may refer to moving vehicles, pedestrians, etc. The drivable road surface occupancy grid may be configured to identify overhead obstacles within the segmented road surface. In some aspects, overhead obstacles may refer to overhead clearance strips, traffic signs, etc.

[0050] In one or more specific implementations, the overhead clearance map component 570 may perform ground obstacle rejection, overhead clearance calculation, and / or overhead clearance occupancy grid generation and fusion. In some aspects, the overhead clearance map component 570 may select non-road stick pixel representations belonging to overhead obstacles by analyzing the geometry of the overhead obstacles and the position relative to the road surface based on the road surface height. In some aspects, the overhead clearance map component 570 may estimate the height of the obstacle by measuring the distance of the obstacle from the road surface height at the location of the obstacle. In one or more specific implementations, the overhead clearance map component 570 may select non-road stick pixel representations that have an estimated height value greater than a height threshold and are stick pixels that cannot be connected to the ground via other stick pixels (classified as overhead obstacles). The overhead clearance map component 570 may reject non-road stick pixel representations that have an estimated height value less than a height threshold and / or are stick pixels that can be connected to the ground via other stick pixels (classified as ground obstacles). In one or more specific implementations, the height threshold may correspond to a vehicle height parameter value. In one or more other implementations, the height threshold may be reconfigurable such that the height threshold may be updated based on dynamic measurements of vehicle height. In one or more other implementations, the vehicle height parameter value may be adjusted to primarily account for towed object heights (e.g., trailer height) that exceed the vehicle height or the overall height of a vehicle with elevated cargo.

[0051] In one or more specific implementations, the overhead clearance map component 570 can estimate the height clearance ahead (or any other location on the vehicle) with a probabilistic occupancy grid using the non-road stick pixel representation corresponding to the selected overhead obstacle. The probabilistic occupancy grid can be a 2.5D top-down view representation of a scene divided into cells, where each cell represents a small area in the scene. The probabilistic occupancy grid can be used to estimate the probability of an obstacle being present in each cell. To estimate the height clearance, the stick pixel representations belonging to the overhead obstacles are projected onto the probabilistic occupancy grid, and their positions and sizes are used to update the occupancy probabilities of the corresponding cells in the probabilistic occupancy grid. The overhead clearance map component 570 can estimate the height clearance of each cell by measuring the height of the obstacle occupying it and subtracting it from the height of the grid cell. In some aspects, the height of the grid cell may correspond to the road surface height. The overhead clearance occupancy grid image 572 may include the resulting height clearance values ​​visualized as a color-coded map 574, where different colors represent different height clearance ranges, and a representation of the position of the vehicle 100 relative to the location of the overhead obstacle. The overhead clearance map component 570 can provide an overhead clearance occupancy grid image 572 for display via the display 160. In one or more other implementations, the overhead clearance map component 570 can provide a 3D road surface image 590 incorporating a color-coded image 574 indicating height clearance values ​​for display via the display 160.

[0052] In one or more specific implementations, one or more of the disparity map component 520, the ground segmentation component 530, the stick pixel fitting component 540, the road modeling component 550, the drivable road surface fusion component 560, the head clearance map component 570, and / or one or more portions thereof may be implemented in software (e.g., subroutines and code), in hardware (e.g., an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a programmable logic device (PLD), a controller, a state machine, gated logic, discrete hardware components, or any other suitable device), and / or a combination of both.

[0053] Figure 6 A flowchart 600 is shown of an example process for stereoscopic vision-based height gap detection on a vehicle according to one or more implementations of the subject technology. Figures 1 to 5 The process 600 is described with reference to the vehicle 100 and / or its various components. However, the process 600 is not limited to Figure 1The vehicle 100 of the present invention is shown, and one or more steps (or operations) of process 600 may be performed by one or more other structural components of the vehicle 100 and / or other suitable mobile devices, equipment or systems. Further for the purpose of explanation, some steps of process 600 are described herein as occurring continuously or linearly. However, multiple steps of process 600 may occur in parallel. In addition, the steps of process 600 do not need to be performed in the order shown, and / or one or more steps of process 600 do not need to be performed and / or may be replaced by other operations.

[0054] At step 602, the processor 402 may acquire first data and second data respectively from different cameras of the vehicle 100. For example, the first data may be acquired from a first camera (e.g., implemented as camera 110), and the second data may be acquired from a second camera (e.g., implemented as camera 111). In one or more specific implementations, the first data and the second data include stereoscopic images.

[0055] At step 604, the processor 402 may generate a depth map based on the disparity between the first data and the second data, as shown in FIG. Figure 5 In one or more implementations, the generation of the depth map may include correcting the first data and the second data to a common horizontal scan line, and determining corresponding feature points between the first data and the second data along the common horizontal scan line. The generation of the depth map may also include calculating the disparity values ​​between corresponding feature points in the first data and the second data. In some aspects, each pixel value in the depth map corresponds to the disparity value of the corresponding point in the scene of the depth map.

[0056] At step 606, the processor 402 may determine the road segmentation based at least in part on the depth map, as shown in FIG. Figure 5As described by the ground segmentation component 530 in . In one or more specific implementations, determining the road segmentation may include generating a gradient map by calculating the magnitude and direction of the gradient at each pixel of the depth map, and generating a gradient-based graph from the gradient map by initiating a first region growing process with a predefined region in the gradient map. Determining the road segmentation may also include generating a road mask from the gradient-based graph, the road mask indicating the road segmentation. In one or more specific implementations, generating the gradient-based graph may include determining whether the gradient difference between a pixel in a predefined region of the depth map and a neighboring pixel in the predefined region is less than a first gradient change threshold. In one or more specific implementations, the pixel may be selected as a seed ground point in the predefined region based on the gradient difference between the pixels in the predefined region of the depth map being less than the first gradient change threshold. Generating the gradient-based graph may also include determining whether the gradient difference between a pixel outside the predefined region and a neighboring pixel outside the predefined region is less than a second gradient change threshold, wherein the second gradient change threshold corresponds to an angle value in degrees less than the first gradient change threshold. Generating a gradient-based graph may also include adjusting the predefined area by adding seed points outside the predefined area using a graph search algorithm based on a gradient difference between a pixel outside the predefined area and a neighboring pixel outside the predefined area being less than a second gradient change threshold that is different from the first gradient change threshold.

[0057] In one or more specific implementations, process 600 may include one or more steps for performing fitting of a secondary model to the road segmentation to generate a refined secondary road model using a region growing process and a stereo depth error model, as described in reference to Figure 5 As described by the road modeling component 550 in . In some aspects, the stereo depth error model characterizes the error in the depth values ​​of the depth map. In one or more specific implementations, fitting the quadratic model to the road segmentation may include selecting a pixel in a road mask as an initial seed point corresponding to a portion of a road surface in the scene, and performing a region growing process on the road mask to iteratively expand the initial seed point into a growing region by adding neighboring pixels around the initial seed point. Fitting the quadratic model to the road segmentation may also include: using the stereo depth error model to refine one or more depth values ​​in the growing region, and fitting the quadratic model to the growing region with the refined one or more depth values ​​to generate a refined secondary road model.

[0058] At step 608, the processor 402 may determine the road surface height information of the corresponding portion of the road segmentation, such as reference Figure 54. For example, the processor 402 may generate a road surface height grid, wherein each cell of the road surface height grid indicates road surface height information of a corresponding portion of the road segmentation. In one or more specific implementations, determining the road surface height information may include calculating a road surface height value using a refined secondary road model at one or more points in the scene corresponding to at least a portion of the road segmentation.

[0059] At step 610, the processor 402 may determine a height estimate of a non-road object corresponding to at least a portion of the road segmentation, such as by referring to Figure 5 In one or more embodiments, process 600 may include one or more steps for performing detection of non-road objects in a depth map and segmenting the depth map into stick-like pixel representations, as described in reference to FIG. Figure 5 In one or more implementations, process 600 may further include classifying the stick pixel representation into a road stick pixel representation and a non-road stick pixel representation, and identifying a non-road object as at least one of the non-road stick pixel representations in the depth map. For example, a non-road object may be composed of a plurality of stick pixels.

[0060] In one or more specific implementations, process 600 may include one or more steps for performing determination of a location in a scene of a depth map corresponding to at least one of the non-road stick pixel representations in the depth map. Process 600 may also include determining a height estimate of at least one of the non-road stick pixel representations at a location in the scene corresponding to the location of the non-road stick pixel representation. Process 600 may also include determining whether the height estimate of at least one of the non-road stick pixel representations exceeds a height threshold relative to the height of the road surface. Process 600 may also include determining whether at least one of the non-road stick pixel representations is connected to the ground via other non-road stick pixel representations. In some aspects, when the height estimate of one of the non-road stick pixel representations exceeds the height threshold and at least one of the non-road stick pixel representations is not connected to the ground via other non-road stick pixel representations, one of the non-road stick pixel representations is identified as an overhead obstacle.

[0061] At 612, processor 402 may provide an indication of a forward (or backward) height gap estimate for at least a portion of the road segmentation based at least in part on the road surface height information and the height estimate of the non-road object, as shown in reference to Figure 5. In one or more implementations, providing an indication of a front (or rear) height gap estimate may include: projecting a height estimate of one of the non-road stick pixel representations into one or more cells of a probabilistic occupancy grid, wherein the probabilistic occupancy grid includes cells, each cell indicating a probability that an overhead obstacle is present in the cell; and determining the height gap estimate by subtracting the height estimate of one of the non-road stick pixel representations in the cell from the height of the cell of the probabilistic occupancy grid. In some aspects, the cell of the probabilistic occupancy grid indicates a value representing the height gap estimate. In one or more implementations, the indication of the height gap estimate is provided for display, such as via a display 160 of the vehicle 100.

[0062] Figure 7 An example electronic system 700 is shown that can be used to implement various aspects of the present disclosure. The electronic system 700 can be used to provide reference Figures 1 to 6 Any electronic device and / or part thereof that may perform the features and processes described herein includes, but is not limited to, vehicles, computers, and servers. Electronic system 700 may include various types of computer-readable media and interfaces for various other types of computer-readable media. Electronic system 700 includes persistent storage device 702, system memory 704 (and / or buffer), input device interface 706, output device interface 708, sensor 710, ROM 712, processing unit 714, network interface 716, bus 718, and / or subsets and variations thereof.

[0063] The bus 718 collectively represents a plurality of internal devices and / or components (such as those described above with respect to FIG. Figure 4 The bus 718 communicatively connects the one or more processing units 714 to the ROM 712, the system memory 704, and the persistent storage device 702. From these various memory units, the one or more processing units 714 retrieve instructions to be executed and data to be processed in order to perform the processes disclosed in the present subject matter. In different implementations, the one or more processing units 714 may be a single processor or a multi-core processor. In one or more implementations, one or more of the processing units 714 may be included on the ECU 150, such as in the form of processor 402.

[0064] ROM 712 stores static data and instructions required by one or more processing units 714 and other modules of electronic system 700. On the other hand, persistent storage device 702 can be a read-write memory device. Persistent storage device 702 can be a non-volatile memory unit that stores instructions and data even when electronic system 700 is turned off. In one or more specific implementations, a mass storage device (such as a magnetic disk or optical disk and its corresponding disk drive) can be used as persistent storage device 702.

[0065] In one or more specific implementations, a removable storage device (such as a flash drive and its corresponding solid-state device) can be used as the persistent storage device 702. Similar to the persistent storage device 702, the system memory 704 can be a read-write memory device. However, unlike the persistent storage device 702, the system memory 704 can be a volatile read-write memory, such as RAM. The system memory 704 can store any instructions and data that one or more processing units 714 may need at run time. In one or more specific implementations, the processes disclosed in the present subject matter are stored in the system memory 704, the persistent storage device 702, and / or the ROM 712. From these various memory units, the one or more processing units 714 retrieve instructions to be executed and data to be processed in order to perform the processes of one or more specific implementations.

[0066] The persistent storage device 702 and / or the system memory 704 may include one or more machine learning models. Machine learning models (such as those described herein) are often used to form predictions, solve problems, identify objects in image data, and the like. For example, the machine learning models described herein may be used to predict whether an authorized user is approaching a vehicle and intends to open a charging port closure. Various specific implementations of machine learning models are possible. For example, the machine learning model may be a deep learning network, a transformer-based model (or other attention-based model), a multilayer perceptron or other feedforward network, a neural network, and the like. In various examples, the machine learning model may be more adaptive because the machine learning model may be improved over time by retraining the model as additional data becomes available.

[0067] The bus 718 is also connected to an input device interface 706 and an output device interface 708. The input device interface 706 enables a user to communicate information and select commands to the electronic system 700. The input devices that can be used with the input device interface 706 may include, for example, an alphanumeric keyboard, a touch screen, and a pointing device. The output device interface 708 may enable the electronic system 700 to communicate information to the user. For example, the output device interface 708 may provide a display of an image generated by the electronic system 700. The output devices that can be used with the output device interface 708 may include, for example, a printer and a display device, such as a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, a flexible display, a flat panel display, a solid state display, a projector, or any other device for outputting information.

[0068] One or more implementations may include a device that acts as both an input and output device, such as a touch screen. In these implementations, the feedback provided to the user may be any form of sensory feedback, such as visual feedback, auditory feedback, or tactile feedback; the input from the user may be received in any form, including acoustic, voice, or tactile input.

[0069] Bus 718 is also connected to sensor 710. Sensor 710 may include a position sensor that can be used to determine the device location based on positioning technology. For example, the position sensor can provide one or more of a global navigation satellite system (GNSS) positioning, a wireless access point positioning, a cellular phone signal positioning, a Bluetooth signal positioning, an image recognition positioning, and / or an inertial navigation system (e.g., via a motion sensor such as an accelerometer and / or a gyroscope). In one or more specific implementations, sensor 710 can be used to detect the movement, travel, and orientation of electronic system 700. For example, the sensor may include an accelerometer, a rate gyroscope, and / or other motion-based sensors. Sensor 710 may include one or more biometric sensors and / or cameras for authenticating users.

[0070] The bus 718 also couples the electronic system 700 to one or more networks and / or one or more network nodes via one or more network interfaces 716. In this manner, the electronic system 700 may be part of a computer network, such as a local area network or a wide area network. Any or all components of the electronic system 700 may be used in conjunction with the subject disclosure.

[0071] Implementations within the scope of the present disclosure may be implemented in part or in whole using a tangible computer-readable storage medium (or multiple tangible computer-readable storage media of one or more types) encoding one or more instructions. Tangible computer-readable storage media may also be non-transitory in nature.

[0072] Computer-readable storage media can be any storage media that can be read, written, or otherwise accessed by a general or special computing device, including any processing electronics and / or processing circuitry capable of executing instructions. For example, but not limited to, computer-readable media can include any volatile semiconductor memory, such as RAM, DRAM, SRAM, T-RAM, Z-RAM, and TTRAM. Computer-readable media can also include any non-volatile memory, such as ROM, PROM, EPROM, EEPROM, NVRAM, flash memory, nvSRAM, FeRAM, FeTRAM, MRAM, PRAM, CBRAM, SONOS, RRAM, NRAM, racetrack memory, FJG, and Millipede memory.

[0073] In addition, the computer-readable storage medium may include any non-semiconductor memory, such as optical disk storage, magnetic disk storage, magnetic tape, other magnetic storage devices, or any other medium capable of storing one or more instructions. In one or more specific implementations, the tangible computer-readable storage medium may be directly coupled to the computing device, while in other specific implementations, the tangible computer-readable storage medium may be indirectly coupled to the computing device, for example, via one or more wired connections, one or more wireless connections, or any combination thereof.

[0074] Instructions can be directly executable or can be used to develop executable instructions. For example, instructions can be implemented as executable or non-executable machine code, or can be implemented as instructions of high-level languages ​​that can be compiled to produce executable or non-executable machine code. In addition, instructions can also be implemented as data or can include data. Computer executable instructions can also be organized in any format, including routines, subroutines, programs, data structures, objects, modules, applications, small applications, functions, etc. As recognized by those skilled in the art, the details including but not limited to the quantity, structure, sequence and organization of instructions can change significantly without changing the underlying logic, function, processing and output.

[0075] Although the above discussion mainly involves microprocessors or multi-core processors that execute software, one or more specific implementations are performed by one or more integrated circuits such as ASICs or FPGAs. In one or more specific implementations, such integrated circuits execute instructions stored on the circuit itself.

[0076] Unless otherwise specified, an element mentioned in the singular is not intended to represent one and only one, but one or more. For example, "a" module may refer to one or more modules. In the absence of further constraints, an element beginning with "a", "an", "the" or "said" does not exclude the presence of other identical elements.

[0077] Headings and subheadings, if any, are used for convenience only and do not limit the present disclosure. The word exemplary is used to mean serving as an example or illustration. To the extent that the terms "including" or "having" and the like are used, such terms are intended to be inclusive in a manner similar to the term "comprising" as understood when "comprising" is used as a transitional word in a claim. Relational terms such as first and second and the like may be used to distinguish one entity or action from another without necessarily requiring or implying any actual such relationship or order between such entities or actions.

[0078] Phrases such as an aspect, this aspect, another aspect, some aspects, one or more aspects, a specific implementation, this specific implementation, another specific implementation, some specific implementations, one or more specific implementations, an embodiment, this embodiment, another embodiment, some embodiments, one or more embodiments, a configuration, this configuration, another configuration, some configurations, one or more configurations, subject technology, this disclosure, the present disclosure, other variations thereof, etc. are for convenience and do not imply that the disclosures associated with such phrases are necessary for the subject technology, or that such disclosures are applicable to all configurations of the subject technology. The disclosures associated with such phrases may be applicable to all configurations or one or more configurations. The disclosures associated with such phrases may provide one or more examples. Phrases such as an aspect or some aspects may refer to one or more aspects, and vice versa, and this is similarly applicable to other aforementioned phrases.

[0079] The phrase "at least one" preceding a list of items, with the terms "and" or "or" used to separate any of those items, modifies the list as a whole, rather than as each constituent item of the list. The phrase "at least one" does not require selection of at least one item; rather, the phrase allows for a meaning that includes at least one of any of those items, and / or at least one of any combination of those items, and / or at least one of each of those items. As an example, each of the phrases "at least one of A, B, and C" or "at least one of A, B, or C" means only A, only B, or only C; any combination of A, B, and C; and / or at least one of each of A, B, and C.

[0080] It should be understood that the specific order or level of the disclosed steps, operations or processes is an illustration of an exemplary method. Unless otherwise clearly stated, it should be understood that the specific order or level of steps, operations or processes can be performed in different orders. Some of the steps, operations or processes can be performed simultaneously. The attached method claims (if any) present the elements of various steps, operations or processes in a sample order, and are not meant to be limited to the specific order or level presented. These can be performed serially, linearly, in parallel or in different orders. It should be understood that the instructions, operations and systems described can usually be integrated together in a single software / hardware product or encapsulated in multiple software / hardware products.

[0081] Terms such as top, bottom, front, back, side, horizontal, vertical, etc. refer to an arbitrary reference frame, not the ordinary gravitational reference frame. Thus, such terms may extend upward, downward, diagonally, or horizontally in a gravitational reference frame.

[0082] The present disclosure is provided to enable any person skilled in the art to practice the various aspects described herein. In some instances, well-known structures and components are shown in block diagram form to avoid confusion with the various concepts of the subject technology. The present disclosure provides various examples of the subject technology, and the subject technology is not limited to these examples. Those skilled in the art will readily appreciate various modifications to these aspects, and the principles described herein may be applied to other aspects.

[0083] All structural and functional equivalents of the various aspects of the elements described throughout this disclosure are known or will become known to those of ordinary skill in the art, and these equivalents are expressly incorporated herein by reference and are intended to be included in the claims. In addition, nothing disclosed herein is intended to serve the public, regardless of whether such disclosure is explicitly stated in the claims. Any claim element should not be interpreted under the provisions of 35 U.S.C. § 112 (f) unless the element is explicitly stated using the phrase "means for..." or, in the case of a method claim, the phrase "step for..." is used to state the element.

[0084] Those skilled in the art will appreciate that the various illustrative frames, modules, elements, parts, methods and algorithms described herein can be implemented as hardware, electronic hardware, computer software or a combination thereof. In order to illustrate this interchangeability of hardware and software, various illustrative frames, modules, elements, parts, methods and algorithms have been generally described above in terms of their functionality. Whether this functionality is implemented as hardware or software depends on the specific application and the design constraints imposed on the entire system. Those skilled in the art can implement the described functionality in different ways for each specific application. Various parts and frames can be arranged differently (e.g., arranged in different orders or divided in different ways), all of which do not depart from the scope of this subject technology.

[0085] The names, figure descriptions, abstracts, and drawings are hereby incorporated into this disclosure and are provided as illustrative examples of the disclosure rather than as limiting descriptions. This document is submitted with the understanding that they will not be used to limit the scope or meaning of the claims. In addition, in the detailed description, it can be seen that for the purpose of simplifying the present disclosure, the description provides illustrative examples and various features are grouped together in various specific implementations. The method of the present disclosure should not be interpreted as reflecting an intention that the claimed subject matter requires more features than those expressly stated in each claim. On the contrary, as reflected in the claims, the inventive subject matter lies in less than all the features of a single disclosed configuration or operation. The claims are hereby incorporated into the detailed description, with each claim independently serving as a separately claimed subject matter.

[0086] The claims are not intended to be limited to the aspects described herein, but should be accorded the full scope consistent with the language of the claims and encompassing all legal equivalents. Nevertheless, none of the claims is intended to encompass subject matter that fails to satisfy the requirements of applicable patent law, nor should it be interpreted in such a manner.

Claims

1. A method, comprising: Acquire, by a processor, first data and second data from different cameras of the vehicle respectively; generating a depth map based on a disparity between the first data and the second data; determining a road segmentation based at least in part on the depth map; Determining road surface height information of a corresponding portion of the road segmentation; determining a height estimate of a non-road object corresponding to at least a portion of the road segmentation; as well as An indication of a height gap estimate for the at least a portion of the road segmentation is provided based at least in part on the road surface height information and the height estimate of the non-road object. The method according to claim 1 , wherein the first data and the second data include stereoscopic images.

3. The method according to claim 1, wherein generating the depth map comprises: Correcting the first data and the second data to a common horizontal scan line; determining corresponding feature points between the first data and the second data along the common horizontal scan line; as well as A disparity value between the corresponding feature points in the first data and the second data is calculated, wherein each pixel value in the depth map corresponds to the disparity value of a corresponding point in the scene of the depth map.

4. The method of claim 1, wherein the determining the road segmentation comprises: Generating a gradient map by calculating the magnitude and direction of the gradient at each pixel of the depth map; generating a gradient-based graph from the gradient map by initiating a first region growing process with predefined regions in the gradient map; and A road mask is generated from the gradient-based graph, the road mask being indicative of the road segmentation.

5. The method of claim 4, wherein said generating said gradient-based graphics comprises: determining, based on the magnitude and direction of the gradient at the pixel, whether a gradient difference between a pixel in the predefined area of ​​the gradient map and an adjacent pixel in the predefined area is less than a first gradient change threshold; Selecting the pixel as a seed ground point in the predefined area based on the gradient difference between the pixel in the predefined area and the neighboring pixel in the predefined area being less than a first gradient change threshold; determining whether a gradient difference between a pixel outside the predefined area and a neighboring pixel outside the predefined area is less than a second gradient change threshold that is different from the first gradient change threshold; and Based on the gradient difference between the pixel outside the predefined area and the neighboring pixel outside the predefined area being less than the second gradient change threshold, the predefined area is adjusted by adding a seed point outside the predefined area using a graph search algorithm.

6. The method of claim 5, further comprising fitting a quadratic model to the road segmentation using a second region growing process and a stereo depth error model to produce a refined quadratic road model, wherein the stereo depth error model characterizes errors in depth values ​​of the depth map, wherein fitting the quadratic model to the road segmentation comprises: Selecting pixels in the road mask as initial seed points; performing the second region growing process on the road mask to iteratively expand the initial seed point into a growing region by adding neighboring pixels around the initial seed point; refining one or more depth values ​​in the growing region using the stereo depth error model; as well as The secondary model is fitted to the growth region having the refined one or more depth values ​​to generate the refined secondary road model. 7 . The method of claim 6 , wherein determining the road surface height information comprises calculating road surface height values ​​at one or more points corresponding to the at least a portion of the road segmentation using the refined secondary road model. 8 . The method of claim 1 , wherein the determining the road surface height information comprises generating a road surface height grid, each cell of the road surface height grid indicating the road surface height information of a corresponding portion of the road segmentation.

9. The method according to claim 1, further comprising: detecting the non-road object in the depth map; Segmenting the depth map into a plurality of rod-like pixel representations; classifying the plurality of stick pixel representations into road stick pixel representations and non-road stick pixel representations; as well as The non-road object is identified as at least one of the non-road stick pixel representations in the depth map.

10. The method of claim 9, wherein said determining said height estimate of said non-road object comprises: determining a location in the scene of the depth map corresponding to the at least one of the non-road stick pixel representations in the depth map; determining the height estimate of the at least one of the non-road stick pixel representations at the location in the scene corresponding to the at least one of the non-road stick pixel representations; and Determine whether the height estimate of at least one of the non-road stick pixel representations exceeds a height threshold and whether the at least one of the non-road stick pixel representations is connected to the ground via other non-road stick pixel representations, wherein when the height estimate of one of the non-road stick pixel representations exceeds the height threshold and the at least one of the non-road stick pixel representations is not connected to the ground via other non-road stick pixel representations, identify the at least one of the non-road stick pixel representations as an overhead obstacle.

11. The method of claim 9, wherein providing an indication of a height clearance estimate comprises: projecting the height estimate of the one of the non-road stick pixel representations into one or more cells of a probabilistic occupancy grid, wherein the probabilistic occupancy grid comprises a plurality of cells, each cell indicating a probability of an overhead obstacle being present in the cell; as well as The height gap estimate is determined by subtracting the height estimate of the one of the non-road stick pixel representations in a cell of the probabilistic occupancy grid from the height of the cell, wherein the cell of the probabilistic occupancy grid indicates a value representing the height gap estimate. 12 . The method of claim 1 , wherein the indication of the height clearance estimate is provided for display on a user interface of a vehicle.

13. A system, comprising: Memory; and at least one processor coupled to the memory and configured to: Acquire a first stereo image and a second stereo image respectively from different cameras of the vehicle; generating a depth map based on a disparity between the first stereoscopic image and the second stereoscopic image; determining a road segmentation based at least in part on the depth map; generating a road surface height grid, wherein each cell of the road surface height grid indicates road surface height information of a corresponding portion of the road segmentation; Detecting non-road objects in the depth map; Segmenting the depth map into a plurality of rod-like pixel representations; classifying the plurality of stick pixel representations into road stick pixel representations and non-road stick pixel representations; identifying the non-road object as at least one of the non-road stick pixel representations in the depth map; determining a height estimate for the at least one of the non-road stick pixel representations corresponding to at least a portion of the road segmentation; and An indication of a height gap estimate for the at least a portion of the road segmentation is provided based at least in part on the height estimate of the at least one of the road surface height information and the non-road stick pixel representation.

14. The system of claim 13, wherein the at least one processor configured to generate the depth map is further configured to: Correcting the first stereoscopic image and the second stereoscopic image to a common horizontal scan line; determining corresponding feature points between the first stereoscopic image and the second stereoscopic image along the common horizontal scan line; and A disparity value between the corresponding feature points in the first stereoscopic image and the second stereoscopic image is calculated, wherein each pixel value in the depth map corresponds to the disparity value of a corresponding point in a scene of the depth map.

15. The system of claim 13, wherein the at least one processor configured to determine the road segmentation is further configured to: Generate a gradient map by calculating the magnitude and direction of the gradient at each pixel of the depth map, generating a gradient-based graph from the gradient map by initiating a first region growing process with predefined regions in the gradient map, and generating a road mask from the gradient-based graph, the road mask indicating the road segmentation, wherein the at least one processor configured to generate the gradient-based graphics is further configured to: determining, based on the magnitude and direction of the gradient at the pixel, whether a gradient difference between a pixel in the predefined area of ​​the gradient map and an adjacent pixel in the predefined area is less than a first gradient change threshold; Selecting the pixel as a seed ground point in the predefined area based on the gradient difference between the pixel in the predefined area and the neighboring pixel in the predefined area being less than a first gradient change threshold; determining whether a gradient difference between a pixel outside the predefined area and an adjacent pixel outside the predefined area is less than a second gradient change threshold; and Based on the gradient difference between the pixel outside the predefined area and the neighboring pixel outside the predefined area being less than a second gradient change threshold different from the first gradient change threshold, the predefined area is adjusted by adding a seed point outside the predefined area using a graph search algorithm.

16. The system of claim 15, wherein the at least one processor is further configured to fit a secondary model to the road segmentation using a second region growing process and a stereo depth error model to produce a refined secondary road model, wherein the stereo depth error model characterizes errors in depth values ​​of the depth map, wherein the at least one processor configured to fit the secondary model to the road segmentation is further configured to: Selecting pixels in the road mask as initial seed points; performing the second region growing process on the road mask to iteratively expand the initial seed point into a growing region by adding neighboring pixels around the initial seed point; refining one or more depth values ​​in the growing region using the stereo depth error model; as well as The secondary model is fitted to the growth region having the refined one or more depth values ​​to generate the refined secondary road model.

17. The system of claim 16, wherein the at least one processor configured to generate the road surface height grid is further configured to calculate road surface height values ​​at one or more points corresponding to the at least a portion of the road segmentation using the refined secondary road model.

18. The system of claim 13, wherein the at least one processor configured to determine the height estimate of the at least one of the non-road stick pixel representations is further configured to: determining a location in the scene of the depth map corresponding to the at least one of the non-road stick pixel representations in the depth map; determining the height estimate of the at least one of the non-road stick pixel representations at the location in the scene corresponding to the at least one of the non-road stick pixel representations; and Determine whether the height estimate of at least one of the non-road stick pixel representations exceeds a height threshold and whether the at least one of the non-road stick pixel representations is connected to the ground via other non-road stick pixel representations, wherein when the height estimate of one of the non-road stick pixel representations exceeds the height threshold and the at least one of the non-road stick pixel representations is not connected to the ground via other non-road stick pixel representations, identify the at least one of the non-road stick pixel representations as an overhead obstacle.

19. The system of claim 13, wherein the at least one processor configured to provide the indication of the altitude gap estimate is further configured to: projecting the height estimate of the one of the non-road stick pixel representations into one or more cells of a probabilistic occupancy grid, wherein the probabilistic occupancy grid comprises a plurality of cells, each cell indicating a probability of an overhead obstacle being present in the cell; and The height gap estimate is determined by subtracting the height estimate of the one of the non-road stick pixel representations in a cell of the probabilistic occupancy grid from the height of the cell, wherein the cell of the probabilistic occupancy grid indicates a value representing the height gap estimate.

20. A vehicle, comprising: First Camera; Second camera; and A processor, the processor being configured to: acquiring stereo images from the first camera and the second camera; generating a depth map based on the disparity between the stereo images; generating an initial road segmentation by performing a gradient-based graph search on the depth map; generating a stereo depth error model using a region growing process driven by the stereo depth error model to fit a refined secondary road model to the initial road segmentation; Determine a road surface height grid, each cell of the road surface height grid indicates road surface height information of a corresponding portion of the initial road segmentation; Segmenting the depth map into a plurality of rod-like pixel representations; as well as A height gap is determined based on the road surface height information using a probabilistic occupancy grid indicating one or more stick pixel representations representing overhead obstacles from the plurality of stick pixel representations.

Images