Method and device for detecting drivable area based on image key points, and vehicle

Abstract

The invention discloses a drivable area detection method and device based on image key points and a vehicle. The drivable area detection method comprises the following steps: acquiring an RGB image acquired by a vehicle-mounted camera of the vehicle; and performing feature extraction on the RGB image by using a preset model, and outputting a feature map. And calculating parameter information of the feature map, wherein the parameter information comprises a confidence coefficient thermodynamic diagram, a category diagram and offset. And screening out key points from each column of pixel points of the feature map according to the parameter information. And sequentially connecting a plurality of key points screened from each column of pixel points to generate a 2D drivable area boundary. And projecting the 2D travelable area boundary to a 3D space coordinate system by using a preset transformation rule, and generating a space coordinate of the 3D travelable area boundary. Compared with the prior art, the drivable area detection method based on the image key points has remarkable advantages in the aspects of precision, robustness, calculation efficiency and the like.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, and in particular to a drivable area detection method and device based on image key points and a vehicle. BACKGROUND

[0002] With the development of automatic driving technology, the requirements for environmental perception technology of unmanned vehicles are becoming higher and higher. Drivable area detection, as an important part of the automatic driving system, directly affects the safety, driving accuracy and navigation ability of the vehicle.

[0003] Existing drivable area detection methods mainly include semantic segmentation-based, laser radar and image fusion-based, and traditional feature-based detection methods. Although the existing drivable area detection methods have improved the environmental perception ability of intelligent driving systems to some extent, these methods generally have the problems of insufficient precision, poor robustness and large consumption of computing resources, which need to be improved. SUMMARY

[0004] To solve the technical problems of insufficient precision, poor robustness and large consumption of computing resources in the prior art, the purpose of the present application is to provide a drivable area detection method and device based on image key points, which has significant advantages in precision, robustness and computing efficiency, and can provide a more reliable, efficient and low-cost solution for automatic driving systems, promoting the wide application of intelligent driving technology.

[0005] The purpose of the present application is achieved by the following technical solutions:

[0006] In a first aspect, the present application provides a drivable area detection method based on image key points, which comprises:

[0007] obtaining an RGB image collected by a vehicle-mounted camera of a vehicle;

[0008] extracting features of the RGB image using a preset model to output a feature map;

[0009] calculating parameter information of the feature map, the parameter information including a confidence heat map, a category map and an offset;

[0010] According to the parameter information, filtering out key points from each column of pixel points in the feature map;

[0011] connecting the multiple key points filtered out from each column of pixel points in sequence to generate a 2D drivable area boundary;

[0012] projecting the 2D drivable area boundary into a 3D space coordinate system using a preset transformation rule to generate a space coordinate of the 3D drivable area boundary;

[0013] The drivable area of ​​the vehicle is determined based on the spatial coordinates of the boundary of the 3D drivable area.

[0014] In one possible implementation, the feature extraction of the RGB image using a preset model specifically involves extracting features from the RGB image through a deep neural network and outputting a downsampled feature map, with a downsampling factor of 8 or 16.

[0015] In one possible implementation, the activation function of the confidence heatmap is the Sigmoid function, with an output value of [0,1], which represents the confidence level of whether the corresponding region of the RGB image corresponding to each pixel in the feature map is a drivable area for a vehicle.

[0016] In one possible implementation, key points are selected from each column of pixels in the feature map based on the parameter information, specifically including:

[0017] The feature map is divided into multiple columns, each column containing multiple pixels;

[0018] Based on the confidence heatmap, the pixels with the highest confidence are selected from each column of pixels and used as key points.

[0019] Based on the offset, generate the sub-pixel offset of the key point;

[0020] Add the sub-pixel offset to the ordinate of the key point.

[0021] In one possible implementation, multiple key points selected from each column of pixels are sequentially connected to generate a 2D drivable area boundary, specifically including:

[0022] Multiple key points selected from each column of pixels are connected in a horizontal order to generate the 2D drivable area boundary.

[0023] Assign corresponding category information to each key point based on the category map to form category information for the 2D drivable area boundary.

[0024] In one possible implementation, the activation function of the category map is the Softmax function, and the output represents the probability distribution of the category to which each pixel belongs.

[0025] In one possible implementation, the preset transformation rule adopts the inverse perspective transformation rule.

[0026] In a second aspect, the present invention provides a drivable region detection device based on image key points, the drivable region detection device comprising:

[0027] The acquisition module is used to acquire RGB images captured by the vehicle's onboard camera;

[0028] The extraction module is used to extract features from the RGB image using a preset model and output a feature map;

[0029] The calculation module is used to calculate the parameter information of the feature map, which includes a confidence heatmap, a category map, and an offset.

[0030] The filtering module is used to filter key points from each column of pixels in the feature map based on the parameter information.

[0031] The generation module is used to sequentially connect multiple key points selected from each column of pixels to generate a 2D drivable area boundary.

[0032] The transformation module is used to project the 2D drivable area boundary into a 3D spatial coordinate system using a preset transformation rule, thereby generating the spatial coordinates of the 3D drivable area boundary.

[0033] The output module is used to determine the drivable area of ​​the vehicle based on the spatial coordinates of the boundary of the 3D drivable area.

[0034] Thirdly, the present invention provides an electronic device, comprising:

[0035] At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform a drivable region detection method based on image key points, as in any possible implementation of the first aspect.

[0036] Fourthly, the present invention also provides a vehicle comprising a drivable area detection device as described in the second aspect, or the vehicle comprising electronic equipment as described in the third aspect.

[0037] Compared with the prior art, the present invention has at least the following beneficial effects:

[0038] This invention provides a method for detecting drivable regions based on image key points. The method includes: acquiring an RGB image captured by a vehicle's onboard camera; extracting features from the RGB image using a preset model to output a feature map; calculating parameter information of the feature map, including a confidence heatmap, a category map, and an offset; selecting key points from each column of pixels in the feature map based on the parameter information; sequentially connecting multiple key points selected from each column of pixels to generate a 2D drivable region boundary; projecting the 2D drivable region boundary onto a 3D spatial coordinate system using a preset transformation rule to generate the spatial coordinates of the 3D drivable region boundary; and determining the drivable region of the vehicle based on the spatial coordinates of the 3D drivable region boundary.

[0039] This invention aims to address the shortcomings of existing drivable area detection technologies in terms of accuracy, robustness, and computational efficiency. Most existing technologies rely on methods such as semantic segmentation, LiDAR, and image fusion, but these methods often suffer from low accuracy, poor robustness, and high computational resource consumption, making it difficult to meet the real-time and high-efficiency requirements in complex environments. This invention utilizes image keypoint-based detection technology to effectively and reliably identify and segment drivable vehicle areas. Compared to traditional methods, it significantly reduces computational load by focusing on key points in the image rather than the entire image region, effectively reducing computational resource consumption, improving computational efficiency, and simultaneously enhancing detection accuracy and system robustness.

[0040] The technology of this invention reduces reliance on high-performance hardware, lowers system costs, and can adapt to various environmental disturbances, such as changes in lighting, weather conditions, and occlusion, ensuring stable operation of the autonomous driving system in various complex scenarios. Furthermore, by reducing computational load, improving detection accuracy, and enhancing system robustness, it lowers hardware requirements, reduces computational burden, and reduces costs, providing a more efficient and feasible solution for the widespread application of intelligent driving technology. Attached Figure Description

[0041] Figure 1 A schematic diagram of the hardware system architecture of the vehicle provided in the embodiments of this application;

[0042] Figure 2 A flowchart illustrating the drivable area detection method provided in this application embodiment;

[0043] Figure 3 A schematic diagram of the key point screening process provided for embodiments of this application;

[0044] Figure 4 A schematic diagram illustrating the process of generating a 2D drivable area boundary provided in an embodiment of this application;

[0045] Figure 5 A schematic diagram of a drivable area detection device based on image key points provided in this application embodiment;

[0046] Figure 6 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application;

[0047] Figure 7 RGB images captured by the vehicle-mounted camera provided in the embodiments of this application;

[0048] Figure 8 A visualization of the key point set of the 2D drivable area boundary provided in the embodiments of this application. Detailed Implementation

[0049] To facilitate understanding of the present invention, the technical solutions and advantages of the invention will be further described in detail below with reference to the accompanying drawings and embodiments. Any mechanisms or methods not elaborated in this invention can be referred to in the prior art. The specific structures and features of the present invention are illustrated below by way of example and should not be construed as limiting the present invention in any way. Furthermore, any of the technical features mentioned below (including implicit or disclosed features), as well as any technical features directly shown or implied in the figures, can be arbitrarily combined or deleted among these technical features to form more other embodiments that may not be directly or indirectly mentioned in this invention. The accompanying drawings show preferred embodiments of the present invention. However, the present invention can be implemented in many different forms and is not limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the disclosure of the present invention.

[0050] With the development of intelligent driving technology, environmental perception systems play a crucial role in applications such as autonomous driving, assisted driving, and low-speed unmanned vehicles. Drivable area detection is one of the core tasks of environmental perception, aiming to identify the drivable area around the vehicle and ensure that the vehicle can drive safely and accurately in complex road environments.

[0051] Currently, existing methods for detecting drivable areas mainly include:

[0052] (a) A drivable region detection scheme based on semantic segmentation:

[0053] This approach typically employs deep semantic segmentation networks (such as FCN, U-Net, and DeepLab) to perform pixel-level classification on 2D RGB images captured by cameras, dividing each pixel into categories such as "drivable area," "obstacle," and "background." A 2D drivable area mask is generated from the segmentation results and then converted into coordinates in 3D space using camera parameters. While this method achieves good region recognition, it suffers from high computational resource consumption and poor real-time performance, especially in high-resolution images and complex environments, easily leading to inference delays and failing to meet the real-time requirements of autonomous driving systems. Furthermore, semantic segmentation methods often cannot effectively predict the grounding points of suspended objects, resulting in inaccurate positioning of the projection points of suspended objects (such as fire extinguisher boxes, warning lines, and turnstile barriers), potentially leading to misjudgments of drivable areas.

[0054] (II) Driving area detection scheme combining lidar and imagery:

[0055] This type of method combines the advantages of LiDAR and camera images. LiDAR collects 3D point cloud data of the environment, filters ground point clouds as preliminary candidates for drivable areas, and then uses camera images for semantic annotation to distinguish categories such as ground, pedestrians, and vehicles. Finally, the two are fused to generate a 3D drivable area. While this approach can enhance the accuracy of drivable area detection by combining the depth information of LiDAR with the semantic information of the images, it faces limitations such as the high cost of LiDAR and its susceptibility to adverse weather conditions (such as rain, snow, fog, haze, and dust storms). Furthermore, LiDAR point cloud data may be lost or contain errors in complex environments, leading to a decrease in the accuracy of drivable area detection.

[0056] (III) Driving area detection scheme based on traditional features:

[0057] Traditional drivable region detection methods, based on hand-designed features (such as color, texture, and edges), extract drivable regions from images using computer vision algorithms such as threshold segmentation, region growing, and contour extraction. These methods are effective for region recognition in simple scenes, but in complex environments, they often suffer from missed detections and false detections due to their sensitivity to changes in lighting and road surface materials. Furthermore, traditional algorithms have low accuracy in extracting region boundaries, making it difficult to meet the requirements of high-precision driving and region boundary detection in autonomous driving systems.

[0058] However, existing methods for detecting drivable areas generally suffer from the following problems:

[0059] (1) High computational resource consumption: Semantic segmentation networks and point cloud data processing usually require a lot of computational resources, especially in high-resolution images and complex environments, which may lead to a decline in real-time performance and make it difficult to meet the real-time response requirements of autonomous driving systems.

[0060] (2) Inability to accurately detect suspended objects: Existing image-based and lidar-based detection methods often cannot effectively locate the projection point of suspended objects (such as fire boxes, railings, etc.) on the ground, which may lead to misjudgment of drivable areas and affect the safety of autonomous driving systems.

[0061] (3) Poor robustness: LiDAR is prone to failure in severe weather conditions such as rain, snow, fog and haze. Furthermore, traditional feature-based algorithms are sensitive to factors such as changes in lighting and differences in road surface materials, resulting in low detection accuracy in complex scenarios and affecting the stability and reliability of the system.

[0062] (4) Low detection accuracy: Traditional methods and existing deep learning methods still have limitations in accuracy in complex scenarios, especially in dynamic environments (such as when obstacles or pedestrians appear), it may be difficult to update the drivable area in real time and accurately.

[0063] To address the problems in existing technologies, this invention proposes a drivable region detection method based on image key points. This method effectively and reliably identifies and segments drivable vehicle regions through image key point detection technology. Compared to traditional methods, it significantly reduces computational load by focusing on key points in the image rather than the entire image region, effectively reducing computational resource consumption, improving computational efficiency, and simultaneously enhancing detection accuracy and system robustness. This invention reduces reliance on high-performance hardware, lowers system costs, and can adapt to various environmental disturbances, such as changes in lighting, weather effects, and occlusion, ensuring stable operation of autonomous driving systems in various complex scenarios. Furthermore, by reducing computational load, improving detection accuracy, and enhancing system robustness, it lowers hardware requirements, reduces computational burden, and reduces costs, providing a more efficient and feasible solution for the widespread application of intelligent driving technology.

[0064] The drivable area detection method, drivable area detection device, and electronic device based on image key points provided in this application can be applied to drivable area detection scenarios for vehicles in different road driving scenarios.

[0065] For example, the drivable area detection method, drivable area detection device, and electronic device provided in this application embodiment can be applied to drivable area detection scenarios in ordinary road driving scenarios (such as urban streets, residential roads, highways, rural roads, etc.). Of course, the drivable area detection method, drivable area detection device, and electronic device provided in this application embodiment can also be applied to other scenarios, and this application embodiment does not limit them.

[0066] In this embodiment of the application, the entity executing the drivable area detection method provided in this embodiment can be a drivable area detection device. Exemplarily, the drivable area detection device can be a chip, chip system, circuit, or module, etc., and this application does not impose any limitations.

[0067] For example, the drivable area detection device involved in the embodiments of this application can be a chip system; of course, it can also be other computing devices with data and / or image processing functions.

[0068] Figure 1 This is a schematic diagram of the hardware system architecture of a vehicle provided in an embodiment of this application. For example, the vehicle provided in this embodiment may be an automobile, a driverless vehicle, or a driverless cleaning vehicle, etc.

[0069] like Figure 1As shown, the hardware system architecture diagram in the vehicle may include, but is not limited to: multiple monocular vehicle cameras 10, a chip system 11, an Electronic Control Unit (ECU) 12, a controller 13, and a Controller Area Network (CAN) bus 14. The monocular vehicle cameras 10 are used to acquire image data; the CAN bus 14 is used to provide vehicle driving status data; the chip system 11 is used to detect drivable areas based on the image data acquired by the monocular vehicle cameras 10; in another embodiment, the chip system 11 is used to detect drivable areas based on the image data acquired by the monocular vehicle cameras 10 and the data provided by the CAN bus 14. The ECU 12 is used to determine control decisions based on the detection results of the chip system 11 and the data provided by the CAN bus 14; the controller 13 is used to control the movement of the vehicle based on the control decisions of the ECU 12. It should be understood that the chip system 11 can employ the drivable area detection method provided in the embodiments of this application.

[0070] The above describes the application scenarios and vehicles of the drivable area detection method provided in the embodiments of this application. The following will describe in detail a drivable area detection method based on image key points provided in this application.

[0071] Please see Figure 2 , Figure 2 This is a flowchart illustrating the drivable area detection method provided in an embodiment of this application. Figure 2 As shown, the drivable area detection method includes the following steps 201-207.

[0072] 201. Obtain RGB images captured by the vehicle's onboard camera.

[0073] In this embodiment, RGB image data of the road environment in front of the vehicle is acquired using an onboard camera mounted on the vehicle. The onboard camera is a monocular camera, and its installation position and orientation are pre-calibrated to ensure that the acquired image covers a predetermined field of view in the vehicle's direction of travel. Step 201 obtains raw image data for subsequent drivable area detection, providing basic input for subsequent feature extraction and key point analysis.

[0074] In one specific implementation, a monocular camera is used to capture RGB images of the road in front of the vehicle. The image resolution can be adjusted according to the application scenario (e.g., 800×600, 1280×720), and the capture frame rate is no less than 30fps to meet real-time driving requirements. For example, the camera is installed inside the vehicle's windshield (with the lens facing directly forward) to ensure that the image covers a 120° field of view directly in front of the vehicle.

[0075] 202. Use a preset model to extract features from the RGB image and output a feature map.

[0076] In this embodiment, the RGB image obtained in step 201 is input into a preset model for feature extraction. The preset model includes a feature extraction network for extracting high-level semantic information and spatial structure information of the image. By performing multi-layer convolution and downsampling on the RGB image, a feature map with reduced resolution relative to the original image is output. The feature map retains key feature information related to the drivable area, such as road boundaries, obstacle outlines, and ground structure.

[0077] Step 202 compresses and abstracts the original image information, providing a unified feature representation for subsequent parameter calculations.

[0078] In some implementations, the step of using a preset model to extract features from the RGB image specifically involves using a deep neural network to extract features from the RGB image and outputting a downsampled feature map, with a downsampling factor of 8 or 16.

[0079] Specifically, the preset model uses a deep neural network (such as ResNet-50, MobileNet, HourGlass, etc.), extracts features from the image through the convolution and pooling layers of the backbone, and outputs the downsampled feature map. Depending on the balance between accuracy and computational efficiency, the downsampling factor can be set to 8 or 16 times.

[0080] Backbone employs multi-layer feature encoding of the input image through convolutional layers, pooling layers, and activation functions. This transforms high-dimensional, redundant pixel information into a low-dimensional, compact feature representation for use by the subsequent keypoint prediction module. The network output is a downsampled feature map, with the downsampling factor set to 8x or 16x depending on the balance between accuracy and computational efficiency. Downsampling reduces the feature map size, significantly decreasing computational load while retaining crucial spatial structural information.

[0081] 203. Calculate the parameter information of the feature map, which includes a confidence heatmap, a category map, and an offset.

[0082] In this embodiment of the application, by simultaneously outputting confidence, category and offset, the model can not only determine whether "driving key points exist", but also clarify the semantic attributes of key points and accurately locate their positions in the original image, providing multi-dimensional constraints for subsequent key point selection and boundary generation.

[0083] In practical implementation, the preset model includes two parts: a backbone network and a key point prediction head.

[0084] Backbone: Extracts multi-layered, compact feature maps from the input RGB image, preserving road boundaries, obstacles, and ground features. Optional network architectures include ResNet-50, MobileNet, and HourGlass, chosen based on vehicle computing power and real-time requirements. Processing flow: Input image → Multi-layer convolution → Activation function → Pooling → Downsampled feature map output. Downsampling ratio: Typically set to 8x or 16x, balancing accuracy and computational efficiency.

[0085] Keypoint Prediction Header: Predicts keypoints for each column of pixels on the downsampled feature map and outputs the required parameter information. Structure: Consists of at least three 1×1 convolutional layers, corresponding to the output confidence heatmap, class map, and offset, respectively. The following provides supplementary explanations of the confidence heatmap, class map, and offset:

[0086] (1) Confidence heatmap: The confidence level of each pixel in the feature map is whether the corresponding area in the RGB image is a "driving key point". For this specific task, the point with the highest confidence level in each column is the key point.

[0087] (2) Category map: Represents the category of key points (such as ground, curb, green plants, vehicles, people, etc.), and uses one-hot encoding ground truth to supervise the model output.

[0088] (3) Offset: Since the output image is generated based on the downsampled image, the pixel precision of the original image will be lost. For example, for an 8x downsampled image, 1 pixel corresponds to 8 pixels in the original image. Therefore, a subpixel offset dy needs to be predicted. This offset is used to correct the precision loss so that it can accurately correspond to the pixels of the original image.

[0089] For example, assuming the true coordinate point of a certain column in the original image is y, and the downsampling factor is s, then the true value of dy is defined by the following formula: ;in, The symbol is for rounding down. As can be seen from the formula, the range of dy is [0, 1].

[0090] In some implementations, the activation function of the confidence heatmap is the Sigmoid function, with an output value of [0,1], used to represent the confidence level of whether the corresponding region of the RGB image corresponding to each pixel in the feature map is a drivable area for a vehicle. The confidence value indicates whether the image region corresponding to the pixel or feature map position belongs to a drivable area for a vehicle; in step 204, key point filtering, the most representative key points can be selected based on the confidence level, and low-confidence regions can be removed, thereby ensuring the accuracy of key point prediction.

[0091] In some implementations, the activation function of the category map is the Softmax function, and the output represents the probability distribution of the category to which each pixel belongs. Categories may include ground, curb, obstacles (vehicles, fences), ground contact points of suspended objects, etc. The category map is used for semantic annotation of key points, providing semantic constraints for 2D boundary generation and 3D projection; combined with a confidence heatmap, it ensures that the key points selected in each column are both the farthest points of the drivable boundary and have clearly defined category information.

[0092] Specifically, this patent precisely defines the drivable area as the region extending downwards from the set of points representing the furthest reachable points in each column of the image. The furthest reachable point refers to the first insurmountable obstacle encountered by the vehicle as it travels upwards along the ground along the column of the image from its current position. This obstacle can be a ground-based obstacle (such as a bollard or stone pier) or the ground projection of a suspended obstacle (such as a gate or suspended warning line projected onto the ground). In essence, predicting the upper boundary of this region, i.e., predicting one furthest reachable point for each column, can be done using a keypoint prediction approach. The ground truth values ​​of the furthest reachable points need to be manually labeled to clarify the maximum distance the vehicle can travel in various scenarios.

[0093] In one example, based on the downsampled feature map, a keypoint detection head composed of 1x1 convolutions outputs three channels. The first channel corresponds to one value, representing a confidence heatmap with a confidence level between 0 and 1 (inclusive). The second channel corresponds to n values, representing the scores of n different categories, with each category's score between 0 and 1 (inclusive) and the sum of the scores being 1. The third channel corresponds to one value, representing the sub-pixel offset dy (a floating-point number).

[0094] 204. Based on the parameter information, key points are selected from each column of pixels in the feature map.

[0095] In this embodiment of the application, based on the parameter information obtained in step 203, the feature map is traversed in the column direction. In each column of pixels, the pixel with the highest confidence is selected as the candidate key point of the column according to the confidence heatmap, and the position of the key point is corrected in combination with the offset.

[0096] Key points are defined as the farthest ground point or contact point of a suspended object that a vehicle can reach along the direction of travel in the corresponding image column direction. It can be understood that the key point represents the farthest feasible position that a vehicle can reach in the corresponding image column direction.

[0097] In one implementation, the feature map is traversed along the column direction. In each column, pixels that meet the preset confidence conditions are selected as candidate key points based on the confidence heatmap. The positions of the candidate key points are then corrected by the offset to determine the key points corresponding to that column.

[0098] By filtering key points by column, the two-dimensional region detection problem is transformed into a one-dimensional column-level key point prediction problem. This ensures that each column retains only one most representative drivable boundary point, fundamentally reducing redundant computation and avoiding multi-point conflicts. Step 204 extracts column-level key points with clear physical meaning from the two-dimensional feature map, avoiding pixel-level classification operations on the entire image.

[0099] 205. Connect the multiple key points selected from each column of pixels in sequence to generate the 2D drivable area boundary.

[0100] In this embodiment, the multiple key points obtained from each column in step 204 are sequentially connected according to their order in the horizontal direction of the image to form a continuous two-dimensional boundary curve, i.e., the 2D drivable area boundary. This 2D drivable area boundary is used to represent the upper boundary of the drivable area of ​​the vehicle in the current image view. It can be understood that the area below the 2D drivable area boundary is defined as the two-dimensional drivable area.

[0101] Step 205 constructs a two-dimensional drivable region based on a set of key points, avoiding redundant computations associated with traditional pixel-level segmentation. It is understood that sequentially connecting column-level key points directly constructs the geometric boundary of the drivable region, avoiding the complex region filling and contour extraction processes of traditional semantic segmentation, and improving the stability and continuity of boundary generation.

[0102] 206. Using preset transformation rules, project the 2D drivable area boundary into a 3D spatial coordinate system to generate the spatial coordinates of the 3D drivable area boundary.

[0103] In this embodiment, by directly mapping the two-dimensional boundary to three-dimensional space, the detection results can be directly used by the autonomous driving system for path planning and motion control, avoiding additional depth estimation steps and improving overall computational efficiency.

[0104] In one implementation, a preset transformation rule is used to convert the coordinates of the two-dimensional drivable area boundary into 3D world coordinates based on the camera's calibration parameters.

[0105] In another implementation, the preset transformation rule employs an inverse perspective transformation rule to map the 2D drivable area boundary from the image coordinate system to three-dimensional spatial coordinates in the vehicle coordinate system. Since the 2D drivable area boundary points predicted by this invention are all ground points, they can be directly back-projected to 3D spatial coordinates in the vehicle coordinate system based on the inverse perspective transformation (IPM) algorithm. Because all key points are ground or ground points (i.e., height coordinates are 0), no additional depth information is required to output the set of spatial coordinates of the 3D drivable area.

[0106] Inverse perspective transformation is a spatial mapping method based on the geometric imaging model of a camera. Its basic principle is to reverse the pixel coordinates in the two-dimensional image to the ground plane in the vehicle coordinate system under the known camera installation posture and calibration parameters, thereby restoring the positional relationship of the pixels in the actual space.

[0107] In this invention, by applying ground constraint assumptions to key points of the drivable area boundary, each two-dimensional key point can uniquely correspond to a three-dimensional spatial coordinate point, thereby realizing a direct mapping from the two-dimensional boundary to the three-dimensional boundary.

[0108] Among them, the inverse perspective transformation rule establishes a mapping relationship between the image coordinate system and the vehicle coordinate system based on the intrinsic and extrinsic parameter matrices of the camera, and introduces ground constraint conditions during the transformation process so that the two-dimensional boundary points directly correspond to the ground spatial positions under the vehicle coordinate system.

[0109] Since the boundary of the two-dimensional drivable area is composed of the key points selected in steps 204 and 205, and the key points all represent ground points or grounding points of suspended objects, when performing inverse perspective transformation, the height value of the key points in the vehicle coordinate system can be preset to a fixed height, so that depth estimation or three-dimensional reconstruction calculation is not required for the key points.

[0110] 207. Determine the drivable area of ​​the vehicle based on the spatial coordinates of the boundary of the 3D drivable area.

[0111] In this embodiment, based on the distribution of the 3D drivable area boundary in the vehicle coordinate system, the area below the 3D drivable area boundary is considered a drivable area, thereby determining the permissible spatial range for the vehicle during its current and future driving. By constraining the space around the vehicle through the three-dimensional drivable area boundary, the autonomous driving system can make decisions based on real spatial coordinates, thereby improving driving safety and the reliability of path planning.

[0112] In some implementations, the drivable area of ​​the vehicle is determined based on the spatial coordinates of the 3D drivable area boundary in the vehicle coordinate system. Specifically, the 3D drivable area boundary is a set of spatial boundary points located in the vehicle coordinate system. This set of spatial boundary points forms a continuous spatial boundary curve along the vehicle's direction of travel, used to define the boundary range of the vehicle's drivable space in the current scene. In the vehicle coordinate system, the spatial area jointly defined by the 3D drivable area boundary, the vehicle's current position, and a preset ground plane is determined as the vehicle's drivable area.

[0113] Understandably, the drivable area represents the spatial range that a vehicle is allowed to travel within the current perception period, and can be directly used for subsequent path planning, motion control, or safety decisions.

[0114] As can be seen from the above scheme, the present invention does not generate drivable regions based on pixel-level semantic segmentation. Instead, it selects key points at the column level from the feature map, determines only one most representative drivable key point in each column, and constructs the boundary of the drivable region accordingly. This technical approach is significantly different from the processing method of densely classifying the entire image in the prior art.

[0115] It should be noted that existing technologies typically assume that pixel-level segmentation or 3D point clouds are required to obtain drivable areas. However, this invention introduces a keypoint prediction mechanism combining confidence, category, and offset, transforming the 2D region detection problem into a 1D column-level farthest reachable point prediction problem. This represents a fundamental difference in computational model, data structure, and processing flow, which would be readily apparent to those not skilled in the art based on conventional thinking. Furthermore, the coordinated calculation of parameter information, keypoint selection, and boundary generation in steps 203 to 205 reduces computational complexity while still stably generating continuous and accurate drivable area boundaries, which can be directly used for 3D spatial projection, thus improving the overall real-time performance and applicability of the system.

[0116] This invention achieves low-computing-power, high-real-time, and high-precision detection of drivable areas under monocular camera conditions through an innovative combination of "column-level key point prediction + multi-parameter description + lightweight feature extraction + 2D-3D mapping". Its technical solution is different from existing pixel-level segmentation, point cloud fusion or traditional feature detection methods, and has significant non-obvious and practical innovative value.

[0117] Please see Figure 3 , Figure 3 This is a schematic diagram illustrating the key point filtering process provided in an embodiment of this application. Specifically, based on the parameter information, key points are filtered from each column of pixels in the feature map, including:

[0118] 301. Divide the feature map into multiple columns, each column containing multiple pixels.

[0119] In practice, the pixels of the downsampled feature map are divided into several columns, with each column containing multiple pixels. Each column corresponds to a projection path in the RGB image along that column direction in front of the vehicle.

[0120] By dividing the data into columns, the problem of detecting two-dimensional drivable areas is transformed into a one-dimensional column-level key point prediction problem, reducing redundant computation and ensuring that each column retains only one most representative key point, thus laying the foundation for two-dimensional boundary generation.

[0121] 302. Based on the confidence heatmap, select the pixels with the highest confidence from each column of pixels as key points.

[0122] In practice, based on the confidence heatmap calculated in step 203, the pixel with the highest confidence in each column is selected as the key point for that column. The confidence heatmap outputs a value in the range [0,1] using the Sigmoid activation function, representing the probability that the area corresponding to the pixel is a drivable area for vehicles. By selecting the highest confidence point, the representativeness of each column of key points is ensured, reducing boundary deviations caused by mistakenly selecting low-confidence areas.

[0123] 303. Based on the offset, generate the sub-pixel offset of the key point.

[0124] In practice, the offset information output in step 203 is used to generate sub-pixel offsets for each keypoint. Since the feature map is a downsampled image, directly mapping the feature map pixel coordinates to the original image will result in accuracy loss. By predicting the sub-pixel offsets, the keypoint positions can be finely corrected, making them more accurately correspond to the drivable boundaries in the RGB image.

[0125] 304. Add the sub-pixel offset to the ordinate of the key point.

[0126] In practice, the sub-pixel offset generated in step 303 is added to the ordinate of the key point selected in step 302 to obtain the corrected vertical position of the key point in the original image coordinate system. Through sub-pixel ordinate correction, precise positioning of the key point is achieved, thereby ensuring the continuity of the 2D drivable area boundary and the accuracy of the 3D projection, providing reliable spatial information for vehicle path planning.

[0127] This application's embodiments achieve high-precision drivable area detection under low computing power conditions by combining column-level key point screening and sub-pixel offset correction. Compared with dense pixel-level segmentation or full-image sliding window detection in the prior art, this solution significantly reduces the amount of computation while improving boundary accuracy. This processing method combines two-dimensional key points with three-dimensional spatial projection to form a unique technical chain, providing a non-obvious technical solution for real-time and accurate drivable area detection under monocular camera conditions.

[0128] Please see Figure 4 , Figure 4 This is a schematic diagram illustrating the process of generating a 2D drivable area boundary provided in an embodiment of this application. Multiple key points selected from each column of pixels are sequentially connected to generate the 2D drivable area boundary, specifically including:

[0129] 401. Connect multiple key points selected from each column of pixels in a horizontal sequence to generate the 2D drivable area boundary.

[0130] In practical implementation, key points are connected horizontally in sequence, transforming the column-level key point set into a continuous two-dimensional boundary, thereby achieving an overall spatial contour description of the drivable area in front of the vehicle. This avoids contour extraction or region filling operations in traditional segmentation schemes, reducing computational complexity while ensuring the continuity and smoothness of the boundary; this two-dimensional boundary can be directly used as input for 3D projection or path planning modules.

[0131] Understandably, the x-axis of the image coordinate system (i.e., the row direction of each row of pixels) is connected in ascending order. The x-coordinate of each keypoint corresponds to the column index of the image; by connecting keypoints with increasing x-coordinates, a continuous boundary curve can be generated from the left to the right of the image, accurately representing the spatial contour of the drivable area in front of the vehicle.

[0132] 402. Assign corresponding category information to each key point according to the category map to form the category information of the 2D drivable area boundary.

[0133] In practical implementation, category information can include ground, curbs, obstacles (such as vehicles and fences), and the ground contact points of suspended objects. Assigning category information to each key point of the two-dimensional boundary ensures that the boundary not only represents geometric location but also carries semantic information. In subsequent two-dimensional-to-three-dimensional mapping or path planning, different categories of key points can be processed differently, such as obstacle avoidance and road boundary constraints, improving the reliability and accuracy of the drivable area.

[0134] This implementation connects column-level key points horizontally to form a two-dimensional boundary, and combines this with category information assignment. This combined technical solution differs from existing pixel-level segmentation or full-image contour extraction methods, enabling high-precision boundary generation on low-computing-power platforms. Category assignment, combined with key point locations, achieves a dual geometric and semantic description of the boundary, providing direct input for 3D mapping and autonomous driving path planning, forming a unique technical chain. This method significantly improves the accuracy and usability of two-dimensional boundaries while maintaining real-time performance, demonstrating the non-obvious nature of the technical solution.

[0135] Based on the above description, this application provides a specific application example for illustration, see [link to example]. Figure 7 and Figure 8 An application example of the drivable region detection method based on image key points described in this application includes the following process:

[0136] Step 1: RGB Image Acquisition. A 1920x1080 resolution vehicle-mounted monocular camera is selected and installed inside the vehicle's windshield, with the lens center aligned with the vehicle's centerline. The field of view (FOV) is 120°, the acquisition frame rate is set to 30fps, and the image format is JPEG. The acquired image is then transmitted to the vehicle's onboard processor. Figure 7 The image shown is an example of raw data from a vehicle-mounted monocular camera.

[0137] Step 2: Feature Extraction. ResNet50 is used as the fixed backbone, implemented using the PyTorch framework. The RGB image is normalized and input into the backbone, passing through 5 convolutional blocks and pooling layers, outputting a feature map downsampled by 16 times, with a size of 128x68.

[0138] Step 3: Keypoint Prediction. The prediction head contains three 1x1 convolutional layers, which output a confidence heatmap, a category map, and an offset (dy). The confidence heatmap has one output channel, uses the Sigmoid activation function, and has an output value range of [0,1]. The category map has C output channels, corresponding to the C categories of business-related keypoints, and uses the Softmax activation function. The offset has one output channel, no activation function, and an output value range of [0,1), corresponding to the sub-pixel offset value.

[0139] Step 4: Obtain the 2D drivable region keypoint set based on the model output. First, extract keypoints from each column of the feature map. In each column, take the keypoint with the highest score in the confidence heatmap as the farthest reachable point, and add the predicted sub-pixel offset dy to obtain the drivable region boundary. Finally, assign a pixel-wise category to this boundary based on the category corresponding to the highest score.

[0140] like Figure 8The system visualizes a set of key points for the 2D drivable area, where pink, yellow, blue, and red represent fences, vehicles, curbs, and other obstacles, respectively. The area enclosed by these key points, connected sequentially from left to right, represents the vehicle's drivable freespace.

[0141] Step 5: Using the camera intrinsic parameters, back-project the image onto the 3D vehicle coordinate system through inverse perspective transformation to guide the vehicle in autonomous driving.

[0142] See Figure 5 , Figure 5 This is a schematic diagram of a drivable region detection device based on image key points, provided in an embodiment of this application. The drivable region detection device can implement the above-described drivable region detection method based on image key points, wholly or partially, through software, hardware, firmware, or any combination thereof. In a specific implementation, the drivable region detection device includes:

[0143] The acquisition module is used to acquire RGB images captured by the vehicle's onboard camera;

[0144] The extraction module is used to extract features from the RGB image using a preset model and output a feature map;

[0145] The calculation module is used to calculate the parameter information of the feature map, which includes a confidence heatmap, a category map, and an offset.

[0146] The filtering module is used to filter out key points from each column of pixels in the feature map based on the parameter information.

[0147] The generation module is used to sequentially connect multiple key points selected from each column of pixels to generate a 2D drivable area boundary.

[0148] The transformation module is used to project the 2D drivable area boundary into a 3D spatial coordinate system using a preset transformation rule, thereby generating the spatial coordinates of the 3D drivable area boundary.

[0149] The output module is used to determine the drivable area of ​​the vehicle based on the spatial coordinates of the boundary of the 3D drivable area.

[0150] This application also provides an electronic device, including at least one processor, a memory, and a communication interface, wherein the processor is used to execute... Figures 2-4 The method for detecting drivable regions based on key points in an image is described above.

[0151] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0152] like Figure 6As shown, the electronic device 600 includes at least one processor 601, a memory 603, and a communication interface 602. The processor 601, memory 603, and communication interface 602 are communicatively connected, or can communicate via wireless transmission or other means. The communication interface 602 is used to receive 3D point cloud data sent by a sensing module (e.g., LiDAR); the memory 603 stores computer instructions, and the processor 601 executes these computer instructions to perform a drivable area detection method based on image key points as described in the aforementioned method embodiment.

[0153] It should be understood that in the embodiments of this application, the processor 601 may be a central processing unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0154] The memory 603 may include read-only memory and random access memory, and provides instructions and data to the processor 601. The memory 603 may also include non-volatile random access memory.

[0155] The memory 603 can be volatile memory or non-volatile memory, or it can include both. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0156] It should be understood that the electronic device 600 according to the embodiments of this application can perform the implementation of the embodiments of this application. Figures 2-4 The diagram illustrates a drivable region detection method based on image key points. A detailed description of the implementation of this drivable region detection method can be found above, and for the sake of brevity, it will not be repeated here.

[0157] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0158] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented in hardware, processor-executed software modules, or a combination of both. The software modules can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disks, removable disks, CD-ROMs, or any other form of storage medium known in the art.

[0159] This application also provides an autonomous driving vehicle, in conjunction with the above-described embodiments. Figure 1 The description states that corresponding [equipment / devices] can be deployed on autonomous vehicles. Figure 5 The drivable area detection device described in the embodiments is used to implement Figures 2 to 4 The corresponding embodiment includes a drivable area detection function based on image key points.

[0160] In another implementation, corresponding [equipment / devices] can be deployed on the autonomous vehicle. Figure 6 The electronic device described in the embodiments is used to implement Figures 2 to 4 The corresponding embodiment includes a drivable area detection function based on image key points.

[0161] The above embodiments are merely preferred embodiments of the present invention and should not be construed as limiting the scope of protection of the present invention. For those skilled in the art, it will be understood that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of the present invention. The scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A method for detecting drivable regions based on image key points, characterized in that, The method for detecting drivable areas includes: Acquire RGB images captured by the vehicle's onboard camera; The RGB image is used to extract features using a preset model, and a feature map is output. Calculate the parameter information of the feature map, which includes a confidence heatmap, a category map, and an offset; Based on the parameter information, key points are selected from each column of pixels in the feature map; Multiple key points selected from each column of pixels are connected sequentially to generate the 2D drivable area boundary. Using preset transformation rules, the 2D drivable area boundary is projected onto a 3D spatial coordinate system to generate the spatial coordinates of the 3D drivable area boundary. The drivable area of ​​the vehicle is determined based on the spatial coordinates of the boundary of the 3D drivable area.

2. The drivable region detection method based on image key points as described in claim 1, characterized in that: The step of using a preset model to extract features from the RGB image specifically involves using a deep neural network to extract features from the RGB image and outputting a downsampled feature map, with a downsampling factor of 8 or 16.

3. The drivable region detection method based on image key points as described in claim 1, characterized in that: The activation function of the confidence heatmap is the Sigmoid function, with an output value of [0,1], which is used to represent the confidence level of whether the corresponding region of the RGB image corresponding to each pixel in the feature map is a drivable area for vehicles.

4. The drivable region detection method based on image key points as described in claim 3, characterized in that, Based on the parameter information, key points are selected from each column of pixels in the feature map, specifically including: The feature map is divided into multiple columns, each column containing multiple pixels; Based on the confidence heatmap, the pixels with the highest confidence are selected from each column of pixels and used as key points. Based on the offset, generate the sub-pixel offset of the key point; Add the sub-pixel offset to the ordinate of the key point.

5. The drivable region detection method based on image key points as described in claim 3, characterized in that, Multiple key points selected from each column of pixels are sequentially connected to generate the 2D drivable area boundary, specifically including: Multiple key points selected from each column of pixels are connected in a horizontal order to generate the 2D drivable area boundary. Assign corresponding category information to each key point based on the category map to form category information for the 2D drivable area boundary.

6. The drivable region detection method based on image key points as described in claim 1, characterized in that: The activation function of the category map is the Softmax function, and the output represents the probability distribution of the category to which each pixel belongs.

7. The drivable region detection method based on image key points as described in claim 1, characterized in that: The preset transformation rule adopts the inverse perspective transformation rule.

8. A drivable region detection device based on image key points, characterized in that, The drivable area detection device includes: The acquisition module is used to acquire RGB images captured by the vehicle's onboard camera; The extraction module is used to extract features from the RGB image using a preset model and output a feature map; The calculation module is used to calculate the parameter information of the feature map, which includes a confidence heatmap, a category map, and an offset. The filtering module is used to filter out key points from each column of pixels in the feature map based on the parameter information. The generation module is used to sequentially connect multiple key points selected from each column of pixels to generate a 2D drivable area boundary. The transformation module is used to project the 2D drivable area boundary into a 3D spatial coordinate system using a preset transformation rule, thereby generating the spatial coordinates of the 3D drivable area boundary. The output module is used to determine the drivable area of ​​the vehicle based on the spatial coordinates of the boundary of the 3D drivable area.

9. An electronic device, characterized in that, include: At least one processor and a memory, the at least one processor being coupled to the memory for reading and executing instructions in the memory to perform a drivable region detection method based on image key points as described in any one of claims 1 to 7.

10. A vehicle, characterized in that, The vehicle includes the drivable area detection device as described in claim 8, or the vehicle includes the electronic device as described in claim 9.

Images