A monocular RGB camera-based three-dimensional drivable region detection method and device

Abstract

The application discloses a kind of three-dimensional drivable area detection method and device based on monocular RGB camera, its method includes obtaining the RGB image of monocular camera output, and the distortion correction is carried out to RGB image;The BTS algorithm is used to the RGB image after distortion correction to generate corresponding depth map for depth estimation;According to depth map, the best surface normal vector of each pixel point in depth map is calculated according to depth map;According to the best surface normal vector of each pixel point in depth map, the two-dimensional drivable area segmentation is carried out to the RGB image after distortion correction using SNE-RoadSeg algorithm, and two-dimensional drivable area graph is generated;Two-dimensional drivable area two-dimensional polygon set is obtained by image processing to two-dimensional drivable area graph;According to two-dimensional polygon set and depth map, two-dimensional polygon of two-dimensional drivable area is converted into three-dimensional polygon, and three-dimensional drivable area is obtained;The present method can output 3D information using only monocular RGB camera, with lower cost and more ease of use.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, and in particular to a method and apparatus for detecting three-dimensional drivable areas based on a monocular RGB camera. Background Technology

[0002] Driving area detection is an important method in the fields of intelligent driving and intelligent robot perception, and a key technology for the safety and obstacle avoidance of vehicles or robots in autonomous driving. Traditional driving area detection mainly achieves 2D driving area output in pixel coordinate system through image segmentation, which lacks depth information. Therefore, how to obtain 3D driving areas containing effective depth information is an important problem.

[0003] Sensors such as lidar or binocular cameras can acquire 3D information and depth distribution of the surrounding environment with relatively high accuracy, but their cost, installation conditions, and subsequent maintenance costs are all high, which raises the threshold for related solutions and reduces ease of use. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method and device for detecting three-dimensional drivable areas based on a monocular RGB camera, thereby solving the technical problems of high cost and low ease of use of the prior art.

[0005] To achieve the above objectives, the present invention is implemented using the following technical solution:

[0006] In a first aspect, the present invention provides a method for detecting three-dimensional drivable areas based on a monocular RGB camera, comprising:

[0007] Acquire the RGB image output by the monocular camera and perform distortion correction on the RGB image;

[0008] The BTS algorithm is used to estimate the depth of the RGB image after distortion correction to generate a corresponding depth map.

[0009] A depth map is constructed based on the depth map, and the optimal surface normal vector for each pixel in the depth map is calculated based on the depth map.

[0010] Based on the optimal surface normal vector of each pixel in the depth map, the SNE-RoadSeg algorithm is used to perform two-dimensional drivable region segmentation on the RGB image after distortion correction, generating a two-dimensional drivable region map.

[0011] Image processing is performed on the two-dimensional drivable area map to obtain a set of two-dimensional polygons of the two-dimensional drivable area;

[0012] Based on the set of two-dimensional polygons and the depth map, the two-dimensional polygons of the two-dimensional drivable area are transformed into three-dimensional polygons to obtain the three-dimensional drivable area.

[0013] Optionally, the depth map is a 16-bit depth map of the RGB image with the same resolution after corresponding distortion correction;

[0014] The step of constructing a depth map based on the depth map includes:

[0015] Using the pixel coordinates of each pixel in the depth map in the two-dimensional pixel coordinate system as an index, the pixel value of each pixel is divided by 256 to obtain its Z-axis coordinate value in the three-dimensional camera coordinate system.

[0016] Calculate the 3D coordinates of each pixel in the 3D camera coordinate system based on the pixel coordinates and Z-axis coordinates of each pixel:

[0017]

[0018] In the formula, p = [x, y] T Let P be the pixel coordinates of pixel p, where P = [X, Y, Z]. T Let p be the 3D coordinates of pixel p in the 3D camera coordinate system, and K be the camera intrinsic parameter matrix. x ,c y f represents the pixel coordinates of the center position of the depth map. x and f y It is the camera's focal length;

[0019] A depth map is constructed based on the three-dimensional coordinates of each pixel in the three-dimensional camera coordinate system.

[0020] Optionally, calculating the optimal surface normal vector for each pixel in the depth map based on the depth map includes:

[0021] Let the normal vector at point P in the depth map be (n x ,n y ,n z If ), then the equation of the normal vector is:

[0022] n x X+n y Y+n z Z+d=0

[0023] In the formula, d is a constant term;

[0024] Performing a coordinate transformation on the equation of the normal vector yields:

[0025]

[0026] Taking the partial derivatives with respect to x and y respectively, we can obtain:

[0027]

[0028]

[0029] The inverse depth image 1 / Z is approximated by convolving it with horizontal and vertical image gradient filters, respectively, for n. x n y After transformation, we get:

[0030] n x =-df x g x

[0031] n y =-df y g y

[0032] Get the nearest points of point P and generate the nearest point set N. p =[Q1,Q2,...Q k ], Q k Let k be the three-dimensional coordinates of the nearest point k;

[0033] For any Q i ∈N p Q i -P=(ΔX i ,ΔY i ,ΔZ i ), combined with n x n y Calculate its corresponding

[0034]

[0035] According to Q i The normal vector n calculated from point P and point P i for:

[0036]

[0037] According to the normal vector n i Let the normal vector (n) at point P be... x ,n y ,n z Transform into the optimal surface normal vector n in spherical coordinates best :

[0038]

[0039] In the formula, θ∈[0,π] is the tilt angle. The rotation angle;

[0040]

[0041]

[0042] In the formula, Let n be the normal vector i The normalized surface normal vector satisfies

[0043] Optionally, the two-dimensional drivable area map has the same resolution as the corresponding distortion-corrected RGB image and is represented as an 8-bit grayscale binary image, where pixel values ​​0 and 255 represent the background and drivable area, respectively.

[0044] Optionally, the step of performing image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons for the two-dimensional drivable area includes:

[0045] Obtain the set of connected components of the two-dimensional drivable region in the two-dimensional drivable region map, and determine the connected components in the set that are less than a preset threshold.

[0046] In the two-dimensional drivable area map, the pixel values ​​of connected components that are less than a preset threshold are changed from 255 to 0;

[0047] An image opening operation is performed on the modified two-dimensional drivable area map, and the contour set of the connected components in the two-dimensional drivable area map after the image opening operation is obtained.

[0048] A two-dimensional polygon set of a two-dimensional drivable region is constructed based on the point set of each contour in the contour set.

[0049] Optionally, converting the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons based on the two-dimensional polygon set and the depth map includes:

[0050] For each two-dimensional polygon in the set of two-dimensional polygons, using the pixel coordinates of each pixel point on the boundary of the two-dimensional polygon as an index, the Z-axis coordinate value corresponding to the matcher on the depth map is used to generate a matching pair ([x′,y′)). T (x′, y′) are the pixel coordinates of the pixel points on the boundary of the two-dimensional polygon, and Z′ is the corresponding Z-axis coordinate value of (x′, y′).

[0051] Based on the matching pair ([x′,y′]) T Generate the 3D coordinates gp of the pixels on the boundary of the 2D polygon in the 3D camera coordinate system. 3d :

[0052]

[0053] Based on the three-dimensional coordinates gp 3d Construct a 3D polygon.

[0054] Secondly, the present invention provides a three-dimensional drivable area detection device based on a monocular RGB camera, the detection device comprising:

[0055] An image preprocessing module is used to acquire the RGB image output by a monocular camera and perform distortion correction on the RGB image;

[0056] The depth estimation module is used to perform depth estimation on the distortion-corrected RGB image using the BTS algorithm to generate a corresponding depth map.

[0057] The normal vector calculation module is used to construct a depth map based on the depth map, and to calculate the optimal surface normal vector for each pixel in the depth map based on the depth map.

[0058] The region segmentation module is used to perform two-dimensional drivable region segmentation on the distortion-corrected RGB image using the SNE-RoadSeg algorithm based on the optimal surface normal vector of each pixel in the depth map, and generate a two-dimensional drivable region map.

[0059] A two-dimensional polygon generation module is used to perform image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons of the two-dimensional drivable area.

[0060] The three-dimensional polygon generation module is used to convert the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons based on the two-dimensional polygon set and the depth map, so as to obtain the three-dimensional drivable area.

[0061] Thirdly, the present invention provides an electronic device, including a processor and a storage medium;

[0062] The storage medium is used to store instructions;

[0063] The processor is configured to operate according to the instructions to perform the steps according to the method described above.

[0064] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the above-described method.

[0065] Compared with the prior art, the beneficial effects achieved by the present invention are as follows:

[0066] This invention provides a method and apparatus for detecting 3D drivable areas based on a monocular RGB camera. It introduces monocular depth estimation to output a dense depth map and employs a staged decoupling approach to fuse depth information with the 2D drivable area. Using only a monocular RGB camera, it can directly output the 3D drivable area, thereby significantly improving the environmental perception capabilities of intelligent vehicles or robots in 3D space. Furthermore, this invention does not require additional sensors; it can achieve 3D drivable area output using only a monocular RGB camera, reducing costs while improving ease of use. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the three-dimensional drivable area detection method based on a monocular RGB camera provided in an embodiment of the present invention.

[0068] Figure 2 This is a schematic diagram of the structure of a three-dimensional drivable area detection device based on a monocular RGB camera provided in an embodiment of the present invention. Detailed Implementation

[0069] The present invention will be further described below with reference to the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and should not be used to limit the scope of protection of the present invention.

[0070] Example 1:

[0071] like Figure 1 As shown, this embodiment of the invention provides a method for detecting three-dimensional drivable areas based on a monocular RGB camera, including the following steps:

[0072] Step S1: Acquire the RGB image output by the monocular camera and perform distortion correction on the RGB image;

[0073] Regarding image distortion, due to various installation limitations, the camera may not be able to be installed perpendicular to the target being measured. This will cause perspective distortion in the generated image, which will greatly affect the accuracy of subsequent image processing. Therefore, distortion correction of the acquired image is necessary. The main steps are as follows: 1. Establish ideal coordinates; 2. Coordinate normalization; 3. Distortion removal; 4. Denormalization. The corrected image can then be obtained by interpolating the obtained coordinates with the original image. For details, please refer to HELP in the Camera Calibration Toolbox in MATLAB. This is a relatively mature technology and will not be elaborated upon here.

[0074] Step S2: Use the BTS algorithm to perform depth estimation on the RGB image after distortion correction to generate a corresponding depth map;

[0075] Firstly, the depth map is a 16-bit depth map of the RGB image with the same resolution after corresponding distortion correction.

[0076] Secondly, the construction of a depth map based on the depth map includes:

[0077] Step (1) Using the pixel coordinates of each pixel in the depth map in the two-dimensional pixel coordinate system as an index, divide the pixel value of each pixel by 256 to obtain its Z-axis coordinate value in the three-dimensional camera coordinate system; the Z-axis coordinate value is the depth value, and the unit is meters.

[0078] Step (2) Calculate the 3D coordinates of each pixel in the 3D camera coordinate system based on the pixel coordinates and Z-axis coordinates of each pixel:

[0079]

[0080] In the formula, p = [x, y] T Let P be the pixel coordinates of pixel p, where P = [X, Y, Z]. T Let p be the 3D coordinates of pixel p in the 3D camera coordinate system, and K be the camera intrinsic parameter matrix. x ,c y f represents the pixel coordinates of the center position of the depth map. x and f y It is the camera's focal length;

[0081] Step (3) Construct a depth map based on the three-dimensional coordinates of each pixel in the three-dimensional camera coordinate system.

[0082] Thirdly, regarding the BTS algorithm, it follows an encoder-decoder scheme that reduces the feature map resolution to H / 8 and then restores it to the original resolution H. After a dense feature extractor that generates the H / 8 feature map, a denser spatial pyramid pooling layer is used as a context information extractor with various dilation rates r∈{3,6,12,18,24}. Then, at each stage of the decoding phase, when the internal output is restored to full resolution by a factor of 2, the proposed Local Plane Guided (LPG) layer is used to more effectively correlate the features with the desired depth estimate. Finally, the outputs of the proposed layers are concatenated into the final convolutional layer to obtain the depth estimate. This is a relatively mature technology, so I won't go into too much detail here.

[0083] Step S3: Construct a depth map based on the depth map, and calculate the optimal surface normal vector for each pixel in the depth map based on the depth map; the specific process is as follows:

[0084] Step S3.1: Let the normal vector at point P in the depth map be (n x ,n y,n z If ), then the equation of the normal vector is:

[0085] n x X+n y Y+n z Z+d=0

[0086] In the formula, d is a constant term;

[0087] Step S3.2: Perform coordinate transformation on the normal vector equation to obtain:

[0088]

[0089] Step S3.3, taking the partial derivatives with respect to x and y respectively, yields:

[0090]

[0091]

[0092] Step S3.4: Approximate the inverse depth image 1 / Z by convolving it with the horizontal and vertical image gradient filters respectively, for n x n y After transformation, we get:

[0093] n x =-df x g x

[0094] n y =-df y g y

[0095] Step S3.5: Obtain the nearest points of point P and generate the nearest point set N. p =[Q1,Q2,...Q k ], Q k Let k be the three-dimensional coordinates of the nearest point k;

[0096] Step S3.6: For any Q i ∈N p Q i -P=(ΔX i ,ΔY i ,ΔZ i ), combined with n x n y Calculate its corresponding

[0097]

[0098] Step S3.7, according to Q iThe normal vector n calculated from point P and point P i for:

[0099]

[0100] Since any normalized surface normal can be projected onto a sphere centered at (0,0,0) with a radius of 1, the optimal surface normal vector at point P can also be projected onto some location on the same sphere.

[0101] Step S3.8: Based on the normal vector n i Let the normal vector (n) at point P be... x ,n y ,n z Transform into the optimal surface normal vector n in spherical coordinates best :

[0102]

[0103] In the formula, θ∈[0,π] is the tilt angle. The rotation angle;

[0104]

[0105]

[0106] In the formula, Let n be the normal vector i The normalized surface normal vector satisfies Assume n best If the angle between point P and any normalized surface normal vector is less than π / 2, then the optimal surface normal vector n is... best Definitely satisfy Thus, the tilt angle θ can be solved.

[0107] Step S4: Based on the optimal surface normal vector of each pixel in the depth map, the SNE-RoadSeg algorithm is used to perform two-dimensional drivable region segmentation on the RGB image after distortion correction, generating a two-dimensional drivable region map.

[0108] Since the depth map and the RGB image have the same resolution, the positions of the pixels also correspond one-to-one. The 2D drivable region segmentation of the RGB image can be performed using the optimal surface normal vector of each pixel in the depth map. The 2D drivable region map has the same resolution as the corresponding distortion-corrected RGB image and is represented as an 8-bit grayscale binary image, where pixel values ​​0 and 255 represent the background and drivable region, respectively.

[0109] The SNE-RoadSeg algorithm, a road segmentation network incorporating surface normal vectors, still employs an encoder-decoder structure. Unlike UNet, it borrows skip connections from DenseNet to achieve more flexible feature fusion in the decoder. Furthermore, it considers the forced aggregation of feature maps at the same scale in both the encoder and decoder via skip connections to be an unnecessary constraint. The extracted RGB and surface normal feature maps are fused hierarchically through element-wise summation. Then, the fused feature maps are fused again in the decoder using DenseNet's skip connections to restore the feature map resolution. Finally, RoadSeg uses a sigmoid layer to generate a probabilistic map for semantic driving scene segmentation; this is a relatively mature technique and will not be elaborated upon here.

[0110] Step S5: Perform image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons for the two-dimensional drivable area; specifically including the following steps:

[0111] Step S5.1: Obtain the set of connected components of the two-dimensional drivable region in the two-dimensional drivable region map, and determine the connected components in the set that are less than a preset threshold.

[0112] Step S5.2: In the two-dimensional drivable area map, the pixel values ​​of connected components that are less than a preset threshold are changed from 255 to 0;

[0113] Step S5.3: Perform image opening operation on the modified two-dimensional drivable area map, and obtain the contour set of connected components in the two-dimensional drivable area map after image opening operation;

[0114] Step S5.4: Construct a two-dimensional polygon set of the two-dimensional drivable region based on the point set of each contour in the contour set.

[0115] Step S6: Based on the set of two-dimensional polygons and the depth map, convert the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons to obtain the three-dimensional drivable area; specifically including the following steps:

[0116] Step S6.1: For each two-dimensional polygon in the set of two-dimensional polygons, using the pixel coordinates of each pixel point on the boundary of the two-dimensional polygon as an index, match the corresponding Z-axis coordinates on the depth map, and generate a matching pair ([x′,y′]). T (x′, y′) are the pixel coordinates of the pixel points on the boundary of the two-dimensional polygon, and Z′ is the corresponding Z-axis coordinate value of (x′, y′).

[0117] Step S6.1: Based on the matching pair ([x′,y′] TGenerate the 3D coordinates gp of the pixels on the boundary of the 2D polygon in the 3D camera coordinate system. 3d :

[0118]

[0119] Based on the three-dimensional coordinates gp 3d Construct a 3D polygon; the interior of the 3D polygon is the drivable area.

[0120] Compared with existing traditional 2D drivable area detection methods, the method provided in this embodiment adds extra depth information by outputting the 3D coordinates of the drivable area in the 3D camera coordinate system, so that the drivable area can be directly applied to real-world scenarios. Secondly, this method does not require additional sensors; it can output the 3D drivable area using only a monocular RGB camera, reducing costs while improving ease of use. In addition, this method uses a decoupling approach to output the monocular depth estimation results and the 2D drivable area segmentation results separately, which makes the implementation process more controllable and can be adjusted according to different real-world scenarios.

[0121] Example 2:

[0122] like Figure 2 As shown, this embodiment of the invention provides a three-dimensional drivable area detection device based on a monocular RGB camera, the detection device comprising:

[0123] An image preprocessing module is used to acquire the RGB image output by a monocular camera and perform distortion correction on the RGB image;

[0124] The depth estimation module is used to perform depth estimation on the distortion-corrected RGB image using the BTS algorithm to generate a corresponding depth map.

[0125] The normal vector calculation module is used to construct a depth map based on the depth map, and to calculate the optimal surface normal vector for each pixel in the depth map based on the depth map.

[0126] The region segmentation module is used to perform two-dimensional drivable region segmentation on the distortion-corrected RGB image using the SNE-RoadSeg algorithm based on the optimal surface normal vector of each pixel in the depth map, and generate a two-dimensional drivable region map.

[0127] A two-dimensional polygon generation module is used to perform image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons of the two-dimensional drivable area.

[0128] The three-dimensional polygon generation module is used to convert the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons based on the two-dimensional polygon set and the depth map, so as to obtain the three-dimensional drivable area.

[0129] Example 3:

[0130] This invention provides an electronic device, including a processor and a storage medium;

[0131] The storage medium is used to store instructions;

[0132] The processor is configured to operate according to the instructions to perform the steps of the method according to Embodiment 1 above.

[0133] Example 4:

[0134] This invention provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the method described in Embodiment 1 above.

[0135] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0136] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0137] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0138] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0139] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for detecting three-dimensional drivable areas based on a monocular RGB camera, characterized in that, include: Acquire the RGB image output by the monocular camera and perform distortion correction on the RGB image; The BTS algorithm is used to estimate the depth of the RGB image after distortion correction to generate a corresponding depth map. A depth map is constructed based on the depth map, and the optimal surface normal vector for each pixel in the depth map is calculated based on the depth map. Based on the optimal surface normal vector of each pixel in the depth map, the SNE-RoadSeg algorithm is used to perform two-dimensional drivable region segmentation on the RGB image after distortion correction, generating a two-dimensional drivable region map. Image processing is performed on the two-dimensional drivable area map to obtain a set of two-dimensional polygons of the two-dimensional drivable area; Based on the set of two-dimensional polygons and the depth map, the two-dimensional polygons of the two-dimensional drivable area are transformed into three-dimensional polygons to obtain the three-dimensional drivable area. Wherein, the depth map is a 16-bit depth map of the RGB image with the same resolution after corresponding distortion correction; The step of constructing a depth map based on the depth map includes: Using the pixel coordinates of each pixel in the depth map in the two-dimensional pixel coordinate system as an index, the pixel value of each pixel is divided by 256 to obtain its Z-axis coordinate value in the three-dimensional camera coordinate system. Calculate the 3D coordinates of each pixel in the 3D camera coordinate system based on the pixel coordinates and Z-axis coordinates of each pixel: ； In the formula, For pixels pixel coordinates, For pixels 3D coordinates in the 3D camera coordinate system For the camera intrinsic parameter matrix, These are the pixel coordinates of the center position of the depth map. and It is the camera's focal length; Construct a depth map based on the three-dimensional coordinates of each pixel in the three-dimensional camera coordinate system; The step of performing image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons for the two-dimensional drivable area includes: Obtain the set of connected components of the two-dimensional drivable region in the two-dimensional drivable region map, and determine the connected components in the set that are less than a preset threshold. In the two-dimensional drivable area map, the pixel values ​​of connected components that are less than a preset threshold are changed from 255 to 0; An image opening operation is performed on the modified two-dimensional drivable area map, and the contour set of the connected components in the two-dimensional drivable area map after the image opening operation is obtained. Construct a two-dimensional polygon set of a two-dimensional drivable region based on the point set of each contour in the contour set; The step of converting the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons based on the two-dimensional polygon set and the depth map includes: For each two-dimensional polygon in the set of two-dimensional polygons, using the pixel coordinates of each pixel point on the boundary of the two-dimensional polygon as an index, the Z-axis coordinate value corresponding to the matcher on the depth map is used to generate a matching pair. , The pixel coordinates of the pixels on the boundary of the two-dimensional polygon. for The corresponding Z-axis coordinate value; According to the matching pair Generate the 3D coordinates of the pixels on the boundary of the 2D polygon in the 3D camera coordinate system. : ； Based on three-dimensional coordinates Construct a 3D polygon.

2. The method for detecting three-dimensional drivable areas based on a monocular RGB camera according to claim 1, characterized in that, The step of calculating the optimal surface normal vector for each pixel in the depth map based on the depth map includes: Let the depth map be in The normal vector at point is Then the equation of the normal vector is: ； In the formula, For constant terms; Performing a coordinate transformation on the equation of the normal vector yields: ； To each Taking the partial derivative with y, we can obtain: ； ； Apply horizontal and vertical image gradient filters to the inverse depth image respectively. Perform convolution to approximate each, for , After transformation, we get: ； ； Get A set of neighboring points is generated from the neighboring points of a point. , For nearby points 3D coordinates; For any , , combined , Calculate its corresponding : ； In the formula, According to Dot and Normal vector calculated at a point The Z-axis is placed towards the vertical component; according to Dot and Normal vector calculated at a point for: ； According to the normal vector Will Normal vector at point Transform into the optimal surface normal vector in spherical coordinates : ； In the formula, The angle of inclination. The rotation angle; ； ； In the formula, Normal vector The normalized surface normal vector satisfies .

3. The method for detecting three-dimensional drivable areas based on a monocular RGB camera according to claim 2, characterized in that, The two-dimensional drivable area map has the same resolution as the corresponding distortion-corrected RGB image and is represented by an 8-bit grayscale binary image, where pixel values ​​0 and 255 represent the background and drivable area, respectively.

4. A three-dimensional drivable area detection device based on a monocular RGB camera, characterized in that, The apparatus is configured to perform the steps of the method as described in any one of claims 1-3, the detection apparatus comprising: An image preprocessing module is used to acquire the RGB image output by a monocular camera and perform distortion correction on the RGB image; The depth estimation module is used to perform depth estimation on the distortion-corrected RGB image using the BTS algorithm to generate a corresponding depth map. The normal vector calculation module is used to construct a depth map based on the depth map, and to calculate the optimal surface normal vector for each pixel in the depth map based on the depth map. The region segmentation module is used to perform two-dimensional drivable region segmentation on the distortion-corrected RGB image using the SNE-RoadSeg algorithm based on the optimal surface normal vector of each pixel in the depth map, and generate a two-dimensional drivable region map. A two-dimensional polygon generation module is used to perform image processing on the two-dimensional drivable area map to obtain a set of two-dimensional polygons of the two-dimensional drivable area. The three-dimensional polygon generation module is used to convert the two-dimensional polygons of the two-dimensional drivable area into three-dimensional polygons based on the two-dimensional polygon set and the depth map, so as to obtain the three-dimensional drivable area.

5. An electronic device, characterized in that, Including processor and storage media; The storage medium is used to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-3.

6. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by a processor, the program implements the steps of the method according to any one of claims 1-3.

Images