A 3D drivable area detection method based on 3D semantic scene completion
Through the 3D semantic scene completion method of monocular RGB camera, the problem of lack of 3D information in traditional detection is solved, and low-cost and high-usability 3D feasible area detection is realized, which is suitable for intelligent driving and intelligent robot fields.
Patent Information
- Application Number
- CN202411082396.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-08
- Publication Date
- 2024-12-06
- Estimated Expiration
- 2044-08-08
AI Technical Summary
Traditional travelable area detection mainly realizes 2D output through image segmentation, lacks 3D information in real space, and the high-cost lidar sensor increases the threshold of the solution and reduces ease of use.
Using the 3D semantic scene completion method based on monocular RGB camera, the 3D travelable area is output through distortion correction, monocular depth estimation, 3D pseudo-point cloud construction, voxelization and 3D semantic scene completion, reducing costs and improving ease Usability.
It realizes the output of 3D coordinates of the travelable area under the three-dimensional camera coordinate system, which increases spatial information, reduces costs, improves ease of use, and dynamically adjusts traffic through semantic categories, with high generalization capabilities.
Smart Images

Figure CN118609082B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to intelligent driving and intelligent robot perception technology, and in particular to a 3D drivable area detection method based on 3D semantic scene completion. Background Art
[0002] Drivable area detection is an important method in the field of intelligent driving and intelligent robot perception, and is also a key technology for vehicle or robot autonomous driving in terms of safety and obstacle avoidance. Traditional drivable area detection mainly uses image segmentation to output 2D drivable area in pixel coordinate system, lacking 3D information in real space. Therefore, how to obtain 3D drivable area is an important issue.
[0003] Sensors such as LiDAR can obtain 3D information of the surrounding environment with high accuracy, but their cost, installation conditions, and subsequent maintenance costs are high, which raises the threshold of related solutions and reduces ease of use. Summary of the invention
[0004] Purpose of the invention: The purpose of the present invention is to provide a 3D drivable area detection method based on 3D semantic scene completion. This method can output 3D position information using only a monocular RGB camera, which is lower in cost and easier to use.
[0005] Technical solution: A 3D drivable area detection method based on 3D semantic scene completion of the present invention comprises the following steps:
[0006] Get the RGB image output by the monocular camera and perform distortion correction on it;
[0007] Perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map, and construct a depth dictionary using pixel coordinates as indexes;
[0008] According to the camera intrinsic parameter matrix and the depth dictionary, each pixel of the depth map is projected from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system to construct a 3D pseudo point cloud.
[0009] voxelize the 3D pseudo point cloud, obtain the 3D occupied voxels of the set spatial range and voxel resolution, and obtain the 3D occupied voxel set;
[0010] Performing 3D semantic scene completion on the 3D occupied voxels in the 3D occupied voxel set, outputting 3D semantic voxels containing semantic categories, and obtaining a 3D semantic voxel set;
[0011] Combined with the semantic categories of the 3D semantic voxels in the 3D semantic voxel set and the 3D coordinates of the center point, it is determined whether the voxel area to be detected is passable, and the drivable area in the entire 3D space is obtained.
[0012] Furthermore, HELP in the camera correction toolbox in Matlab is used to correct the distortion of the RGB image.
[0013] Furthermore, the monocular depth estimation is performed on the distortion-corrected RGB image to obtain a depth map, and a depth dictionary is constructed with pixel coordinates as indexes; specifically:
[0014] Based on the BTS algorithm, the monocular depth estimation is performed on the distortion-corrected RGB image, and a 16-bit depth map with the same resolution as the distortion-corrected RGB image is output. The depth dictionary is constructed with the pixel coordinates of the 16-bit depth map as the index; the depth dictionary stores data in the form of key:value, where the key is the two-dimensional pixel coordinate of each pixel in the 16-bit depth map, and the value is the pixel value at the corresponding coordinate divided by 256, that is, the value is the pixel value in the three-dimensional camera coordinate system. Axis coordinate values.
[0015] Furthermore, each pixel in the depth map is projected from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system to construct a 3D pseudo point cloud; specifically:
[0016] 3D point in the 3D camera coordinate system and 2D pixel points in a 2D pixel coordinate system The following conversion relationship exists:
[0017] ;
[0018] in, is the camera intrinsic parameter matrix, is the 16-bit depth image center; and is the camera focal length; , , are 3D points in the 3D camera coordinate system of X axis, Y Axis and Z Coordinate values in the axis direction; , They are 2D pixel points in the 2D pixel coordinate system of X Axis and Y Coordinate values in the axis direction;
[0019] The above conversion is performed on each pixel in the 16-bit depth image to obtain the 3D point corresponding to each pixel, thereby generating a 3D pseudo point cloud corresponding to the 16-bit depth image.
[0020] Furthermore, the 3D pseudo point cloud is voxelized, the spatial range and voxel resolution are set in the 3D camera coordinate system, and the 3D occupied voxel set is obtained; specifically:
[0021] (1) Get all 3D point sets in the 3D pseudo point cloud , and They are 3D point sets The first and second 3D points in ;
[0022] (2) Set the voxel resolution to , set the space range to front and back ,about ,up and down , defining the 3D voxel space , its length, width and height are:
[0023] ;
[0024] ;
[0025] ;
[0026] in, The three-dimensional voxel space Length, The three-dimensional voxel space The width of The three-dimensional voxel space Height, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system Minimum value in the axis direction;
[0027] (3) Based on the 3D point set , get the voxel index set corresponding to all 3D points for:
[0028] ;
[0029] in, For the first voxel in length The index value on For the first voxel in width The index value on For the first voxel at height The index value on For the second voxel at length The index value on For the second voxel in width The index value on For the second voxel at height The index value on ;
[0030] 3D Point Collection Middle 3D points The corresponding voxel index The calculation formula is: ;
[0031] ;
[0032] ;
[0033] in, To calculate the rounding function, For the Voxel length The index value on For the Voxel width The index value on For the Voxel at height The index value on ; , , Respectively 3D points in the 3D camera coordinate system axis, axis, The coordinate value in the axis direction, T represents transpose;
[0034] (4) Based on the voxel index set , determine whether the voxel is within the set space range;
[0035] The first 3D points The corresponding voxel index If satisfied:
[0036] ;
[0037] in, is logical AND;
[0038] The first The voxels corresponding to the 3D points are located within the set spatial range;
[0039] right All indexes in the , output voxel index set within the set space range for:
[0040] ;
[0041] in, To set the length of the first voxel in the space range The index value on To set the width of the first voxel in the spatial range The index value on To set the first voxel in the space range at height The index value on To set the length of the second voxel in the spatial range The index value on To set the width of the second voxel in the spatial range The index value on To set the second voxel in the space range at height The index value on ;
[0042] (5) Construct a 3D voxel set in the form of a three-dimensional matrix , the dimension is , the initial value is 0; traverse the voxel index set within the set space range ,according to The index in the 3D voxel set The voxel in is changed from the initial value 0 to 1, which means that the voxel is occupied; after traversal and modification, the 3D occupied voxel set is output .
[0043] Furthermore, 3D semantic scene completion is performed on the 3D occupied voxels in the 3D occupied voxel set, and 3D semantic voxels containing semantic categories are output to obtain a 3D semantic voxel set; specifically:
[0044] For a set of 3D occupied voxels , LMSCNet algorithm is used to complete 3D semantic scenes and output a set of 3D semantic voxels containing semantic categories , and Same dimension, The value of each element is Any integer in the range is the total number of semantic categories, and 0 is the background category.
[0045] Furthermore, the semantic categories of the 3D semantic voxels in the 3D semantic voxel set and the 3D coordinates of the center point are combined to determine whether the voxel area to be detected is passable, and the drivable area in the entire 3D space is obtained; specifically:
[0046] (1) Based on the 3D semantic voxel set , get The set of 3D coordinates corresponding to the center points of all voxels in for:
[0047] ;
[0048] in, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, T represents transpose;
[0049] 3D semantic voxel collection Middle The voxel index set within the set space corresponds to the voxel Middle Voxel Index , the 3D coordinates of the voxel center are: ;
[0050] ;
[0051] ;
[0052] in, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, is the voxel resolution; To set the space range Voxel length The index value on To set the space range Voxel width The index value on To set the space range Voxel at height The index value on ;
[0053] According to the 3D semantic voxel set , get the semantic category set corresponding to all voxels ;
[0054] in, is a set of 3D semantic voxels The category corresponding to the first voxel in is a set of 3D semantic voxels The category corresponding to the second voxel in is a set of 3D semantic voxels Middle The category to which the voxels correspond;
[0055] (3) According to the 3D coordinate set corresponding to the center points of all voxels and the set of semantic categories , and According to the one-to-one correspondence of the index, determine whether the voxel area corresponding to the 3D coordinates and semantic category is passable;
[0056] 3D semantic voxel collection Middle The voxel area of a voxel is:
[0057] Front and rear: ;
[0058] about: ;
[0059] up and down: ;
[0060] If the semantic category corresponding to the voxel is not an obstacle, the voxel area is passable; the above judgment is performed on all voxels, and the drivable area within the entire space is output.
[0061] Based on the same inventive concept, in one embodiment of the present invention, a 3D drivable area detection system based on 3D semantic scene completion includes:
[0062] The data processing module is used to obtain the RGB image output by the monocular camera and perform distortion correction on it;
[0063] The monocular depth estimation module is used to perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map and construct a depth dictionary using pixel coordinates as indexes.
[0064] The 3D pseudo point cloud generation module is used to project each pixel of the depth map from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system according to the camera intrinsic parameter matrix and the depth dictionary to construct a 3D pseudo point cloud;
[0065] A 3D voxel generation module is used to voxelize the 3D pseudo point cloud, obtain 3D occupied voxels of a set spatial range and voxel resolution, and obtain a 3D occupied voxel set;
[0066] A 3D semantic scene completion module is used to complete the 3D semantic scene for the 3D occupied voxels in the 3D occupied voxel set, output the 3D semantic voxels containing the semantic category, and obtain the 3D semantic voxel set;
[0067] The passability judgment module is used to combine the semantic category and 3D position information of the 3D semantic voxels in the 3D semantic voxel set to judge whether the voxel area to be detected is passable and obtain the drivable area in the entire 3D space.
[0068] Based on the same inventive concept, in one embodiment of the present invention, an electronic device includes:
[0069] A memory storing executable program code;
[0070] a processor coupled to the memory;
[0071] The processor calls the executable program code stored in the memory to execute the steps of the above-mentioned 3D drivable area detection method based on 3D semantic scene completion.
[0072] Based on the same inventive concept, in one embodiment of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and when the computer instructions are called, the steps of the 3D drivable area detection method based on 3D semantic scene completion are executed as described above.
[0073] Beneficial effects: Compared with the prior art, the advantages of the present invention are as follows: additional spatial information is added by outputting the 3D coordinates of the drivable area in the three-dimensional camera coordinate system, so that the drivable area can be directly applied to actual scenes; secondly, there is no need to add additional sensors, and the output of the 3D drivable area can be achieved using only a monocular RGB camera, which reduces costs while improving ease of use; in addition, by outputting the semantic categories of each sub-area in the 3D space, the passability of each sub-area can be dynamically adjusted according to the semantic categories in different application scenarios, and it has a high generalization ability. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] Figure 1 It is a flow chart of the method of the present invention. DETAILED DESCRIPTION
[0075] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments.
[0076] like Figure 1 As shown, a 3D drivable area detection method based on 3D semantic scene completion of the present invention comprises the following steps:
[0077] Step 1: Get the RGB image output by the monocular camera and perform distortion correction on it.
[0078] Regarding image distortion, due to various installation conditions, the camera may not be installed perpendicular to the target, which will cause perspective distortion in the generated image, which will have a great impact on the accuracy of subsequent image processing. Therefore, the acquired image needs to be distorted. There are mainly the following steps: 1. Establish ideal coordinates, 2. Normalize coordinates, 3. Remove distortion, 4. Denormalize, and then interpolate the obtained coordinates to get the corrected image. For details, see HELP in the camera calibration toolbox in matlab for distortion correction; it is a relatively mature technology and will not be described in detail here.
[0079] Step 2: Perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map, and construct a depth dictionary using pixel coordinates as indexes; specifically:
[0080] Based on the BTS algorithm (From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation), the monocular depth estimation is performed on the RGB image after distortion correction in step 1, and a 16-bit depth map with the same resolution as the RGB image after distortion correction is output. The depth dictionary is constructed with the pixel coordinates of the 16-bit depth map as the index. The depth dictionary stores data in the form of key:value, where the key is the two-dimensional pixel coordinate of each pixel in the 16-bit depth map, and the value is the pixel value at the corresponding coordinate divided by 256 (that is, the pixel value in the three-dimensional camera coordinate system). axis coordinate values in meters).
[0081] Regarding the BTS algorithm, the BTS algorithm follows an encoding-decoding scheme that reduces the feature map resolution to H / 8 and then restores it to the original resolution H. After the dense feature extractor that generates the feature map of H / 8, a denser spatial pyramid pooling layer is used as the kernel with various dilation rates. Then, at each stage of the decoding phase, when the internal output is restored to full resolution by a factor of 2, the proposed local plane guidance (LPG) layer is used to more effectively link the features to the desired depth estimate. Finally, the outputs of the proposed layers are concatenated into the last convolutional layer to obtain the depth estimate , which is a relatively mature technology and will not be elaborated on here.
[0082] Step 3: According to the camera intrinsic parameter matrix, each pixel point of the depth map in the depth dictionary is projected from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system to construct a 3D pseudo point cloud; specifically:
[0083] For the 16-bit depth image obtained in step 2, the 3D point in the 3D camera coordinate system and 2D pixel points in a 2D pixel coordinate system The following conversion relationship exists:
[0084] (1);
[0085] in, is the camera intrinsic parameter matrix, is the 16-bit depth image center; and is the camera focal length (in pixels); , , are 3D points in the 3D camera coordinate system of X axis, Y Axis and Z Coordinate values in the axis direction; , They are 2D pixel points in the 2D pixel coordinate system of X Axis and Y Coordinate value in the axis direction.
[0086] The above conversion is performed on each pixel in the 16-bit depth image to obtain the 3D point corresponding to each pixel, thereby generating a 3D pseudo point cloud corresponding to the 16-bit depth image.
[0087] Step 4: voxelize the 3D pseudo point cloud to obtain a 3D occupied voxel set with a set spatial range and voxel resolution.
[0088] (4.1) Get all 3D point sets in the 3D pseudo point cloud , and They are 3D point sets The first and second 3D points in .
[0089] (4.2) Set the voxel resolution to (i.e. the length of the voxel side corresponds to the length in real space, in meters); set the space range to front and back ,about ,up and down (That is, the distance range of the voxel space corresponding to the real space, the unit is meter). Define the three-dimensional voxel space , whose length, width and height are:
[0090] (2);
[0091] (3);
[0092] (4);
[0093] in, The three-dimensional voxel space Length, The three-dimensional voxel space The width of The three-dimensional voxel space Height, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system Minimum value along the axis.
[0094] (4.3) According to the 3D point set , get the voxel index set corresponding to all 3D points for:
[0095] (5);
[0096] in, For the first voxel in length The index value on For the first voxel in width The index value on For the first voxel at height The index value on For the second voxel at length The index value on For the second voxel in width The index value on For the second voxel at height The index value on .
[0097] A collection of 3D points Middle 3D points For example, represents the transpose; its corresponding voxel index They are:
[0098] (6);
[0099] (7);
[0100] (8);
[0101] in, To calculate the rounding function, For the Voxel length The index value on For the Voxel width The index value on For the Voxel at height The index value on ; , , They are 3D point sets Middle 3D points of X axis, Y Axis and Z Coordinate value in the axis direction.
[0102] (4.4) According to the voxel index set , determine whether the voxel is within the set space range;
[0103] Take the first 3D points The corresponding voxel index For example, if:
[0104] (9);
[0105] in, is logical AND;
[0106] The voxel corresponding to the point is within the set spatial range.
[0107] right All indexes in the , and output the voxel index set within the set space range for:
[0108] (10);
[0109] in, To set the length of the first voxel in the space range The index value on To set the width of the first voxel in the spatial range The index value on To set the first voxel in the space range at height The index value on To set the length of the second voxel in the spatial range The index value on To set the width of the second voxel in the spatial range The index value on To set the second voxel in the space range at height The index value on ;
[0110] (4.5) Construct a 3D voxel set in the form of a three-dimensional matrix , the dimension is , the initial value is 0; traverse the voxel index set within the set space range ,according to The index in the 3D voxel set The voxel in is changed from the initial value 0 to 1, which means that the voxel is occupied. After traversing and modifying, the 3D occupied voxel set is output .
[0111] Step 5: perform 3D semantic scene completion on the 3D occupied voxels in the 3D occupied voxel set, output 3D semantic voxels containing semantic categories, and obtain a 3D semantic voxel set; specifically:
[0112] For the 3D occupied voxel set obtained in step 4 , the LMSCNet algorithm (LMSCNet: LightweightMultiscale 3D Semantic Completion) is used to complete 3D semantic scenes and output a set of 3D semantic voxels containing semantic categories (and Same dimension, that is ), The value of each element is Any integer in the range is the total number of semantic categories (i.e., the algorithm supports The judgment of each category, 0 is the background category).
[0113] Regarding the LMSCNet algorithm, the LMSCNet algorithm takes 3D occupied voxels as input, and first uses a multi-scale feature extraction module and a context aggregation module. The multi-scale feature extraction module uses multiple convolutional layers and pooling layers to extract features at different scales, so that different levels of information can be captured, from local details to global context; the context aggregation module uses multi-scale context information for fusion and aggregation to enhance semantic understanding. Then, the decoder module is used for step-by-step decoding and upsampling, restored to the original resolution, and generates semantic labels and completed 3D scene representations, namely 3D semantic voxels, which are relatively mature technologies and will not be described in detail here.
[0114] Step 6: Combine the semantic categories of the 3D semantic voxels in the 3D semantic voxel set and the 3D coordinates of the center point to determine whether the voxel area to be detected is passable, and obtain the drivable area in the entire 3D space; specifically:
[0115] (6.1) According to the 3D semantic voxel set , get The set of 3D coordinates corresponding to the center points of all voxels in for:
[0116] (11);
[0117] in, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values; Indicates transpose.
[0118] 3D semantic voxel collection Middle Taking a voxel as an example, it corresponds to the voxel index set within the set space range. Middle Voxel Index , the 3D coordinates of the voxel center point are:
[0119] (12);
[0120] (13);
[0121] (14);
[0122] in, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, To set the space range Voxel length The index value on To set the space range Voxel width The index value on To set the space range Voxel at height The index value on .
[0123] (6.2) According to the 3D semantic voxel set , get the semantic category set corresponding to all voxels ;
[0124] in, is a set of 3D semantic voxels The category corresponding to the first voxel in is a set of 3D semantic voxels The category corresponding to the second voxel in is a set of 3D semantic voxels Middle The class to which the voxel corresponds.
[0125] (6.3) According to the 3D coordinate set corresponding to the center points of all voxels and the set of semantic categories , and According to the one-to-one correspondence between the indices, it is determined whether the voxel area corresponding to the 3D coordinates and the semantic category is passable.
[0126] 3D semantic voxel collection Middle Taking a voxel as an example, its voxel area is (in meters):
[0127] Front and rear: (15);
[0128] about: (16);
[0129] up and down: (17);
[0130] If the semantic category corresponding to the voxel is not an obstacle, the voxel area is passable. The above judgment is performed on all voxels, and the drivable area within the entire space is output.
[0131] The present invention first uses the monocular depth estimation module to output the depth map corresponding to the monocular RGB image, and constructs the depth map using the pixel coordinates as the index. ; Secondly, the 3D pseudo point cloud generation module is based on depth , output 3D pseudo point cloud; then, the 3D voxel generation module voxelizes the 3D pseudo point cloud and generates a 3D occupied voxel set based on the set spatial range and resolution; then, the 3D semantic scene completion module takes the 3D occupied voxels as input, outputs the semantic information corresponding to the 3D occupied voxels, and obtains the 3D semantic voxels; finally, the passability judgment module uses the 3D position and semantic category contained in the 3D semantic voxels as the basis to judge whether the area where the voxels are located is passable, thereby obtaining the drivable area in the entire 3D space. This method can directly output the drivable area in the three-dimensional camera coordinate system when only using a monocular RGB camera, which is both practical and easy to use.
[0132] The present invention introduces monocular depth estimation to output a dense depth map, and adopts a 3D semantic scene completion method to output the semantic category of each position in 3D space, and designs a drivable area detection method with the ability to output both 3D position information and semantic information. When only a monocular RGB camera is used, the method can directly output the drivable area in the three-dimensional camera coordinate system, which is both practical and easy to use. When only a monocular RGB camera is used, a 3D drivable area containing semantic information can be directly and effectively output, thereby greatly improving the environmental perception ability of intelligent vehicles or robots for 3D space. Compared with the traditional 2D drivable area, the method of the present invention can output the 3D coordinates of the drivable area in real space, which is more practical in the field of autonomous driving and intelligent robots. Compared with the use of ranging devices such as laser radar or binocular cameras, the method of the present invention can output 3D position information using only a monocular RGB camera, which is lower in cost and easier to use.
[0133] Based on the same inventive concept, the present invention also provides a 3D drivable area detection system based on 3D semantic scene completion, including: a data processing module, a monocular depth estimation module, a 3D pseudo point cloud generation module, a 3D voxel generation module, a 3D semantic scene completion module and a passability judgment module, wherein:
[0134] The data processing module is used to obtain the RGB image output by the monocular camera and perform distortion correction on it;
[0135] The monocular depth estimation module is used to perform monocular depth estimation on the distortion-corrected RGB image, obtain the corresponding depth map, and construct a depth dictionary with pixel coordinates as indexes;
[0136] The 3D pseudo point cloud generation module is used to project the pixel points of the depth map in the depth dictionary from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system according to the camera intrinsic parameter matrix to construct a 3D pseudo point cloud;
[0137] 3D voxel generation module, used to voxelize 3D pseudo point cloud and generate 3D occupied voxel set based on set spatial range and voxel resolution;
[0138] A 3D semantic scene completion module is used to complete the 3D semantic scene for the 3D occupied voxels in the 3D occupied voxel set, output the 3D semantic voxels containing the semantic category, and obtain the 3D semantic voxel set;
[0139] The passability judgment module is used to combine the semantic category and 3D position information in the 3D semantic voxel set to judge whether the voxel area to be detected is passable and obtain the drivable area in the entire 3D space.
[0140] Based on the same inventive concept, the present invention also provides an electronic device, the device comprising:
[0141] A memory storing executable program code;
[0142] a processor coupled to the memory;
[0143] The processor calls the executable program code stored in the memory to execute the steps of the 3D drivable area detection method based on 3D semantic scene completion as described above, and can achieve the same technical effect as the method.
[0144] Based on the same inventive concept, the present invention also provides a computer-readable storage medium, which stores computer instructions. When the computer instructions are called, they are used to execute the steps of the 3D drivable area detection method based on 3D semantic scene completion as mentioned above, and can achieve the same technical effect as the method.
Claims
1. A 3D drivable area detection method based on 3D semantic scene completion, characterized in that: The following steps are involved: Get the RGB image output by the monocular camera and perform distortion correction on it; Perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map, and construct a depth dictionary using pixel coordinates as indexes; According to the camera intrinsic parameter matrix and the depth dictionary, each pixel of the depth map is projected from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system to construct a 3D pseudo point cloud. Voxelize the 3D pseudo point cloud, obtain the 3D occupied voxels of the set spatial range and voxel resolution, and obtain the 3D occupied voxel set; specifically: (1) Get all 3D point sets in the 3D pseudo point cloud , and They are 3D point sets The first and second 3D points in ; (2) Set the voxel resolution to , set the space range to front and back ,about ,up and down , defining the 3D voxel space , its length, width and height are: ; ; ; in, The three-dimensional voxel space Length, The three-dimensional voxel space The width of The three-dimensional voxel space Height, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system Minimum value in the axis direction; (3) Based on the 3D point set , get the voxel index set corresponding to all 3D points for: ; in, For the first voxel in length The index value on For the first voxel in width The index value on For the first voxel at height The index value on For the second voxel at length The index value on For the second voxel in width The index value on For the second voxel at height The index value on ; 3D Point Collection Middle 3D points The corresponding voxel index The calculation formula is: ; ; ; in, To calculate the rounding function, For the Voxel length The index value on For the Voxel width The index value on For the Voxel at height The index value on ; , , Respectively 3D points in the 3D camera coordinate system axis, axis, The coordinate value in the axis direction, T represents transpose; (4) Based on the voxel index set , determine whether the voxel is within the set space range; The first 3D points The corresponding voxel index If satisfied: ; in, is logical AND; The first The voxels corresponding to the 3D points are located within the set spatial range; right All indexes in the , output voxel index set within the set space range for: ; in, To set the length of the first voxel in the space range The index value on To set the width of the first voxel in the spatial range The index value on To set the first voxel in the space range at height The index value on To set the length of the second voxel in the spatial range The index value on To set the width of the second voxel in the spatial range The index value on To set the second voxel in the space range at height The index value on ; (5) Construct a 3D voxel set in the form of a three-dimensional matrix , the dimension is , the initial value is 0; traverse the voxel index set within the set space range ,according to The index in the 3D voxel set The voxel in is changed from the initial value 0 to 1, which means that the voxel is occupied; after traversal and modification, the 3D occupied voxel set is output ; Performing 3D semantic scene completion on the 3D occupied voxels in the 3D occupied voxel set, outputting 3D semantic voxels containing semantic categories, and obtaining a 3D semantic voxel set; Combine the semantic categories and 3D coordinates of the center points of the 3D semantic voxels in the 3D semantic voxel set to determine whether the voxel area to be detected is passable, and obtain the drivable area in the entire 3D space; specifically: (1) Based on the 3D semantic voxel set , get The 3D coordinate set corresponding to the center points of all voxels in for: ; in, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, T represents transpose; 3D semantic voxel collection Middle The voxel index set within the set space corresponds to the voxel Middle Voxel Index , the 3D coordinates of the voxel center are: ; ; ; in, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, To set the space range Voxel length The index value on To set the space range Voxel width The index value on To set the space range Voxel at height The index value on ; (2) Based on the 3D semantic voxel set , get the semantic category set corresponding to all voxels ; in, is a set of 3D semantic voxels The category corresponding to the first voxel in is a set of 3D semantic voxels The category corresponding to the second voxel in is a set of 3D semantic voxels Middle The category to which the voxels correspond; (3) According to the 3D coordinate set corresponding to the center points of all voxels and the set of semantic categories , and According to the one-to-one correspondence of the index, determine whether the voxel area corresponding to the 3D coordinates and semantic category is passable; 3D semantic voxel collection Middle The voxel area of a voxel is: Front and rear: ; about: ; up and down: ; If the semantic category corresponding to the voxel is not an obstacle, the voxel area is passable; the above judgment is performed on all voxels, and the drivable area within the entire space is output.
2. The 3D drivable area detection method based on 3D semantic scene completion according to claim 1, characterized in that: The HELP in the camera correction toolbox in MATLAB is used to correct the distortion of the RGB image.
3. The 3D drivable area detection method based on 3D semantic scene completion according to claim 1, characterized in that: Perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map, and construct a depth dictionary using pixel coordinates as indexes; Specifically: Based on the BTS algorithm, the monocular depth estimation is performed on the distortion-corrected RGB image, and a 16-bit depth map with the same resolution as the distortion-corrected RGB image is output. The depth dictionary is constructed with the pixel coordinates of the 16-bit depth map as the index; the depth dictionary stores data in the form of key:value, where the key is the two-dimensional pixel coordinate of each pixel in the 16-bit depth map, and the value is the pixel value at the corresponding coordinate divided by 256, that is, the value is the pixel value in the three-dimensional camera coordinate system. Axis coordinate values.
4. The 3D drivable area detection method based on 3D semantic scene completion according to claim 1, characterized in that: Project each pixel in the depth map from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system to construct a 3D pseudo point cloud; specifically: 3D point in the 3D camera coordinate system and 2D pixel points in a 2D pixel coordinate system The following conversion relationship exists: ; in, is the camera intrinsic parameter matrix, is the 16-bit depth image center; and is the camera focal length; , , are 3D points in the 3D camera coordinate system of X axis, Y Axis and Z Coordinate values in the axis direction; , They are 2D pixel points in the 2D pixel coordinate system of X Axis and Y Coordinate values in the axis direction; The above conversion is performed on each pixel in the 16-bit depth image to obtain the 3D point corresponding to each pixel, thereby generating a 3D pseudo point cloud corresponding to the 16-bit depth image.
5. The 3D drivable area detection method based on 3D semantic scene completion according to claim 1, characterized in that: The 3D occupied voxels in the 3D occupied voxel set are completed with 3D semantic scenes, and 3D semantic voxels containing semantic categories are output to obtain a 3D semantic voxel set; specifically: For a set of 3D occupied voxels , LMSCNet algorithm is used to complete 3D semantic scenes and output a set of 3D semantic voxels containing semantic categories , and Same dimension, The value of each element is Any integer in the range is the total number of semantic categories, and 0 is the background category.
6. A 3D drivable area detection system based on 3D semantic scene completion, characterized in that: include: The data processing module is used to obtain the RGB image output by the monocular camera and perform distortion correction on it; The monocular depth estimation module is used to perform monocular depth estimation on the distortion-corrected RGB image to obtain a depth map and construct a depth dictionary using pixel coordinates as indexes. The 3D pseudo point cloud generation module is used to project each pixel of the depth map from the two-dimensional pixel coordinate system to the three-dimensional camera coordinate system according to the camera intrinsic parameter matrix and the depth dictionary to construct a 3D pseudo point cloud; The 3D voxel generation module is used to voxelize the 3D pseudo point cloud, obtain the 3D occupied voxels of the set spatial range and voxel resolution, and obtain the 3D occupied voxel set; specifically: (1) Get all 3D point sets in the 3D pseudo point cloud , and They are 3D point sets The first and second 3D points in ; (2) Set the voxel resolution to , set the space range to front and back ,about ,up and down , defining the 3D voxel space , whose length, width and height are: ; ; ; in, The three-dimensional voxel space Length, The three-dimensional voxel space The width of The three-dimensional voxel space Height, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system The minimum value in the axis direction, The three-dimensional voxel space In the 3D camera coordinate system The maximum value along the axis, The three-dimensional voxel space In the 3D camera coordinate system Minimum value in the axis direction; (3) Based on the 3D point set , get the voxel index set corresponding to all 3D points for: ; in, For the first voxel in length The index value on For the first voxel in width The index value on For the first voxel at height The index value on For the second voxel at length The index value on For the second voxel in width The index value on For the second voxel at height The index value on ; 3D Point Collection Middle 3D points The corresponding voxel index The calculation formula is: ; ; ; in, To calculate the rounding function, For the Voxel length The index value on For the Voxel width The index value on For the Voxel at height The index value on ; , , Respectively 3D points in the 3D camera coordinate system axis, axis, The coordinate value in the axis direction, T represents transpose; (4) Based on the voxel index set , determine whether the voxel is within the set space range; The first 3D points The corresponding voxel index If satisfied: ; in, is logical AND; The first The voxels corresponding to the 3D points are located within the set spatial range; right All indexes in the , output voxel index set within the set space range for: ; in, To set the length of the first voxel in the space range The index value on To set the width of the first voxel in the spatial range The index value on To set the first voxel in the space range at height The index value on To set the length of the second voxel in the spatial range The index value on To set the width of the second voxel in the spatial range The index value on To set the second voxel in the space range at height The index value on ; (5) Construct a 3D voxel set in the form of a three-dimensional matrix , the dimension is , the initial value is 0; traverse the voxel index set within the set space range ,according to The index in the 3D voxel set The voxel in is changed from the initial value 0 to 1, which means that the voxel is occupied; after traversal and modification, the 3D occupied voxel set is output ; A 3D semantic scene completion module is used to complete the 3D semantic scene for the 3D occupied voxels in the 3D occupied voxel set, output the 3D semantic voxels containing the semantic category, and obtain the 3D semantic voxel set; The passability judgment module is used to combine the semantic category and 3D position information of the 3D semantic voxels in the 3D semantic voxel set to judge whether the voxel area to be detected is passable and obtain the drivable area in the entire 3D space; specifically: (1) Based on the 3D semantic voxel set , get The 3D coordinate set corresponding to the center points of all voxels in for: ; in, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels The center of the first voxel in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, For the The voxel center in the 3D camera coordinate system Axis coordinate values, T represents transpose; 3D semantic voxel collection Middle The voxel index set within the set space corresponds to the voxel Middle Voxel Index , the 3D coordinates of the voxel center are: ; ; ; in, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, is a set of 3D semantic voxels Middle The voxel center in the 3D camera coordinate system Axis coordinate values, To set the space range Voxel length The index value on To set the space range Voxel width The index value on To set the space range Voxel at height The index value on ; (2) Based on the 3D semantic voxel set , get the semantic category set corresponding to all voxels ; in, is a set of 3D semantic voxels The category corresponding to the first voxel in is a set of 3D semantic voxels The category corresponding to the second voxel in is a set of 3D semantic voxels Middle The category to which the voxels correspond; (3) According to the 3D coordinate set corresponding to the center points of all voxels and the set of semantic categories , and According to the one-to-one correspondence of the index, determine whether the voxel area corresponding to the 3D coordinates and semantic category is passable; 3D semantic voxel collection Middle The voxel area of a voxel is: Front and rear: ; about: ; up and down: ; If the semantic category corresponding to the voxel is not an obstacle, the voxel area is passable; the above judgment is performed on all voxels, and the drivable area within the entire space is output.
7. An electronic device, characterized in that: The device comprises: A memory storing executable program code; a processor coupled to the memory; The processor calls the executable program code stored in the memory to execute the steps of the 3D drivable area detection method based on 3D semantic scene completion as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, which, when called, are used to execute the steps of the 3D drivable area detection method based on 3D semantic scene completion as described in any one of claims 1 to 5.
Citation Information
Patent Citations
A system and method for semantic scene completion
CN109447923A
Method for semantic completion of single depth map point cloud scene
CN112037138A
Semantic scene completion method based on image and point cloud fusion in automatic driving scene
CN116503825A
Instance-aware monocular semantic scene completion method, medium and equipment
CN117422629A
Three-dimensional drivable area detection method and device based on monocular RGB camera
CN117830172A
