Unstructured scene passable area detection method based on fusion guidance

Abstract

The invention discloses an unstructured scene passable region detection method based on fusion guidance. The method comprises the following steps: constructing a dense point cloud; segmenting the ground; sampling dynamic key points; and segmenting the SAM large model. Based on the design of the fusion guide large model, the dense point cloud is constructed to increase the point cloud proportion of the to-be-detected target, so that the structure information and semantic information of the target are enriched, and the point cloud data quality and effectiveness of a ground segmentation algorithm are enhanced. Meanwhile, by means of dynamic key point sampling, feature point clouds can be generated through regions according to the probability obtained by point cloud segmentation, and wrong guidance of point cloud segmentation misjudgment on the SAM large model is reduced. According to the method, the feature point cloud generated by dynamic key point sampling is taken as priori knowledge, and segmentation reasoning of the SAM large model is guided, so that the accuracy and generalization ability of passable region detection in an unstructured scene are effectively improved.

Description

Technical Field

[0001] The present invention relates to the field of automatic driving passable area detection in unstructured scenes, and in particular to an unstructured scene passable area detection method based on fusion guidance. Background Art

[0002] Compared with structured road environments, unstructured scenes have dynamic and changeable terrain, limited data sample size, and weak model generalization ability. These factors pose severe challenges to the detection of passable areas in unstructured environments. Therefore, accurately identifying the passability of the terrain around unmanned autonomous vehicles is crucial for reliable path planning and control. In recent years, passable area detection methods are mainly divided into two categories: single-modal and multi-modal. Single-modal methods are further divided into two paths: image-based and lidar point cloud-based. Image-based methods regard passable area detection as a pixel-level classification problem, which belongs to the semantic segmentation task; early lidar point cloud-based detection algorithms mostly rely on rule settings, but nowadays deep learning networks are gradually used for point cloud segmentation. However, single-modal methods are insufficient in dealing with complex unstructured scenes, and cannot fully understand and analyze the complexity of the scene, resulting in a decrease in the accuracy of passable area detection, which is difficult to meet actual needs. In order to solve these problems, some studies have explored methods that fuse multimodal information. However, the detection accuracy of these methods still has room for improvement, and they can often only achieve good results on specific data sets and lack the ability to generalize across scenarios. As a new paradigm of generative artificial intelligence, large-scale visual models are closely connected with downstream tasks by virtue of pre-training on ultra-large-scale datasets and combined with cross-modal cue learning. Therefore, how to design a passable region detection method that can integrate large models and is applicable to unstructured scenes, and improve the accuracy and generalization ability of passable region detection, has become a key issue that needs to be solved urgently. Summary of the invention

[0003] In order to solve the above problems in the prior art, the present invention proposes a passable region detection method for unstructured scenes based on fusion guidance, which improves the accuracy and generalization ability of passable region detection.

[0004] The basic idea of ​​the present invention is to adopt a cascade fusion method, build a dense local point cloud map through a time-series multi-frame dynamic adaptive key frame strategy, pre-extract probable passable areas in combination with ground segmentation operations, and generate feature point clouds through dynamic key point sampling, which are used as prior knowledge to guide the segmentation reasoning of the SAM large model, thereby significantly improving the accuracy and generalization ability of passable area detection.

[0005] In order to achieve the above object, the technical solution of the present invention is as follows: A method for detecting regions in an unstructured scene based on fusion guidance comprises the following steps:

[0006] A. Building a dense point cloud

[0007] A1. In order to improve the point cloud data quality of the ground segmentation algorithm, the sparse point cloud data is densified.

[0008] A laser inertial navigation odometer is used to receive laser point cloud and inertial navigation data, and these data are used to estimate the state and motion trajectory of the vehicle at different times, and the optimal posture transformation matrix is ​​calculated.

[0009] A2. Use the optimal pose transformation matrix to convert the current frame point cloud to the global coordinate system and insert the converted point cloud into the double-ended queue container. The size of the double-ended queue container is controlled by a preset capacity limit threshold, and the first-in-first-out principle is adopted. When the number of stored point clouds exceeds the capacity limit, all point clouds in the double-ended queue container are accumulated to construct a dense local point cloud map. Since the point cloud ground segmentation algorithm requires point cloud coordinates in the vehicle body coordinate system, the transformation relationship of the laser radar coordinate system relative to the global coordinate system is obtained according to the vehicle pose saved by the laser inertial navigation odometer, and the coordinates of the dense point cloud map are transformed from the global coordinate system to the laser radar coordinate system using the transformation relationship of the first frame of the head in the double-ended queue container. Then, the static pose transformation matrix from the laser radar coordinate system to the vehicle body coordinate system obtained by calibration is used to calculate the coordinates of the point cloud in the vehicle body coordinate system. The specific formula is as follows:

[0010]

[0011]

[0012] Among them, x v ,y v 、z v are t i The x-coordinate, y-coordinate, and z-coordinate of the original position of the laser point at the moment w ,y w 、z w are the x-coordinate, y-coordinate, and z-coordinate of the laser point position transformed to the global map coordinate system, R(t i ) is the rotation matrix in the optimal pose transformation matrix, is the translation vector in the optimal pose transformation matrix. Represents the current time T c Densified laser point cloud, P′ t Represents the laser point cloud frame at time t, T c represents the current moment, b n Represents the interval between the timestamps of the first and last frame of point cloud stored in the deque container. Represents T c -bn The pose transformation matrix from the global coordinate system to the lidar coordinate system at the moment, Represents the pose transformation matrix from the lidar coordinate system to the global coordinate system at time t, T bl Represents the static pose transformation matrix from the LiDAR coordinate system to the vehicle coordinate system.

[0013] B. Ground segmentation

[0014] The dense local point cloud map is subjected to passable region detection, and the probable passable region consisting of ground points and non-ground points is pre-extracted in a coarse-grained manner, as shown in the following formula:

[0015]

[0016] in, Represents the point cloud after ground segmentation, Segment represents the ground segmentation operation, Represents the dense point cloud generated in step A.

[0017] C. Dynamic key point sampling

[0018] First, perform 3D voxel division on the ground point cloud and non-ground point cloud generated in step B, then calculate the point cloud voxel density in each voxel and filter it according to the preset voxel density threshold, delete the voxels with a density less than the voxel density threshold and do not use them as input for subsequent sampling. Then loop through the filtered voxels and perform the farthest point sampling algorithm on each voxel. At the same time, dynamically adjust the number of sampling points of the farthest point sampling algorithm according to the average distance density of the point cloud in each voxel. The specific calculation formula is as follows:

[0019]

[0020]

[0021] in, represents the point cloud generated after ground segmentation in step B; VOXEL represents the three-dimensional voxel division operation, which determines the index position of the voxel where each point in the point cloud is located through the x, y, and z position coordinates of each point in the vehicle coordinate system; V represents the three-dimensional voxel structure; V n Represents a specific voxel in a three-dimensional voxel structure; FPS represents the farthest point sampling operation; is the final feature point cloud; N represents the total number of all voxels in the three-dimensional voxel structure.

[0022] The point cloud average distance density estimates the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The distance of each point in the point cloud is expressed as the distance between a point p in the point cloud and the point closest to the point p in the point cloud. The calculation formula is as follows:

[0023] d p =min(dis(p, q)), q=1, 2,...,N, p≠q

[0024]

[0025] In a point cloud with N points, dis(p, q) is used to represent the distance between point p and any other point q, and d is used to represent the distance between point p and any other point q. p represents the minimum distance between point p and other points, Represents the average distance density of the point cloud. A feature point cloud is constructed by accumulating all ground points and non-ground points sampled in each voxel. Then, the feature point cloud is synchronously transformed with the input image of the SAM large model in time and space, and the ground points and non-ground points in the feature point cloud are projected from the lidar coordinate system to the image coordinate system to generate regions of interest. These regions of interest will serve as positive prompt information and negative prompt information for subsequent guidance of SAM large model reasoning. The coordinate systems of the radar point cloud and the camera image are different. The relative pose of the two is obtained through the joint calibration of the lidar camera, and the relative pose includes a rotation matrix and a translation vector.

[0026] D. SAM large model segmentation

[0027] The positive and negative prompt information generated in step C are uniformly input into the parameters of the SAM large model to guide the segmentation reasoning of the SAM large model and generate a fused passable area map.

[0028] Furthermore, the ground segmentation operation Segment in step B adopts an improved Patchwork++ ground filtering algorithm based on local plane fitting as a ground segmentation algorithm.

[0029] Compared with the prior art, the present invention has the following beneficial effects:

[0030] 1. Based on the design of the fusion-guided large model, the present invention increases the point cloud proportion of the target to be detected by constructing a dense point cloud, thereby enriching the structural information and semantic information of the target and enhancing the point cloud data quality and effectiveness of the ground segmentation algorithm. At the same time, by using dynamic key point sampling, the probability obtained by point cloud segmentation can generate feature point clouds through regions, reducing the misleading of point cloud segmentation misjudgment on the SAM large model.

[0031] 2. The present invention uses the feature point cloud generated by dynamic key point sampling as prior knowledge to guide the segmentation reasoning of the SAM large model, thereby effectively improving the accuracy and generalization ability of regional detection in unstructured scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 It is a schematic diagram of the process of the present invention.

[0033] Figure 2 It is a deployment architecture diagram of the present invention. DETAILED DESCRIPTION

[0034] The specific implementation of the present invention is described in detail below with reference to the accompanying drawings. The present invention proposes a method for detecting regions in unstructured scenes based on fusion guidance. The overall process of the method is as follows: Figure 1 As shown, the following steps are included:

[0035] A. Building a dense point cloud

[0036] A1. In order to improve the point cloud data quality of the ground segmentation algorithm, the sparse point cloud data is densified.

[0037] A laser inertial navigation odometer is used to receive laser point cloud and inertial navigation data, and these data are used to estimate the state and motion trajectory of the vehicle at different times, and the optimal posture transformation matrix is ​​calculated.

[0038] A2. Use the optimal pose transformation matrix to convert the current frame point cloud to the global coordinate system and insert the converted point cloud into the double-ended queue container. The size of the container is controlled by a preset capacity limit threshold, and the first-in-first-out principle is adopted. When the number of stored point clouds exceeds the capacity limit, all point clouds in the container are accumulated to construct a dense local point cloud map. Since the point cloud ground segmentation algorithm requires point cloud coordinates in the vehicle body coordinate system, the transformation relationship of the laser radar coordinate system relative to the global coordinate system is obtained according to the vehicle pose saved by the laser inertial navigation odometer, and the coordinates of the dense point cloud map are transformed from the global coordinate system to the laser radar coordinate system using the transformation relationship of the first frame of the head in the double-ended queue container. Then, the static pose transformation matrix from the laser radar coordinate system to the vehicle body coordinate system obtained by calibration is used to calculate the coordinates of the point cloud in the vehicle body coordinate system. The specific formula is as follows:

[0039]

[0040]

[0041] Among them, x v ,y v 、z v are t i The x-coordinate, y-coordinate, and z-coordinate of the original position of the laser point at the moment w ,y w 、z w are the x-coordinate, y-coordinate, and z-coordinate of the laser point position transformed to the global map coordinate system, R(t i ) is the rotation matrix in the optimal pose transformation matrix, is the translation vector in the optimal pose transformation matrix. Represents the current time T c Densified laser point cloud, P′ t Represents the laser point cloud frame at time t, T c represents the current moment, b n Represents the interval between the timestamps of the first and last frame of point cloud stored in the deque container. Represents T c -b n The pose transformation matrix from the global coordinate system to the lidar coordinate system at the moment, Represents the pose transformation matrix from the lidar coordinate system to the global coordinate system at time t, T bl Represents the static pose transformation matrix from the LiDAR coordinate system to the vehicle coordinate system.

[0042] In an embodiment of the present invention, the capacity limit of the double-ended queue container is set to 5 to construct a local dense point cloud map, thereby better balancing the real-time and reliability requirements.

[0043] B. Ground segmentation

[0044] The dense local point cloud map is subjected to passable region detection, and the probable passable region consisting of ground points and non-ground points is pre-extracted in a coarse-grained manner, as shown below:

[0045]

[0046] in, Represents the point cloud after ground segmentation, Segment represents the ground segmentation operation, Represents the dense point cloud generated in step A.

[0047] For the ground segmentation operation Segment, in the embodiment of the present invention, an improved Patchwork++ ground filtering algorithm based on local plane fitting is adopted as the ground segmentation algorithm.

[0048] C. Dynamic key point sampling

[0049] First, perform 3D voxel division on the ground point cloud and non-ground point cloud generated in step B, then calculate the point cloud voxel density in each voxel and filter it according to the preset voxel density threshold, delete the voxels with a density less than the voxel density threshold and do not use them as input for subsequent sampling. Then loop through the filtered voxels and perform the farthest point sampling algorithm on each voxel. At the same time, dynamically adjust the number of sampling points of the farthest point sampling algorithm according to the average distance density of the point cloud in each voxel. The specific calculation formula is as follows:

[0050]

[0051]

[0052] in, represents the point cloud generated after ground segmentation in step B; VOXEL represents the three-dimensional voxel division operation, which determines the index position of the voxel where each point in the point cloud is located through the x, y, and z position coordinates of each point in the vehicle coordinate system; V represents the three-dimensional voxel structure; V n Represents a specific voxel in a three-dimensional voxel structure; FPS represents the farthest point sampling operation; is the final feature point cloud; N represents the total number of all voxels in the three-dimensional voxel structure.

[0053] The average distance density of the point cloud is used to estimate the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The distance of each point in the point cloud is expressed as the distance between a point p in the point cloud and the point closest to the point p in the point cloud. The calculation formula is as follows:

[0054] d p =min(dis(p, q)), q=1, 2,...,N, p≠q

[0055]

[0056] In a point cloud with N points, dis(p, q) is used to represent the distance between point p and any other point q, and d is used to represent the distance between point p and any other point q. p represents the minimum distance between point p and other points, Represents the average distance density of the point cloud. Finally, a feature point cloud is constructed by accumulating all ground points and non-ground points sampled in each voxel. Then, the feature point cloud is synchronously transformed with the input image of the SAM large model in time and space, and the ground points and non-ground points in the feature point cloud are projected from the lidar coordinate system to the image coordinate system to generate regions of interest. These regions of interest will serve as positive prompt information and negative prompt information for subsequent guidance of SAM large model reasoning. The coordinate systems of the radar point cloud and the camera image are different. The relative pose of the two is obtained through the joint calibration of the lidar camera, and the relative pose includes a rotation matrix and a translation vector.

[0057] D. SAM large model segmentation

[0058] The positive and negative prompt information generated in step C are uniformly input into the parameters of the SAM large model to guide the segmentation reasoning of the SAM large model and generate a fused passable area map.

[0059] The method deployment architecture of the embodiment of the present invention is as follows Figure 2As shown, the software platform relies on a Linux-like system architecture and uses Docker containerization technology for rapid deployment and startup. The basic operating system is Ubuntu20.04 Focal Fossa, and the driving operating system middleware is ROS2 Galactic. At the same time, the ROS1 Noetic system and Ros1_brigdge message bridge components are installed, which are compatible with ROS1 algorithm deployment and meet the requirements of sensor ROS1 driver limitations. The program deployed in this embodiment not only includes building dense point clouds, ground segmentation, dynamic key point sampling, and SAM large model segmentation, but also adds point cloud preprocessing and image preprocessing to remove noise from point clouds and images. In order to improve the real-time perception, all functional modules are deployed in the C++ programming language, and point cloud processing is adapted to both the Point Cloud Library library based on CPU computing and the cuPCL library based on GPU computing. The maximum adaptive resolution size of the image processing part is 1920×1280. The SAM large model segmentation is implemented based on the NVIDIA TensorRT high-performance inference SDK. First, the pth model file obtained by training SAM under the Pytorch framework is converted into an ONNX model, and then the trtexec tool is used to convert the ONNX model into the engine file of TensorRT. The engine file is the model file format loaded by TensorRT. The conversion process involves operator fusion, layer optimization and quantization, etc. Finally, based on the converted engine file, the C++ interface API of TensorRT is called for model inference.

[0060] The above describes the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments, and the above embodiments and descriptions are only for explaining the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention may have various changes and improvements, and these changes and improvements fall within the scope of the present invention to be protected.

Claims

1. A method for detecting passable regions in unstructured scenes based on fusion guidance, characterized in that: The following steps are involved: A. Building a dense point cloud A1. To improve the point cloud data quality of the ground segmentation algorithm, the sparse point cloud data is densified; Use the laser inertial navigation odometer to receive laser point cloud and inertial navigation data, and use these data to estimate the state and motion trajectory of the vehicle at different times, and calculate the optimal posture transformation matrix; A2. Use the optimal posture transformation matrix to transform the current frame point cloud into the global coordinate system and insert the transformed point cloud into the double-ended queue container; the size of the double-ended queue container is controlled by a preset capacity limit threshold, and the first-in-first-out principle is adopted. When the number of stored point clouds exceeds the capacity limit, all point clouds in the double-ended queue container are accumulated to construct a dense local point cloud map; since the point cloud ground segmentation algorithm requires point cloud coordinates in the vehicle coordinate system, the transformation relationship of the laser radar coordinate system relative to the global coordinate system is obtained according to the vehicle posture stored in the laser inertial navigation odometer, and the coordinates of the dense point cloud map are transformed from the global coordinate system to the laser radar coordinate system using the transformation relationship of the first frame of the head in the double-ended queue container, and then the static posture transformation matrix from the laser radar coordinate system to the vehicle coordinate system obtained by calibration is used to calculate the coordinates of the point cloud in the vehicle coordinate system. The specific formula is as follows: Among them, x v ,y v 、z v are t i The x-coordinate, y-coordinate, and z-coordinate of the original position of the laser point at the moment w ,y w 、z w are the x-coordinate, y-coordinate, and z-coordinate of the laser point position transformed to the global map coordinate system, R(t i ) is the rotation matrix in the optimal pose transformation matrix, is the translation vector in the optimal posture transformation matrix; Represents the current time T c Densified laser point cloud, P′ t Represents the laser point cloud frame at time t, T c represents the current moment, b n Represents the interval between the timestamps of the first and last frame of point cloud stored in the deque container. Represents T c -b n The pose transformation matrix from the global coordinate system to the lidar coordinate system at the moment, Represents the pose transformation matrix from the lidar coordinate system to the global coordinate system at time t, T bl Represents the static pose transformation matrix from the LiDAR coordinate system to the vehicle coordinate system; B. Ground segmentation The dense local point cloud map is subjected to passable region detection, and the probable passable region consisting of ground points and non-ground points is pre-extracted in a coarse-grained manner, as shown in the following formula: in, Represents the point cloud after ground segmentation, Segment represents the ground segmentation operation, Represents the dense point cloud generated in step A; C. Dynamic key point sampling First, perform three-dimensional voxel division on the ground point cloud and non-ground point cloud generated in step B, then calculate the point cloud voxel density in each voxel and filter it according to the preset voxel density threshold, delete the voxels less than the voxel density threshold and do not use them as input for subsequent sampling; then loop through the filtered voxels, perform the farthest point sampling algorithm on each voxel, and dynamically adjust the number of sampling points of the farthest point sampling algorithm according to the average distance density of the point cloud in each voxel. The specific calculation formula is as follows: in, represents the point cloud generated after ground segmentation in step B; VOXEL represents the three-dimensional voxel division operation, which determines the index position of the voxel where each point in the point cloud is located through the x, y, and z position coordinates of each point in the vehicle coordinate system; V represents the three-dimensional voxel structure; V n Represents a specific voxel in a three-dimensional voxel structure; FPS represents the farthest point sampling operation; is the final feature point cloud; N represents the total number of all voxels in the three-dimensional voxel structure; The point cloud average distance density estimates the density of the point cloud distribution by calculating the average distance of each point in the point cloud. The distance of each point in the point cloud is expressed as the distance between a point p in the point cloud and the point closest to the point p in the point cloud. The calculation formula is as follows: d p =min(dis(p,q)),q=1,2,…,N,p≠q In a point cloud with N points, dis(p, q) is used to represent the distance between point p and any other point q, and d is used to represent the distance between point p and any other point q. p represents the minimum distance between point p and other points, Represents the average distance density of the point cloud; constructs a feature point cloud by accumulating all ground points and non-ground points sampled in each voxel; then, performs a spatiotemporal synchronous transformation between the feature point cloud and the input image of the SAM large model, projects the ground points and non-ground points in the feature point cloud from the lidar coordinate system to the image coordinate system, and generates regions of interest, which will serve as positive prompt information and negative prompt information for subsequent guidance of the SAM large model reasoning; the radar point cloud and the camera image have different coordinate systems, and the relative pose of the two is obtained through the joint calibration of the lidar camera, and the relative pose includes a rotation matrix and a translation vector; D. SAM large model segmentation The positive and negative prompt information generated in step C are uniformly input into the parameters of the SAM large model to guide the segmentation reasoning of the SAM large model and generate a fused passable area map.

2. According to claim 1, a method for detecting passable regions in unstructured scenes based on fusion guidance, characterized in that: The ground segmentation operation Segment described in step B adopts the improved Patchwork++ ground filtering algorithm based on local plane fitting as the ground segmentation algorithm.

Images