Walkable pavement detection method, device, medium and equipment

Abstract

The application discloses a walkable pavement detection method based on radar and vision fusion, comprising the following steps: acquiring point cloud data collected by a 3D laser radar and image information collected by a camera on a robot; performing ground segmentation on the point cloud data to obtain ground laser points and non-ground laser points; projecting the ground laser points into the image information to obtain corresponding projection points; taking a middle lower edge region of the image information as a ground region to intercept a ground reference block; comparing a projection block where the projection points are located with the ground reference block to acquire target ground points in the ground laser points. The application effectively solves the problems of poor segmentation effect, high misjudgment rate and low detection accuracy of the prior art when detecting a pavement.

Description

Methods, devices, media and equipment for detecting walkable pavements Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method, apparatus, medium and equipment for detecting walkable surfaces based on radar and vision fusion. Background Technology

[0002] In general environments, robots or mobile devices navigate or avoid obstacles by constantly sensing their surroundings to make judgments and responses. Detecting walkable surfaces is a crucial step for robots or mobile devices, directly impacting obstacle avoidance and SLAM (Site-Based Automation). Failure to accurately detect walkable surfaces can lead to uncontrollable situations such as the robot walking into potholes or onto grassy or non-concrete surfaces.

[0003] Existing technologies primarily employ radar-based 3D point cloud detection or visible light image detection to detect walkable road surfaces. Radar-based 3D point cloud detection is ineffective at segmenting abnormal road surfaces, failing to accurately distinguish between concrete roads and grass. Visible light image detection, on the other hand, is prone to misclassifying similar patterns, such as mistaking shaded areas for non-road surfaces. Therefore, existing technologies suffer from poor segmentation, high false positive rates, and low detection accuracy when detecting road surfaces. Summary of the Invention

[0004] This invention provides a method, apparatus, medium, and equipment for detecting walkable pavement based on radar and vision fusion, in order to solve the problems of poor segmentation effect, high misjudgment rate, and low detection accuracy in existing technologies for pavement detection.

[0005] A method for detecting walkable surfaces based on radar and vision fusion, the method comprising:

[0006] Acquire point cloud data collected by the 3D LiDAR on the robot and image information captured by the camera;

[0007] The point cloud data is segmented into ground laser points and non-ground laser points;

[0008] The ground laser points are projected onto the image information to obtain the corresponding projection points;

[0009] Using the lower middle edge of the image information as the ground region, a ground reference patch is extracted.

[0010] By comparing the projection block where the projection point is located with the ground reference block, the target ground point in the ground laser point is obtained.

[0011] Optionally, the step of performing ground segmentation on the point cloud data to obtain ground laser points and non-ground laser points includes:

[0012] Acquire the innermost point cloud data collected by 3D LiDAR;

[0013] The innermost point cloud data is segmented into ground based on the laser point height information to obtain the innermost ground laser points and the innermost non-ground laser points.

[0014] The point cloud data is converted into polar coordinates and divided into several point cloud sets according to a preset horizontal angle interval. Each point cloud set includes the innermost laser point and the outermost laser point.

[0015] Traverse each point cloud set and perform ground segmentation on the outer ring laser points according to the slope difference of the laser points to obtain the outer ring ground laser points and the outer ring non-ground laser points;

[0016] Median filtering is applied to ground laser points on the same circle to generate ground laser points and non-ground laser points.

[0017] Optionally, the step of performing median filtering on the ground laser points on the same circle to generate ground laser points and non-ground laser points includes:

[0018] Traverse each ground laser point on each ring, take the ground laser point as the point to be processed, and obtain the continuous laser points on the same ring starting from the point to be processed according to the preset filtering radius;

[0019] The continuous laser points are binarized and sorted according to the ground segmentation results to obtain a binarized sequence;

[0020] Obtain the median of the binarized sequence. When the median represents a ground laser point, the point to be processed is a ground laser point. When the median represents a non-ground laser point, the point to be processed is a non-ground laser point.

[0021] Optionally, the step of using the lower middle edge region of the image information as the ground region to extract the ground reference patch includes:

[0022] Starting from the midpoint of the bottom edge of the image information, obtain a preset number of anchor points at equal intervals to the left and right;

[0023] Iterate through each anchor point and, using the anchor point as the center, cut out a ground reference block according to the preset size.

[0024] Optionally, the step of comparing the projection block where the projection point is located with the ground reference block to obtain the target ground point in the ground laser points includes:

[0025] Traverse each projection point, and with the projection point as the center, extract the corresponding projection block from the image information according to the preset size;

[0026] Compare the projected block with each ground reference map block, and calculate the similarity between the projected block and each ground reference map block;

[0027] Find the minimum similarity value;

[0028] When the minimum similarity value is greater than the preset similarity threshold, the laser point corresponding to the projection point is a non-ground laser point; when the minimum similarity value is less than or equal to the preset similarity threshold, the laser point corresponding to the projection point is a target ground laser point.

[0029] Optionally, the preset size is 20*20 pixels.

[0030] Optionally, the spectral cornerstone mapping algorithm is used to calculate the similarity between the projected patch and each ground reference patch, and the calculation formula is as follows:

[0031]

[0032] Where θ represents the similarity between the projected patch and the corresponding ground reference patch, x 1、2、3、……、n Represents the pixels on the projection block, y 1、2、3、……、n The x represents the pixels on the ground reference tile, where n represents the total number of pixels corresponding to the projected tile and the ground reference tile; n and y n This represents the pixels at the same location as the projected block and the ground reference block.

[0033] A walkable road surface detection device based on radar and vision fusion, the device comprising:

[0034] The acquisition module is used to acquire point cloud data collected by the 3D LiDAR on the robot and image information captured by the camera;

[0035] The ground segmentation module is used to segment the point cloud data into ground laser points and non-ground laser points.

[0036] The projection module is used to project the ground laser points onto the image information to obtain the corresponding projection points;

[0037] The cropping module is used to crop ground reference tiles using the lower middle edge area of ​​the image information as the ground area.

[0038] The ground acquisition module is used to compare the projection block where the projection point is located with the ground reference block to obtain the target ground point in the ground laser points.

[0039] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the walkable road surface detection method based on radar and vision fusion as described above.

[0040] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the walkable road surface detection method based on radar and vision fusion as described above.

[0041] This invention acquires point cloud data from a 3D LiDAR on a patrol robot and image information from a camera; it performs ground segmentation on the point cloud data to obtain ground LiDAR points and non-ground LiDAR points, achieving preliminary ground segmentation; then, it projects the ground LiDAR points onto the image information to obtain corresponding projection points; it uses the lower middle edge of the image information as the ground region to extract a ground reference patch; it compares the projection patch containing the projection point with the ground reference patch to obtain the target ground point from the ground LiDAR points, achieving secondary ground segmentation. This effectively distinguishes planar ground similar to walkable concrete pavement, reduces misjudgments, greatly recalls walkable pavement, achieves better segmentation results, higher pavement detection accuracy, and enables real-time detection. Attached Figure Description

[0042] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0043] Figure 1 is a flowchart of a walkable road surface detection method based on radar and vision fusion provided in an embodiment of the present invention;

[0044] Figure 2 is a top view of point cloud data collected by a 3D lidar according to an embodiment of the present invention;

[0045] Figure 3 is a schematic diagram of a point cloud set provided in an embodiment of the present invention;

[0046] Figure 4 is a schematic diagram of an anchor point provided in an embodiment of the present invention;

[0047] Figure 5 is a schematic diagram of the target ground points provided in an embodiment of the present invention;

[0048] Figure 6 is a schematic diagram of the structure of a walkable road surface detection device based on radar and vision fusion according to an embodiment of the present invention;

[0049] Figure 7 is a schematic diagram of a computer device according to an embodiment of the present invention. Detailed Implementation

[0050] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0051] This invention provides a method for detecting walkable surfaces based on radar and vision fusion. The walkable surface refers to a surface that a robot or mobile device can traverse. This embodiment combines the advantages of both 3D LiDAR and a vision camera, complementing each other and effectively solving the problems that arise when using either alone. Unlike existing technologies, this invention first uses a polar coordinate method to segment 3D point cloud data, focusing more on local and detailed ground point cloud judgment. Moreover, the algorithm does not involve large-scale computation and achieves real-time performance. Then, the segmented ground LiDAR points are projected onto the image information acquired by the camera using intrinsic and extrinsic parameters. Finally, image processing methods are used to perform secondary filtering and confirmation on the projected points to obtain target surface points, distinguishing walkable and non-walkable surfaces. This embodiment not only distinguishes planar surfaces similar to walkable concrete surfaces, greatly recalling walkable surfaces and improving the accuracy of ground detection, but also uses basic logic control algorithms, avoiding large-scale computation, thus offering significant advantages in real-time performance.

[0052] The walkable road surface detection method based on radar and vision fusion provided in this embodiment is described in detail below. As shown in Figure 1, the walkable road surface detection method based on radar and vision fusion includes:

[0053] Step S101: Obtain point cloud data collected by the 3D LiDAR on the robot and image information collected by the camera;

[0054] Step S102: Perform ground segmentation on the point cloud data to obtain ground laser points and non-ground laser points;

[0055] Step S103: Project the ground laser points onto the image information to obtain the corresponding projection points;

[0056] Step S104: Using the lower middle edge region of the image information as the ground region, extract the ground reference patch;

[0057] Step S105: Compare the projection block where the projection point is located with the ground reference block to obtain the target ground point in the ground laser points.

[0058] The 3D LiDAR and camera are stacked on the robot or mobile device, allowing for simultaneous data acquisition. The 3D LiDAR is used to acquire point cloud data, while the camera is used to acquire image information. The extrinsic parameters of the 3D LiDAR and camera, and the intrinsic parameters of the camera, have been pre-calibrated.

[0059] As an example, in step S101, the robot or mobile device moves along a path in a preset location, collects point cloud data of the surrounding environment through 3D LiDAR, and collects image information of the surrounding environment through a camera.

[0060] Ground segmentation refers to separating the laser points representing ground information from the point cloud data. The point cloud data is acquired by a 3D LiDAR that simultaneously emits multiple laser lines rotating one revolution. For ease of understanding, Figure 2 is a top view of the point cloud data acquired by the 3D LiDAR according to an embodiment of the present invention. In Figure 2, the point cloud data on the same revolution corresponds to the data acquired by the 3D LiDAR emitting laser lines at the same emission angle rotating one revolution. Therefore, this embodiment uses polar coordinates to perform ground segmentation on the point cloud data for each revolution angle.

[0061] As an example, in step S102, this embodiment performs ground segmentation based on the logical position and statistical characteristics between laser points. Polar coordinates are used to detect and distinguish ground laser points from non-ground laser points in the point cloud data from each acquisition angle. Finally, the ground segmentation results from all acquisition angles are integrated to complete the initial ground segmentation. This embodiment uses polar coordinates for ground segmentation of 3D laser point cloud data, which allows for greater focus on the judgment of local and detailed ground point clouds, and the algorithm can achieve real-time processing.

[0062] Here, the projection point refers to the corresponding pixel of the image information onto which the ground laser point is projected. In this embodiment, the ground laser points in the collected point cloud data are projected onto the image information collected at the same time.

[0063] As an example, in step S103, this embodiment pre-calibrates the extrinsic matrix of the 3D LiDAR and the camera using a calibration board, i.e., the position of the camera relative to the 3D LiDAR. The intrinsic matrix of the camera, i.e., the camera projection matrix, uses the parameters provided by the camera manufacturer. The formula for projecting the ground laser points onto the image information in step S103 is as follows:

[0064] Z[uv 1]T=intrinsic_mat*extrinsic_mat*[xyz 1]T

[0065] In the above formula, extrinsic_mat represents the extrinsic parameter matrix, intrinsic_mat represents the intrinsic parameter matrix, x, y, z represent the coordinate information of the ground laser point, u, v represent the coordinate information of the projection point on the image information, Z represents the homogenization function, and T represents the matrix transpose function.

[0066] The ground reference block refers to an image block containing ground information, which serves as a reference standard for secondary judgment of ground laser points.

[0067] As an example, in step S104, since the robot or mobile device acquires image information via a camera while moving on the ground, the lower-middle edge region of the image information is typically the ground. Based on this premise, this embodiment extracts a ground reference patch from the lower-middle edge region of the image information.

[0068] As an example, in step S105, after acquiring the ground laser points based on polar coordinates, this embodiment further compares and judges the ground laser points based on ground reference tiles to achieve secondary filtering and confirmation. Optionally, this embodiment uses image similarity to compare the similarity between the ground laser points segmented from the 3D laser point cloud data and the ground reference tiles, thereby ultimately accurately identifying the target ground point among the ground laser points. This embodiment uses visual image information for secondary filtering of ground laser points, which greatly improves the accuracy of ground detection, effectively distinguishes planar ground similar to walkable cement pavement, and reduces misjudgments.

[0069] In one embodiment, step S102, which involves performing ground segmentation on the point cloud data to obtain ground laser points and non-ground laser points, includes:

[0070] Step S1021: Obtain the innermost point cloud data collected by the 3D LiDAR;

[0071] Step S1022: Perform ground segmentation on the innermost point cloud data according to the laser point height information to obtain the innermost ground laser points and the innermost non-ground laser points;

[0072] Step S1023: Convert the point cloud data into polar coordinates and divide the point cloud data into several point cloud sets according to a preset horizontal angle interval. Each point cloud set includes the innermost laser point and the outermost laser point.

[0073] Step S1024: Traverse each point cloud set and perform ground segmentation on the outer ring laser points according to the slope difference of the laser points to obtain the outer ring ground laser points and the outer ring non-ground laser points.

[0074] Step S1025: Perform median filtering on the ground laser points on the same circle to generate ground laser points and non-ground laser points.

[0075] The innermost point cloud data refers to the point cloud data collected by the 3D LiDAR when emitting laser lines at the lowest emission angle. Typically, the innermost point cloud data is closest to the robot or mobile device.

[0076] As an example, in step S1021, this embodiment obtains the point cloud data corresponding to the lowest emission angle as the innermost circle point cloud data collected by the 3D LiDAR; other point cloud data are recorded as outer circle point cloud data.

[0077] The laser point height information refers to the height of the laser point in the lidar coordinate system. In this embodiment, the point cloud data closest to the 3D lidar is segmented into ground segments. The segmented ground laser points are used as reference points for polar coordinate segmentation in subsequent steps, which helps improve the accuracy of segmentation.

[0078] As an example, in step S1022, this embodiment uses the average value and variance of laser point height information for ground segmentation. First, the average value and variance of the z-axis coordinates of all innermost point cloud data are calculated, that is, the average value and variance of the height information of all innermost point cloud data are calculated, denoted as H_inner_avg and H_inner_var respectively. A height threshold parameter H_inner_thresh is preset. Each innermost point is traversed to obtain its height information, denoted as H_inner_i, where i represents the i-th point of the innermost circle. The absolute value of the difference between the i-th point of the innermost circle and the average value of the height information is calculated, and the absolute value is compared with the aforementioned height threshold parameter. If the absolute value is greater than the height threshold parameter, that is, if abs(H_inner_i-H_inner_avg)>H_inner_thresh, where abs represents the absolute value function, then the i-th point of the innermost circle is marked as a non-ground laser point; otherwise, it is marked as a ground laser point. Optionally, as a preferred example of the present invention, the height threshold parameter H_inner_thresh is set to twice the height information variance, i.e., 2.0 * H_inner_var.

[0079] Here, the horizontal angle refers to the angle on a horizontal plane perpendicular to the plane containing the laser line of the 3D LiDAR. The preset horizontal angle interval refers to a preset range of horizontal angles; point cloud data falling within the preset horizontal angle interval constitute a point cloud set.

[0080] As an example, in step S1023, this embodiment re-represents each point in all point cloud data in polar coordinates, that is, represents the data of each point in the x-axis and y-axis dimensions of the LiDAR coordinate system in polar coordinates. The coordinate system transformation formula is as follows:

[0081]

[0082] t = arctan(y ÷ x)

[0083] Where x represents the abscissa in the lidar coordinate system, y represents the ordinate in the lidar coordinate system, r represents the radius coordinate in the polar coordinate expression, and t represents the azimuth angle in the polar coordinate expression.

[0084] In this embodiment, the horizontal angle interval is preferably 1°, so as to quantize the point cloud data in polar coordinate form into 360 angles, resulting in 360 point cloud sets. Each point cloud set includes the innermost laser point and the outermost laser point. For ease of understanding, Figure 3 is a schematic diagram of the point cloud set provided in this embodiment of the invention.

[0085] The laser point slope difference refers to the slope difference between the outer circle laser points and the innermost laser points.

[0086] As an example, in step S1024, each point cloud set is traversed, the point cloud characteristics within each angular interval are analyzed and statistically analyzed, and the ground laser points in the outer ring laser points are determined and obtained. First, a point cloud set is obtained, denoted as C_j, where j represents the j-th point cloud set. The laser points in the point cloud set C_j are sorted in ascending order according to their radius coordinates r in polar coordinates, resulting in the point cloud sequence C_order_j. Generally, the innermost laser point is the closest to the 3D LiDAR and is sorted first. Therefore, starting from the second laser point in the point cloud sequence C_order_j, each laser point p in the point cloud sequence C_order_j is traversed. k k = 2, 3, ..., K, where K represents the number of laser points in the point cloud sequence C_order_j. For each laser point p... k The slope difference, denoted as slp_k, is calculated by comparing k = 2, 3, ..., K with the first point in the point cloud sequence C_order_j. All slope differences slp_k, k = 2, 3, ..., K, are combined to obtain the slope difference set C_slp_j. Among them, the laser point p... k The formula for calculating the slope difference between k = 2, 3, ..., K and the first laser point P1 in sequence C_order_j is as follows: slp_k = (z k -z1)÷(r k-r1), z represents the height information of the laser point in the lidar coordinate system, and r represents the radius coordinate of the laser point in polar coordinates. For the sequence difference set C_slp_j, the corresponding mean and variance are calculated, denoted as C_slp_j_avg and C_slp_j_var, respectively. Finally, for all laser points p in the point cloud set C_j... k For k = 2, 3, ..., K, ground point identification is performed. Since step S1022 has already divided the innermost point cloud data into ground laser points and non-ground laser points, the identification logic is as follows: traverse each outer laser point p in the point cloud set C_j. k k = 2, 3, ..., K, compare the outer ring laser points p k The slope difference slp_k is compared with the first preset slope difference threshold or the outer ring laser point p. k The difference between the slope difference slp_k and the variance is compared with the second preset slope difference threshold. When the innermost laser point in the point cloud set C_j is a ground laser point, if the outermost laser point p k The slope difference slp_k is greater than the first preset slope difference threshold or the outer ring laser point p k If the difference between the slope difference slp_k and the variance is greater than the second preset slope difference threshold, then the outer ring laser point p k If the innermost laser point in the point cloud set C_j is a non-ground laser point, then the outermost laser point p is a non-ground laser point. k The slope difference slp_k is greater than the first preset slope difference threshold or the outer ring laser point p k If the difference between the slope difference slp_k and the variance is greater than the second preset slope difference threshold, then the outer ring laser point p k If it is a ground laser point, then it is a non-ground laser point.

[0087] Optionally, the first preset slope difference threshold is preferably tan(5°), and the second slope difference threshold is preferably 1.2 times the variance. When the innermost laser point is a ground laser point, the outermost laser point p k If the slope difference slp_k is greater than tan(5°) or when abs(slp_k-C_slp_j_avg) is greater than 1.2*C_slp_j_var, then the outermost laser point can be determined to be a non-ground laser point; otherwise, it is a ground laser point. If the innermost laser point is a non-ground laser point, then the outermost laser point p... k If the slope difference slp_k is greater than tan(5°) or when abs(slp_k-C_slp_j_avg) is greater than 1.2*C_slp_j_var, then the outer circle laser point can be determined to be a ground laser point; otherwise, it is a non-ground laser point.

[0088] The median filtering process refers to using median replacement to reconfirm the ground laser points on the same circle.

[0089] As an example, in step S1025, the ground laser points on each circle are subjected to median filtering to obtain the final ground laser points, while the remaining laser points are defaulted to non-ground laser points.

[0090] In one embodiment, step S1025, which involves performing median filtering on the ground laser points on the same circle to generate ground laser points and non-ground laser points, includes:

[0091] Step S2501: Traverse each ground laser point on each circle, take the ground laser point as the point to be processed, and obtain the continuous laser points on the same circle starting from the point to be processed according to the preset filtering radius;

[0092] Step S2502: The continuous laser points are binarized and sorted according to the ground segmentation results to obtain a binarized sequence;

[0093] Step S2503: Obtain the median of the binarized sequence. When the median represents a ground laser point, the point to be processed is a ground laser point. When the median represents a non-ground laser point, the point to be processed is a non-ground laser point.

[0094] The preferred filtering radius is 10. Binarization refers to marking ground laser points as 0 and non-ground laser points as 1, or vice versa. Each ground laser point on each ring is traversed, starting with the ground laser point to be processed. The next 10 laser points on the same ring are sequentially acquired, and these 10 points are binarized. The binarization results are then sorted to obtain a binarized sequence. The median of the binarized sequence is then obtained, and the ground segmentation result represented by the median is assigned to the point to be processed. That is, when the median represents a ground laser point, the point to be processed is a ground laser point; when the median represents a non-ground laser point, the point to be processed is a non-ground laser point. This completes the first ground segmentation of the point cloud data, obtaining ground laser points and non-ground laser points. This embodiment uses polar coordinates to segment the 3D laser point cloud data for ground, which can pay more attention to the judgment of local and detailed ground point clouds, effectively distinguishing planar ground similar to walkable cement road surfaces, reducing false judgments, and the algorithm can also achieve real-time processing.

[0095] In one embodiment, step S104, which involves using the lower middle edge region of the image information as the ground region and cropping a ground reference patch, includes:

[0096] Step S1041: Starting from the midpoint of the bottom edge of the image information, obtain a preset number of anchor points at equal intervals to the left and right;

[0097] Step S1042: Traverse each anchor point and, with the anchor point as the center, cut out a ground reference block according to the preset size.

[0098] The ground reference block refers to an image block containing ground information, which serves as a reference standard for secondary judgment of ground laser points.

[0099] As an example, the preset number is preferably 11. In this embodiment, the size of the equal intervals is determined based on the bottom edge width of the image information. Starting from the midpoint of the bottom edge of the image information, a point is obtained as an anchor point for each equal interval moved to the left and right, for a total of 11 points as anchor points. This helps to improve the accuracy and generalization ability of the algorithm. Then, with the anchor points as the center, the image is cropped at a preset size, such as 20*20 pixels, to obtain 11 ground reference patches, denoted as p. _img_l l = 1, 2, ..., L, where L represents the number of ground reference tiles. Based on the premise that the lower and middle edge areas of image information are usually the ground, the 11 ground reference tiles are all ground images. For ease of understanding, Figure 4 is a schematic diagram of the anchor points provided in an embodiment of the present invention.

[0100] In one embodiment, step S105, which involves comparing the projection block containing the projection point with the ground reference block to obtain the target ground point from the ground laser points, includes:

[0101] Step S1051: Traverse each projection point, and with the projection point as the center, extract the corresponding projection block from the image information according to the preset size;

[0102] Step S1052: Compare the projection block with each ground reference block, and calculate the similarity between the projection block and each ground reference block.

[0103] Step S1053: Obtain the minimum similarity value;

[0104] Step S1054: When the minimum similarity value is greater than a preset similarity threshold, the laser point corresponding to the projection point is a non-ground laser point; when the minimum similarity value is less than or equal to the preset similarity threshold, the laser point corresponding to the projection point is a target ground laser point.

[0105] The projection block is an image block cropped from projection points. One projection point corresponds to one image block.

[0106] As an example, in step S1051, this embodiment denotes the projection point on the image information as p. mm = 1, 2, ..., M, where M represents the number of projection points, i.e., the number of ground laser points. The preset size is preferably 20*20 pixels. Each projection point is traversed and denoted as p. m A 20*20 pixel image block is extracted from this point as the center point, and this block is used as the projection block corresponding to the projection point, denoted as p. _img_m .

[0107] As an example, in step S1052, the projection block p _img_m With all ground reference tiles p _img_l Image similarity comparison is performed to obtain several similarity scores. In calculating the similarity between the projected patch and each ground reference patch, this embodiment uses a spectral angle mapping algorithm, and the calculation formula is as follows:

[0108]

[0109] Where θ represents the similarity between the projected patch and the corresponding ground reference patch, x 1、2、3、……、n Represents the pixels on the projection block, y 1、2、3、……、n x represents the pixels on the ground reference tile, and n represents the total number of pixels corresponding to the projected tile and the ground reference tile. n and y n This represents the pixels at the same location as the projected block and the ground reference block.

[0110] The greater the similarity, the less similar the projected block is to the ground reference block; conversely, the smaller the similarity, the more similar the projected block is to the ground reference block.

[0111] As an example, in step S1053, this embodiment obtains the minimum similarity as the ground judgment score of the projection point, denoted as p. _score_m .

[0112] As an example, in step S1054, the similarity threshold is preferably 0.2. When the minimum similarity value of the projection point is greater than the preset similarity threshold, the difference between the projection block and the ground reference block is large, and the laser point corresponding to the projection point is a non-ground laser point. When the minimum similarity value is less than or equal to the preset similarity threshold, the difference between the projection block and the ground reference block is small, and the laser point corresponding to the projection point can be considered as the target ground laser point, and the projection point is the target ground point.

[0113] By traversing each projection point and executing steps S1051 to S1054, all target ground points are obtained, completing the secondary judgment of the ground. These target ground points constitute the ground information. Figure 5 is a schematic diagram of the target ground points provided in this embodiment. This embodiment greatly improves the accuracy of ground detection and can segment out the truly walkable road surface.

[0114] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0115] In one embodiment, the present invention also provides a walkable road surface detection device based on radar and vision fusion, which corresponds one-to-one with the walkable road surface detection method based on radar and vision fusion in the above embodiments. As shown in Figure 6, the walkable road surface detection device based on radar and vision fusion includes an acquisition module 61, a ground segmentation module 62, a projection module 63, a cropping module 64, and a ground acquisition module 65. Detailed descriptions of each functional module are as follows:

[0116] The acquisition module 61 is used to acquire point cloud data collected by the 3D LiDAR on the robot and image information collected by the camera;

[0117] Ground segmentation module 62 is used to perform ground segmentation on the point cloud data to obtain ground laser points and non-ground laser points;

[0118] Projection module 63 is used to project the ground laser points onto the image information to obtain corresponding projection points;

[0119] The cropping module 64 is used to crop a ground reference patch using the lower middle edge region of the image information as the ground region.

[0120] The ground acquisition module 65 is used to compare the projection block where the projection point is located with the ground reference block to acquire the target ground point in the ground laser point.

[0121] Optionally, the ground segmentation module 62 includes:

[0122] The acquisition unit is used to acquire the innermost point cloud data collected by the 3D LiDAR.

[0123] The first segmentation unit is used to perform ground segmentation on the innermost point cloud data according to the laser point height information to obtain the innermost ground laser points and the innermost non-ground laser points.

[0124] The conversion unit is used to convert the point cloud data into polar coordinates and divide the point cloud data into several point cloud sets according to a preset horizontal angle interval. Each point cloud set includes the innermost laser point and the outermost laser point.

[0125] The second segmentation unit is used to traverse each point cloud set and perform ground segmentation on the outer ring laser points according to the slope difference of the laser points to obtain the outer ring ground laser points and the outer ring non-ground laser points.

[0126] The filtering unit is used to perform median filtering on ground laser points on the same circle to generate ground laser points and non-ground laser points.

[0127] Optionally, the filtering unit includes:

[0128] The sub-unit is used to traverse each ground laser point on each circle, and take the ground laser point as the point to be processed, and obtain the continuous laser points on the same circle starting from the point to be processed according to the preset filtering radius.

[0129] The binarization sorting subunit is used to perform binarization processing and sorting on the continuous laser points according to the ground segmentation results to obtain a binarization sequence;

[0130] The filtering subunit is used to obtain the median of the binarized sequence. When the median represents a ground laser point, the point to be processed is a ground laser point. When the median represents a non-ground laser point, the point to be processed is a non-ground laser point.

[0131] Optionally, the interception module 64 includes:

[0132] Anchor point acquisition unit is used to acquire a preset number of anchor points at equal intervals to the left and right, starting from the midpoint of the bottom edge of the image information;

[0133] The cut-out unit is used to traverse each anchor point and cut out a ground reference block according to a preset size, with the anchor point as the center.

[0134] Optionally, the ground acquisition module 65 includes:

[0135] The cropping unit is used to traverse each projection point and, with the projection point as the center, crop the corresponding projection block from the image information according to a preset size.

[0136] A similarity acquisition unit is used to compare the projection block with each ground reference map block and calculate the similarity between the projection block and each ground reference map block.

[0137] The minimum value acquisition unit is used to obtain the minimum similarity value;

[0138] The target ground point acquisition unit is configured to determine that the laser point corresponding to the projection point is a non-ground laser point when the minimum similarity value is greater than a preset similarity threshold, and that the laser point corresponding to the projection point is a target ground laser point when the minimum similarity value is less than or equal to the preset similarity threshold.

[0139] Optionally, the preset size is 20*20 pixels.

[0140] Optionally, the spectral cornerstone mapping algorithm is used to calculate the similarity between the projected patch and each ground reference patch, and the calculation formula is as follows:

[0141]

[0142] Where θ represents the similarity between the projected patch and the corresponding ground reference patch, x 1、2、3、……、n Represents the pixels on the projection block, y 1、2、3、……、n The x represents the pixels on the ground reference tile, where n represents the total number of pixels corresponding to the projected tile and the ground reference tile; n and y n This represents the pixels at the same location as the projected block and the ground reference block.

[0143] Specific limitations regarding the radar and vision fusion-based walkable pavement detection device can be found in the limitations of the radar and vision fusion-based walkable pavement detection method described above, and will not be repeated here. Each module in the aforementioned radar and vision fusion-based walkable pavement detection device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in memory as software, so that the processor can call and execute the corresponding operations of each module.

[0144] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram is shown in Figure 7. The computer device includes a processor, memory, a network interface, and a database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a walkable road surface detection method based on radar and vision fusion.

[0145] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0146] Acquire point cloud data collected by the 3D LiDAR on the robot and image information captured by the camera;

[0147] The point cloud data is segmented into ground laser points and non-ground laser points;

[0148] The ground laser points are projected onto the image information to obtain the corresponding projection points;

[0149] Using the lower middle edge of the image information as the ground region, a ground reference patch is extracted.

[0150] By comparing the projection block where the projection point is located with the ground reference block, the target ground point in the ground laser point is obtained.

[0151] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0152] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0153] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

Claims

1. A method for detecting walkable surfaces based on radar and vision fusion, characterized in that, include: Acquire point cloud data collected by the 3D LiDAR on the robot and image information captured by the camera; The point cloud data is segmented to obtain ground laser points and non-ground laser points; the ground laser points are projected onto the image information to obtain corresponding projection points; a ground reference patch is extracted using the lower middle edge of the image information as the ground region; the projection patch containing the projection point is compared with the ground reference patch to obtain the target ground point among the ground laser points; wherein, the segmentation of the point cloud data to obtain ground laser points and non-ground laser points includes: acquiring the innermost circle point cloud data collected by 3D LiDAR; segmenting the innermost circle point cloud data according to the laser point height information to obtain the innermost circle ground laser points and the innermost circle non-ground laser points; converting the point cloud data into polar coordinates, dividing the point cloud data into several point cloud sets according to a preset horizontal angle interval, each point cloud set including the innermost circle laser points and the outermost circle laser points; traversing each point cloud set, segmenting the outermost circle laser points according to the laser point slope difference to obtain the outermost circle ground laser points and the outermost circle non-ground laser points. Ground laser points; median filtering is performed on ground laser points on the same circle to generate ground laser points and non-ground laser points; the median filtering of ground laser points on the same circle to generate ground laser points and non-ground laser points includes: traversing each ground laser point on each circle, taking the ground laser point as the point to be processed, and obtaining continuous laser points on the same circle starting from the point to be processed according to a preset filtering radius; binarizing and sorting the continuous laser points according to the ground segmentation result to obtain a binarized sequence; obtaining the median of the binarized sequence, when the median represents a ground laser point, the point to be processed is a ground laser point, and when the median represents a non-ground laser point, the point to be processed is a non-ground laser point; the step of using the middle lower edge region of the image information as the ground region and cropping the ground reference patch includes: starting from the midpoint of the bottom edge of the image information, obtaining a preset number of anchor points at equal intervals to the left and right; traversing each anchor point, and cropping the ground reference patch according to a preset size with the anchor point as the center.

2. The walkable road surface detection method based on radar and vision fusion as described in claim 1, characterized in that, The step of comparing the projection block where the projection point is located with the ground reference map block to obtain the target ground point among the ground laser points includes: traversing each projection point, taking the projection point as the center, and extracting the corresponding projection block from the image information according to a preset size; comparing the projection block with each ground reference map block, and calculating the similarity between the projection block and each ground reference map block; obtaining the minimum similarity value; when the minimum similarity value is greater than a preset similarity threshold, the laser point corresponding to the projection point is a non-ground laser point, and when the minimum similarity value is less than or equal to the preset similarity threshold, the laser point corresponding to the projection point is a target ground laser point.

3. The walkable road surface detection method based on radar and vision fusion as described in claim 1 or 2, characterized in that, The preset size is 20*20 pixels.

4. The walkable road surface detection method based on radar and vision fusion as described in claim 1 or 2, characterized in that, The similarity between the projected patch and each ground reference patch is calculated using the spectral cornerstone mapping algorithm, and the calculation formula is as follows: ;in, This represents the similarity between the projected tile and the corresponding ground reference tile, where x1, 2, 3, ..., n represent the pixels on the projected tile, and y1, 2, 3, ..., n represent the pixels on the ground reference tile. This represents the total number of pixels corresponding to the projected block and the ground reference block; xn and yn represent the pixels at the same position as the projected block and the ground reference block.

5. A walkable road surface detection device based on radar and vision fusion, characterized in that, The device includes: an acquisition module for acquiring point cloud data collected by a 3D LiDAR on a robot and image information acquired by a camera; specifically including: acquiring the innermost circle point cloud data collected by the 3D LiDAR; performing ground segmentation on the innermost circle point cloud data according to the laser point height information to obtain the innermost circle ground laser points and the innermost circle non-ground laser points; converting the point cloud data into polar coordinate representation, and dividing the point cloud data into several point cloud sets according to a preset horizontal angle interval, each point cloud set including the innermost circle laser points and the outer circle laser points; traversing each point cloud set, and performing ground segmentation on the outer circle laser points according to the laser point slope difference to obtain the outer circle ground laser points and the outer circle non-ground laser points; performing median filtering on the ground laser points on the same circle to generate ground laser points and non-ground laser points, specifically including: traversing each ground laser point on each circle, taking the ground laser point as the point to be processed, and acquiring the continuous laser points on the same circle starting from the point to be processed according to a preset filtering radius. The system comprises the following modules: a ground segmentation module for dividing the point cloud data into ground laser points and non-ground laser points; a projection module for projecting the ground laser points onto the image information to obtain corresponding projection points; a cropping module for cropping ground reference patches using the lower middle edge region of the image information as the ground region, specifically including: obtaining a preset number of anchor points at equal intervals to the left and right from the midpoint of the bottom edge of the image information; traversing each anchor point and cropping ground reference patches according to a preset size with the anchor point as the center; and a ground acquisition module for comparing the projection patch where the projection point is located with the ground reference patch to obtain the target ground point among the ground laser points.

6. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the walkable road surface detection method based on radar and vision fusion as described in any one of claims 1 to 4.

7. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the walkable road surface detection method based on radar and vision fusion as described in any one of claims 1 to 4.

Images