A method for detecting mud-slipped pavement in mining areas based on image and point cloud fusion

Abstract

This invention relates to a method for detecting frost-prone roads in mining areas based on image and point cloud fusion, belonging to the field of autonomous driving environmental monitoring technology. It solves the problem that existing monitoring methods cannot accurately detect the location and specific attributes of frost-prone areas. The method combines images with point cloud data, utilizing the characteristics of point clouds to extract attributes of the frost-prone area, supplementing the lack of depth information in images. This allows for precise detection of frost-prone areas, fully integrating the advantages of images and point clouds to achieve a quantitative description of the frost-prone area, providing quantitative indicators for input to the control terminal of autonomous vehicles.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving environment monitoring technology, and relates to a method for detecting muddy road surfaces in mining areas based on image and point cloud fusion. Background Technology

[0002] Road surface frost detection is crucial for the normal operation of autonomous vehicles. Chinese patent CN109653185B describes a method that uses a micro-weather station and data acquisition and transmission device to collect data from multiple sources on the roadbed. The obtained data is then converted into soil dynamic modulus, net evaporation rate, salt content, and cumulative roadbed deformation for analysis, enabling the monitoring of road surface frost, roadbed settlement, and road cracking after long-term operation of highways in saline-alkali areas. However, this method is only applicable to structured scenarios such as highways in saline-alkali areas and not to unstructured scenarios such as mining areas. Similarly, Chinese patent CN212294203U uses electrodes to facilitate water migration and convergence, allowing water in the roadbed soil to drain through drainage pipes, preventing water retention in the roadbed and reducing frost. However, if this device fails in specific scenarios, it does not provide an effective detection method for frost after it occurs.

[0003] With the increasing demand for autonomous driving in mining areas, accurate detection of unstructured road surfaces in these areas has become an urgent need. When autonomous vehicles travel on unstructured roads, the wheels rolling over the soft surface can cause slurry heaving, which severely affects the normal operation of autonomous vehicles. Therefore, the ability of autonomous vehicles to accurately detect the location of slurry heaving and the specific attributes of the heaving area (such as heaving height and area) is crucial for their decision-making, control, and path planning. Thus, accurate detection of slurry-affected road surfaces in mining areas is particularly pressing. Summary of the Invention

[0004] In view of the above problems, the present invention provides a method for detecting mudslide road surfaces in mining areas based on image and point cloud fusion, in order to solve the problem that existing monitoring methods cannot accurately detect the location of mudslide road surfaces and the specific attributes of mudslide areas.

[0005] This invention provides a method for detecting mudslide-prone road surfaces in mining areas based on image and point cloud fusion, specifically including the following steps:

[0006] Acquire synchronized image and point cloud data;

[0007] Pixels of the frost heaving area are obtained from the acquired image;

[0008] Expand all pixels in the frost-affected area;

[0009] Binarize and dilate all pixels in the frost heave region to obtain the pixel region of the frost heave region;

[0010] Based on the connected region threshold and the ratio of the shortest distance between the boundaries of adjacent pixel regions to the center distance, adjacent pixel regions with the same pixel value are grouped into a pixel set and marked to form a connected region.

[0011] Pixel regions that do not meet the aforementioned connectivity conditions are not processed, and disconnected pixel regions are obtained.

[0012] Connected and non-connected pixel regions form the target frost heave area;

[0013] Extract the contour of the target frost heave area;

[0014] The point cloud data is mapped to the image of the synchronized data to obtain the marker index of each point in the road surface frost heave area;

[0015] After mapping the point cloud data onto the image, and combining it with the extracted frost heave region contour, the point cloud within and on the frost heave region contour is selected as the original point cloud of the frost heave region based on the positional relationship between the pixel points of the point cloud data mapped onto the image and the pixel points of the frost heave region contour, and based on the marker index of the point cloud mapped in the image.

[0016] The acquired frosting region with original point cloud data is meshed, dividing the frosting region into... n The side length is m The grid is used to determine the planar road surface by calculating the planar normal vectors of the original point cloud; the average height of a single grid of all original point clouds within each grid is calculated. K 1, K 2,…, K n and the overall average height of all original point clouds in the entire frost heave area. L The average height difference of each grid is obtained based on the average height of a single grid cell and the overall average height. H i , H i =| LK i |, K i For the first i The average height of a single grid cell. i ∈1,2,…, n ; Calculate the average height difference of point clouds between two adjacent grids. h , h =| K i - K j |, K j For the first iThe average height of a single grid cell in adjacent grid cells. j ∈1,2,…, n The original point cloud is selected based on the following conditions: if the average height difference between two adjacent grid points is less than the average height difference between the point clouds... h Less than the average height difference between the two adjacent grids H i and H j If the average value is obtained, the original point cloud in the two adjacent grids is used; the height difference of the point cloud in all two adjacent grids in the frost heave area is traversed, all the original point clouds that are used are recorded, and the height value of the highest point in all the original point clouds is selected as the maximum height of the frost heave area.

[0017] The area of ​​the frost heave zone is obtained based on the number and size of the grid.

[0018] Optionally, while the autonomous vehicle acquires image and point cloud data in real time, it also records the timestamps of the image and point cloud data, and selects the image and point cloud data with the most recent timestamp as the synchronization data.

[0019] Optionally, when acquiring the pixels of the frost heave area, the frost heave area of ​​the road surface in the image of the synchronized data is first segmented;

[0020] The method for segmenting pixels in the frost heave region is as follows:

[0021] Obtain a training image set, label the frost heave areas in each image of the training image set, and use the backpropagation algorithm to segment and train the images to obtain an image segmentation model;

[0022] The image segmentation model is used to extract the frost heave region from the images in the synchronous data, and the extracted frost heave region pixels are obtained.

[0023] Optionally, when training the image segmentation model, a network with an encoder-decoder architecture is used, where the encoder structure downsamples layers 1, 2, and 8 using one-dimensional dilated convolutions; and the decoder structure upsamples using deconvolutions with a stride of 2.

[0024] Optionally, the connectivity condition of the connected region is as follows: set a connected region threshold; calculate the ratio of the nearest boundary distance d and the center distance D of two adjacent pixel regions; if the ratio is less than the connected region threshold, obtain the pixel values ​​of the two adjacent pixel regions; if the pixel values ​​are the same, form a pixel set of adjacent pixel regions with the same pixel values ​​and mark them to form a connected region.

[0025] Wherein, the nearest boundary distance d and the center distance D between two adjacent pixel regions are as follows: For two adjacent pixel regions A and B, the straight line ab between the center point a of pixel region A and the center point b of pixel region B is the center distance D between the adjacent pixel regions, and the line connecting the intersection point c of the straight line ab with the boundary of pixel region A and the intersection point d of the straight line ab with the boundary of pixel region B is the intersection line cd, which is the nearest boundary distance d between the adjacent target pixel regions.

[0026] Optionally, the specific steps for mapping point cloud data to the image of the synchronized data and obtaining the marker index of each point in the road surface slurry area are as follows: calibrating the internal parameters of the image acquisition device and the external parameters between the image acquisition device and the point cloud data acquisition device; mapping the point cloud data to the image using the calibrated internal and external parameters, and marking and indexing each point in the point cloud data.

[0027] Optionally, the internal parameters include five internal matrix parameters of the image device; the external parameters include three rotation parameters and three translation parameters between the image device mounting position and the point cloud data device mounting position.

[0028] Optionally, the grid size is 0.3 × 0.3 m.

[0029] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0030] (1) The present invention segments the image and processes the image in combination with point cloud data. It utilizes the characteristics of point cloud to extract the attributes of the frost heave area, making up for the lack of depth information in the image. It can accurately detect the frost heave area and fully integrates the advantages of image and point cloud.

[0031] (2) The present invention performs dilation processing on the segmented image pixel regions; then calculates the center distance D and the shortest boundary distance d of each pair of adjacent dilated target pixel regions, sets a certain connectivity threshold, and if the ratio of the shortest boundary distance to the center distance D of the target region is less than the connectivity threshold, then the corresponding target pixel regions are connected, thereby achieving the purpose of optimizing the segmentation effect;

[0032] (3) Based on the three-dimensional information obtained from point cloud, this invention rasterizes the frost heave area, calculates the average height of the point cloud in each small grid, and compares the difference in the average height of adjacent grids. If the difference in the average height is less than a set threshold height, the grid is retained and its average height is recorded. If the difference in the average height of adjacent grids is greater than the set height, the grid is directly merged to calculate the average height of the point cloud in the large grid. At the same time, the average height of each adjacent grid is iteratively calculated to form an adaptive average height plane. This is then combined with the normal vector calculated from the original point cloud to realize the acquisition of the relative height and area output of the frost heave area, realizing a quantitative description of the frost heave area and providing quantitative indicators for the input of the control terminal of the unmanned vehicle. Attached Figure Description

[0033] The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of the invention.

[0034] Figure 1 This is a flowchart of the road surface frost heave detection method according to the present invention. Detailed Implementation

[0035] To better understand the above-described objectives, features, and advantages of the present invention, the invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, unless otherwise specified, the embodiments of the present invention and the features thereof can be combined with each other. Furthermore, the present invention can be implemented in other ways different from those described herein; therefore, the scope of protection of the present invention is not limited to the specific embodiments disclosed below.

[0036] A specific embodiment of the present invention, such as Figure 1 A method for detecting mud-blown pavement in mining areas based on image and point cloud fusion is disclosed, which specifically includes the following steps:

[0037] S1: Acquire synchronized image and point cloud data;

[0038] While acquiring images and point cloud data in real time, the autonomous vehicle records the timestamps of the images and point cloud data, and selects the images and point cloud data with the most recent timestamps as the synchronization data.

[0039] Optionally, the image is acquired by a camera; the point cloud is acquired by a LiDAR.

[0040] S2: Obtain pixels of the frost heave area based on the image;

[0041] The road surface frost heave area in the synchronous data of step S1 is segmented out;

[0042] Alternatively, the method for segmenting the pixels in the frost heave region is as follows:

[0043] Obtain a training image set, label the frost heave areas in each image of the training image set, and use the backpropagation algorithm to segment and train the images to obtain an image segmentation model;

[0044] When training the image segmentation model, a network with an encoder-decoder architecture is adopted. The encoder structure performs downsampling three times. Preferably, downsampling is performed on layers 1, 2, and 8. One-dimensional dilated convolution is used to increase the receptive field, ensuring accuracy while reducing the number of parameters. Furthermore, a nonlinear function is added between the two one-dimensional convolution layers to increase the learning ability during training. The decoder uses deconvolution with a stride of 2 for upsampling to segment different areas of the road surface and finally obtain the frost heave area.

[0045] The image segmentation model is used to extract the frost heave region from the image obtained in step S1, and the pixels of the extracted frost heave region are obtained.

[0046] S3: Extract the outline of the frost heaving area;

[0047] S3-1: Pixel dilation processing in the frost heave region; The local maxima operator in the pixel neighborhood is used to dilate all pixels in the frost heave region obtained in step S2.

[0048] S3-2: Formation of the target frost heave zone:

[0049] The target turbulence region includes connected regions and disconnected pixel regions.

[0050] The formation of connected and disconnected pixel regions specifically includes the following steps:

[0051] Binarize and dilate all pixels in the frost heave region to obtain the pixel region of the frost heave region;

[0052] Based on the connected region threshold and the ratio of the shortest distance between the boundaries of adjacent pixel regions to the center distance, adjacent pixel regions with the same pixel value are grouped into a pixel set and marked to form a connected region.

[0053] Specifically, a connected region threshold is set; the ratio of the nearest boundary distance d and the center distance D between two adjacent pixel regions is calculated; if the ratio is less than the connected region threshold, the pixel values ​​of the two adjacent pixel regions are obtained; if the pixel values ​​are the same, the adjacent pixel regions with the same pixel values ​​are grouped into a pixel set and marked to form a connected region.

[0054] Wherein, the nearest boundary distance d and the center distance D between two adjacent pixel regions are as follows: For two adjacent pixel regions A and B, the straight line ab between the center point a of pixel region A and the center point b of pixel region B is the center distance D between the adjacent pixel regions, and the line connecting the intersection point c of the straight line ab with the boundary of pixel region A and the intersection point d of the straight line ab with the boundary of pixel region B is the intersection line cd, which is the nearest boundary distance d between the adjacent target pixel regions.

[0055] Pixel regions that do not meet the aforementioned connectivity conditions are not processed; that is, they are non-connected pixel regions.

[0056] S3-3: Contour Extraction of the Frost Heating Area: Extract the contour of the target frost heating area;

[0057] S4: Map the point cloud data to the image of the synchronized data in step S1, and obtain the marker index of each point in the road surface slurry area;

[0058] The internal parameters of the image device and the external parameters between the image device and the point cloud data device are calibrated; the point cloud data is mapped to the image using the calibrated internal and external parameters, and each point in the point cloud data is labeled and indexed.

[0059] The internal parameters include the internal matrix parameters of the five image devices; the external parameters include three rotation parameters and three translation parameters between the installation position of the image devices and the installation position of the point cloud data devices.

[0060] S5: Obtain the original point cloud of the frost heave area;

[0061] After mapping the point cloud data onto the image, and combining it with the contour of the frost heave area extracted in step S3, the point cloud in the frost heave area is used as the original point cloud based on the positional relationship between the pixels of the point cloud data mapped onto the image and the pixels of the frost heave area contour, and the point cloud in the image marked with indexes within and on the frost heave area contour.

[0062] S6: Obtain the maximum height and area of ​​the frost heave zone;

[0063] The mudslide area with the original point cloud obtained in step S5 is meshed, and the mudslide area is divided into... n The side length is m The grid is used to determine the planar road surface by calculating the planar normal vectors of the original point cloud; the average height of a single grid of all original point clouds within each grid is calculated. K 1, K 2,…, K n and the overall average height of all original point clouds in the entire frost heave area. LThe average height difference of each grid is obtained based on the average height of a single grid cell and the overall average height. H i , H i =| LK i |, K i For the first i The average height of a single grid cell. i ∈1,2,…, n ; Calculate the average height difference of point clouds between two adjacent grids. h , h =| K i - K j |, K j For the first i The average height of a single grid cell in adjacent grid cells. j ∈1,2,…, n The original point cloud is selected based on the following conditions: if the average height difference between two adjacent grid points is less than the average height difference between the point clouds... h Less than the average height difference between the two adjacent grids H i and H j If the average height difference between the two adjacent grid points is the mean, then the original point cloud within those two adjacent grids is used; if the average height difference between the point clouds of the two adjacent grids is less than the mean height difference, then the original point cloud within those two adjacent grids is used. h If the height difference is greater than the average of the height difference between adjacent grids, it is not accepted. Traverse the height difference of the point cloud of all two adjacent grids in the frost heave area, record all the original point clouds that are accepted, and select the height value of the highest point in all the original point clouds accepted as the maximum height of the frost heave area.

[0064] The area of ​​the frost heave region is obtained based on the number of grid cells and the grid size; that is, if the number of grid cells in the frost heave region is n, the area of ​​the frost heave region is... n × m × m .

[0065] Preferably, the side length of the grid is 0.3 × 0.3 m.

[0066] It is understood that the present invention uses a mining unmanned vehicle, and a camera and lidar are installed on the front of the mining unmanned vehicle, which can perceive the forward environment when the vehicle is driving in real time. Based on the installed camera and lidar sensors, the detection of road surface frost heave in the mining area is carried out.

[0067] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion, characterized in that, Specifically, the steps include the following: Acquire synchronized image and point cloud data; Pixels of the frost heaving area are obtained from the acquired image; Expand all pixels in the frost-affected area; Binarize and dilate all pixels in the frost heave region to obtain the pixel region of the frost heave region; Based on the connected region threshold and the ratio of the shortest distance to the center distance between adjacent pixel regions, adjacent pixel regions with the same pixel value are grouped into a pixel set and identified to form a connected region. Specifically: The connectivity condition for a connected region is set as follows: Set a connected region threshold; calculate the ratio of the nearest boundary distance and the center distance between two adjacent pixel regions; if the ratio is less than the connected region threshold, obtain the pixel values ​​of the two adjacent pixel regions; if the pixel values ​​are the same, form a pixel set by combining the adjacent pixel regions with the same pixel values ​​and mark them to form a connected region. Pixel regions that do not meet the aforementioned connectivity conditions are not processed, and disconnected pixel regions are obtained. Connected and non-connected pixel regions form the target frost heave region; Extract the contour of the target frost heave area; The point cloud data is mapped to the image of the synchronized data to obtain the marker index of each point in the road surface frost heave area; After mapping the point cloud data onto the image, and combining it with the extracted frost heave region contour, the point cloud within and on the frost heave region contour is selected as the original point cloud of the frost heave region based on the positional relationship between the pixel points of the point cloud data mapped onto the image and the pixel points of the frost heave region contour, and based on the marker index of the point cloud mapped in the image. The acquired frosting region with original point cloud data is meshed, dividing the frosting region into... n The side length is m The grid is used to determine the planar road surface by calculating the planar normal vectors of the original point cloud; the average height of a single grid of all original point clouds within each grid is calculated. K 1, K 2,…, K n and the overall average height of all original point clouds in the entire frost heave area. L The average height difference of each grid is obtained based on the average height of a single grid cell and the overall average height. H i , H i =| LK i |, K i For the first i The average height of a single grid cell. i ∈1,2,…, n ; Calculate the average height difference of point clouds between two adjacent grids. h , h =| K i - K j |, K j For the first i The average height of a single grid cell in adjacent grid cells. j ∈1,2,…, n ; The original point cloud is selected based on the following condition: if the average height difference between two adjacent grid points is less than the average height difference between the point clouds... h Less than the average height difference between the two adjacent grids H i and H j If the average value is obtained, the original point cloud in the two adjacent grids is used; the height difference of the point cloud in all two adjacent grids in the frost heave area is traversed, all the original point clouds that are used are recorded, and the height value of the highest point in all the original point clouds is selected as the maximum height of the frost heave area. The area of ​​the frost heave zone is obtained based on the number and size of the grid.

2. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 1, characterized in that, While acquiring images and point cloud data in real time, the autonomous vehicle records the timestamps of the images and point cloud data, and selects the images and point cloud data with the most recent timestamps as the synchronization data.

3. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 1, characterized in that, When acquiring the pixels of the frost heave area, the frost heave area of ​​the road surface in the image of the synchronous data is first segmented; The method for segmenting pixels in the frost heave region is as follows: Obtain a training image set, label the frost heave areas in each image of the training image set, and use the backpropagation algorithm to segment and train the images to obtain an image segmentation model; The image segmentation model is used to extract the frost heave region from the images in the synchronous data, and the extracted frost heave region pixels are obtained.

4. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 3, characterized in that, When training the image segmentation model, a network with an encoder-decoder architecture is used. The encoder structure downsamples layers 1, 2, and 8 using one-dimensional dilated convolutions; the decoder uses deconvolutions with a stride of 2 for upsampling.

5. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 1, characterized in that, The nearest boundary distance d and center distance D between two adjacent pixel regions are as follows: For two adjacent pixel regions A and B, the straight line ab between the center point a of pixel region A and the center point b of pixel region B is the center distance D between the adjacent pixel regions, and the line connecting the intersection point c of the straight line ab with the boundary of pixel region A and the intersection point d of the straight line ab with the boundary of pixel region B is the intersection line cd, which is the nearest boundary distance d between the adjacent target pixel regions.

6. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 1, characterized in that, The specific steps for mapping point cloud data to the image of the synchronized data and obtaining the label index of each point in the road surface slurry area are as follows: calibrate the internal parameters of the image acquisition device and the external parameters between the image acquisition device and the point cloud data acquisition device; map the point cloud data to the image using the calibrated internal and external parameters, and label and index each point in the point cloud data.

7. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 6, characterized in that, The internal parameters include five internal matrix parameters of the image device; the external parameters include three rotation parameters and three translation parameters between the installation position of the image device and the installation position of the point cloud data device.

8. The method for detecting mud-blown road surfaces in mining areas based on image and point cloud fusion according to claim 1, characterized in that, The grid size is 0.3 × 0.3 m.

Images