Pavement defect detection method and device, terminal and storage medium

Abstract

The invention provides a pavement defect detection method and device, a terminal and a storage medium, and belongs to the technical field of image processing, and the method comprises the steps: obtaining a pavement point cloud and an RGB image corresponding to a target pavement; processing the road surface point cloud and the RGB image to obtain RGBD multi-source data; inputting the RGBD multi-source data into a trained defect detection segmentation network, processing the RGBD multi-source data through the defect detection segmentation network, and outputting a prediction result of the pavement defect; and determining a corresponding pavement defect point cloud based on the prediction result, and determining the position, area and depth of the pavement defect based on the pavement defect point cloud. The RGBD multi-source data is generated by combining the point cloud data and the RGB image, defect prediction is performed on the RGBD multi-source data by using the trained defect detection segmentation network, and the specific position, area and depth of the defect are determined according to the prediction result, so that the accuracy of pavement defect detection can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing, and particularly to a road surface defect detection method, device, terminal and storage medium. Background Art

[0002] In existing road surface defect detection solutions, the method based on image processing is most widely used. This method collects road surface images through a camera, and then uses algorithms such as edge detection and morphological processing to identify obvious road surface defects. This method is intuitive and low-cost, but there are certain limitations. It is easily affected by lighting and weather conditions, resulting in difficulty in ensuring the accuracy of road surface defect detection.

[0003] With the rapid development of deep learning neural networks in recent years, especially the application of convolutional neural networks (CNNs) in the field of object recognition and classification of images, the automation level of detection has been greatly improved. Through the training of a large amount of labeled data, the deep learning model can automatically learn to identify different types of road surface defects and perform accurate classification. However, this method still processes based on image data and is still limited by lighting and weather conditions, and it is still difficult to ensure the accuracy of road surface defect detection.

[0004] Therefore, there are defects in the existing technology and it needs to be improved and developed. Summary of the Invention

[0005] The technical problem to be solved by the present invention is to provide a road surface defect detection method, device, terminal and storage medium for the above-mentioned defects of the existing technology, aiming to solve the problem that it is difficult to ensure the accuracy of road surface defect detection in the existing technology.

[0006] The technical solution adopted by the present invention to solve the technical problem is as follows:

[0007] In a first aspect, an embodiment of the present invention provides a road surface defect detection method, and the method includes:

[0008] Obtain the road surface point cloud and RGB image corresponding to the target road surface;

[0009] Process the road surface point cloud and the RGB image to obtain RGBD multi-source data;

[0010] Input the RGBD multi-source data into a trained defect detection segmentation network, and after being processed by the defect detection segmentation network, output the prediction result of the road surface defect;

[0011] Determine the corresponding road surface defect point cloud based on the prediction result, and determine the position, area and depth of the road surface defect based on the road surface defect point cloud.

[0012] In one implementation, before obtaining the point cloud of the target road surface, it further includes:

[0013] Obtain the regional point cloud collected by the lidar for the target area, where the target area includes the target road surface and the surrounding area of the target road surface;

[0014] Use the road surface point cloud segmentation algorithm to segment the regional point cloud to obtain the point cloud of the target road surface.

[0015] In one implementation, processing the road surface point cloud and the RGB image to obtain RGBD multi-source data includes:

[0016] Use a preset cylinder model to divide the road surface point cloud into normal point cloud and outlier point cloud;

[0017] Use the normal point cloud to fit the road surface to obtain a fitted road surface, calculate the distance from the road surface point cloud to the fitted road surface to obtain the depth value of the road surface point cloud;

[0018] After interpolating the depth value of the road surface point cloud, obtain the interpolated depth value of the road surface point cloud, and generate a depth map corresponding to the road surface point cloud from the interpolated depth value of the road surface point cloud;

[0019] After de-distorting the RGB image, obtain a first RGB image, register it with the depth map to obtain RGBD multi-source data.

[0020] In one implementation, the defect detection and segmentation network includes an input layer, an encoder module, a multi-modal fusion module, a decoder module, and an output layer; input the RGBD multi-source data into the trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, output the prediction result of road surface defects, including:

[0021] Input the RGBD multi-source data into the input layer of the defect detection and segmentation network, and transmit it to the encoder module through the input layer;

[0022] Use the encoder module to process the first RGB image and the depth map in the RGBD multi-source data to obtain several image features and depth features;

[0023] Input all the image features and the depth features into the multi-modal fusion module, use the multi-modal fusion module to perform channel attention processing and spatial attention processing on each image feature and each depth feature respectively, and perform feature enhancement and fusion through exponential operations to obtain several fusion features;

[0024] Input all the fusion features into the decoder module, and process all the fusion features through the decoder module to obtain the prediction result of the road surface defect;

[0025] Output the prediction result through the output layer.

[0026] In one implementation, the encoder module includes a Unet encoder, a Mamba encoder, and several visual state space modules. The Unet encoder includes several convolutional layers, and the convolutional layers are used to extract image features of different scales through convolution operations. The Mamba encoder includes an initial layer and several image patch embedding layers. The initial layer is used to normalize the depth image, and the image patch embedding layers are used to divide the input depth image into multiple image patches and map each image patch to a feature vector of a preset dimension. The visual state space module is used to align the depth features output by the corresponding image patch embedding layer with the image features output by the corresponding convolutional layer in the Unet encoder in terms of size.

[0027] In one implementation, determining the position, area, and depth of the road surface defect based on the road surface defect point cloud includes:

[0028] Calculate the longitude mean, latitude mean, and height mean of all the point clouds in the road surface defect point cloud as the position coordinates of the road surface defect;

[0029] Project the road surface defect point cloud onto a two-dimensional plane to obtain two-dimensional plane data, and use the convex hull algorithm to calculate the two-dimensional plane data to obtain the boundary area of the road surface point cloud;

[0030] Calculate based on the boundary area to obtain the area of the road surface defect;

[0031] Calculate the height difference between the height of the road surface defect point cloud and the height of the fitted road surface to obtain the depth of the road surface defect.

[0032] In one implementation, the method further includes:

[0033] Generate a condition assessment report of the road defect based on the position, area, and depth of the road surface defect.

[0034] In a second aspect, an embodiment of the present invention further provides a road surface defect detection device, including:

[0035] A data acquisition module, configured to acquire the road surface point cloud and the RGB image corresponding to the target road surface;

[0036] A multi-source data generation module, configured to process the road surface point cloud and the RGB image to obtain RGBD multi-source data;

[0037] A defect prediction module, configured to input the RGBD multi-source data into a trained defect detection and segmentation network, and output a prediction result of road surface defects after being processed by the defect detection and segmentation network;

[0038] A defect information generation module, configured to determine corresponding road surface defect point clouds based on the prediction result, and determine the position, area, and depth of the road surface defects based on the road surface defect point clouds.

[0039] In a third aspect, an embodiment of the present invention further provides a terminal, where the terminal includes: a memory, a processor, and a road surface defect detection program stored on the memory and executable on the processor. When the road surface defect detection program is executed by the processor, the steps of the road surface defect detection method described above are implemented.

[0040] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, where the computer-readable storage medium stores a road surface defect detection program, and the road surface defect detection program can be executed to implement the steps of the road surface defect detection method described above.

[0041] Advantages of the present invention: The present invention obtains a road surface point cloud and an RGB image corresponding to a target road surface; processes the road surface point cloud and the RGB image to obtain RGBD multi-source data; inputs the RGBD multi-source data into a trained defect detection and segmentation network, and outputs a prediction result of road surface defects after being processed by the defect detection and segmentation network; determines corresponding road surface defect point clouds based on the prediction result, and determines the position, area, and depth of the road surface defects based on the road surface defect point clouds. By combining point cloud data and RGB images to generate RGBD multi-source data, using a trained defect detection and segmentation network to perform defect prediction on it, and determining the specific position, area, and depth of the defects according to the prediction result, the present invention can effectively improve the accuracy of road surface defect detection. Description of the Drawings

[0042] Figure 1 is a flowchart of a preferred embodiment of the road surface defect detection method in the present invention.

[0043] Figure 2 is a schematic diagram of the generation process of RGBD multi-source data in the present invention.

[0044] Figure 3 is a schematic diagram of the process of image processing using a defect detection and segmentation network in the present invention.

[0045] Figure 4 is a schematic diagram of the data processing flow of the multi-modal fusion module in the present invention.

[0046] Figure 5 is a flowchart of generating a status evaluation report of road defects in the present invention.

[0047] Figure 6 It is a schematic structural diagram of a preferred embodiment of the road surface defect detection device in the present invention.

[0048] Figure 7 It is a principle block diagram of the terminal of the present invention. Specific embodiments

[0049] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further describes the present invention in detail with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not used to limit the present invention.

[0050] In the existing road surface defect detection solutions, the method based on image processing is the most widely used. This method collects road surface images through a camera, and then uses algorithms such as edge detection and morphological processing to identify obvious road surface defects. This method is intuitive and has a low cost, but there are certain limitations. It is easily affected by lighting and weather conditions, resulting in difficulty in ensuring the accuracy of road surface defect detection.

[0051] With the rapid development of deep learning neural networks in recent years, especially the application of convolutional neural networks (CNNs) in the field of object recognition and classification of images, the automation level of detection has been greatly improved. Through the training of a large amount of labeled data, the deep learning model can automatically learn to identify different types of road surface defects and perform accurate classification. However, this method still processes based on image data and is still limited by lighting and weather conditions, and it is still difficult to ensure the accuracy of road surface defect detection.

[0052] Aiming at the above defects of the prior art, the present invention provides a road surface defect detection method, device, terminal and storage medium, belonging to the technical field of image processing. The method includes: obtaining the road surface point cloud and RGB image corresponding to the target road surface; processing the road surface point cloud and the RGB image to obtain RGBD multi-source data; inputting the RGBD multi-source data into a trained defect detection segmentation network, and after being processed by the defect detection segmentation network, outputting the prediction result of the road surface defect; determining the corresponding road surface defect point cloud based on the prediction result, and determining the position, area and depth of the road surface defect based on the road surface defect point cloud. The present invention combines point cloud data and RGB images to generate RGBD multi-source data, uses a trained defect detection segmentation network to perform defect prediction on it, and determines the specific position, area and depth of the defect according to the prediction result, which can effectively improve the accuracy of road surface defect detection.

[0053] Please refer to Figure 1 , the road surface defect detection method described in the embodiment of the present invention includes the following steps:

[0054] Step S100: Obtain the road surface point cloud and RGB image corresponding to the target road surface.

[0055] Specifically, the road surface point cloud can be obtained by a lidar collecting the road surface and its surrounding environment, and the RGB image can be obtained by devices such as a surveillance camera or a panoramic camera shooting the target road surface. Since the lidar is not affected by illumination and weather conditions, it can stably collect high-precision point cloud data. These point cloud data record the three-dimensional shape and minute changes of the road surface in detail, providing a reliable basis for the accurate identification of road surface defects.

[0056] In one implementation, before obtaining the road surface point cloud corresponding to the target road surface, it further includes:

[0057] Obtain the regional point cloud collected by the lidar for the target area, where the target area includes the target road surface and the surrounding area of the target road surface;

[0058] Use a road surface point cloud segmentation algorithm to segment the regional point cloud to obtain the road surface point cloud corresponding to the target road surface.

[0059] Specifically, the lidar is a Riegl single-line lidar, and the road surface point cloud segmentation algorithm is the Cloth SimulationFilter (CSF) algorithm.

[0060] Please refer to Figure 1 , the road surface defect detection method described in the embodiments of the present invention includes the following steps:

[0061] Step S200: Process the road surface point cloud and the RGB image to obtain RGBD multi-source data.

[0062] Specifically, the RGBD multi-source data is composite image data combining the RGB image and the depth map.

[0063] In one implementation, processing the road surface point cloud and the RGB image to obtain RGBD multi-source data includes:

[0064] Use a preset cylinder model to divide the road surface point cloud into normal point cloud and outlier point cloud;

[0065] Use the normal point cloud for fitting the road surface to obtain a fitted road surface, calculate the distance from the road surface point cloud to the fitted road surface to obtain the depth value of the road surface point cloud;

[0066] After interpolating the depth value of the road surface point cloud, obtain the interpolated depth value of the road surface point cloud, and generate a depth map corresponding to the road surface point cloud from the interpolated depth value of the road surface point cloud;

[0067] After performing distortion removal processing on the RGB image, a first RGB image is obtained, and it is registered with the depth map to obtain RGBD multi-source data.

[0068] Specifically, the generation process of the RGBD multi-source data is as Figure 2 shown. The cylinder model is a geometric model used for road surface point cloud segmentation. Its core idea is to define a cylindrical space to distinguish normal points (road surface points) from outlier points (non-road surface points). Specifically, the central axis of the cylinder is aligned with the point cloud acquisition direction, and the radius and height of the cylinder are used to limit the distribution range of road surface points. Road surface points are usually distributed near the cylindrical surface, while outlier points (such as buildings, trees, vehicles, etc.) are distributed outside or on top of the cylindrical surface. The preset cylinder model can effectively distinguish normal point clouds and outlier point clouds. The moving least squares (MLS) method is used to fit the road surface. The interpolation processing of the depth values of the road surface point cloud can be in the form of local mean interpolation processing.

[0069] Please refer to Figure 1 , the road surface defect detection method described in the embodiments of the present invention further includes the following steps:

[0070] Step S300: Input the RGBD multi-source data into a trained defect detection segmentation network, and after being processed by the defect detection segmentation network, output a prediction result of road surface defects.

[0071] Specifically, the defect detection segmentation network includes an input layer, an encoder module, a multi-modal fusion module, a decoder module, and an output layer, and is used to predict road surface defects in the RGBD multi-source data.

[0072] In one implementation, inputting the RGBD multi-source data into a trained defect detection segmentation network, and after being processed by the defect detection segmentation network, outputting a prediction result of road surface defects includes:

[0073] Input the RGBD multi-source data into the input layer of the defect detection segmentation network, and the input layer transmits it to the encoder module;

[0074] Use the encoder to process the first RGB image and the depth map in the RGBD multi-source data to obtain several image features and depth features;

[0075] Input all the image features and the depth features into the multi-modal fusion module, and use the multi-modal fusion module to perform channel attention processing and spatial attention processing on each image feature and each depth feature respectively, and perform feature enhancement and fusion through an exponential operation to obtain several fusion features;

[0076] Input all the fusion features into the decoder module, and process all the fusion features through the decoder module to obtain the prediction result of the road surface defect;

[0077] Output the prediction result through the output layer.

[0078] Specifically, in the present invention, an RGB image of 3*H*W and a depth image of 1*H*W are input into the defect detection and segmentation network. The encoder module of the present invention includes a Unet encoder, a Mamba encoder, and several visual state space modules. The RGB image contains more road details and is processed by the Unet encoder. The Unet encoder includes several convolutional layers, and the convolutional layers are used to extract image features of different scales through convolutional operations. The Mamba encoder is fast and pays more attention to position information, so it is used to process depth images that are insensitive to environmental conditions.

[0079] In the Unet encoder, the feature map (i.e., the output after convolutional operation) performs two convolutional operations in each layer, increasing the number of channels to C*H / 2*W / 2, 2C*H / 4*W / 4, 4C*H / 8*W / 8, where C represents the initial number of channels. In one implementation, C is 64, and H and W are 256.

[0080] The Mamba encoder includes an initial layer and several image patch embedding layers. The initial layer is used to normalize the depth image, and the image patch embedding layers are used to divide the input depth image into multiple image patches and map each image patch to a feature vector of a preset dimension. The visual state space module is used to align the depth features output by the corresponding image patch embedding layer with the image features output by the corresponding convolutional layer in the Unet encoder in terms of size, and then extract the depth features. As Figure 3 shown, after the depth image is normalized by the initial layer of the Mamba encoder, it is processed by the image patch embedding layers to obtain several feature vectors. Then, the visual state space module (i.e., the VSS module) aligns its scale with the features output by the convolutional layer of the Unet encoder, and inputs the image features and depth features into the multi-modal fusion module. The multi-modal fusion module includes a spatial channel exponential fusion attention module (i.e., the SCEFA module). The SCEFA module is used for dual-channel convolutional feature enhancement and exponential fusion, and includes two parallel attention mechanisms. The SCEFA module can fuse more features and transmit more information when predicting images to help segment defects. The feature map V i (i ∈ 1, 2, 3, 4) is processed by a 1D (i.e., one-dimensional) channel attention module to obtain the channel attention processing result M c (V). The feature map C output by the Unet encoder i(i ∈ 1, 2, 3, 4) is processed by a 2D (i.e., two-dimensional) spatial attention module to obtain the spatial attention processing result M S (C). Then, after performing exponential operations respectively and fusing them using an activation function, the fused result (i.e., the fused feature) C is obtained ′ . The flowchart of this process is as Figure 4 shown

[0081] The formulas involved in the above process are as follows

[0082] M c (V) = σ[MLP(P max (V i )) + MLP(P avg (V i ))];

[0083] M s (C) = σ[f 7×7 (P max (C i ))(P avg (C i ))];

[0084]

[0085] where σ represents the sigmoid function (i.e., the activation function), MLP represents the perceptron, P max (.) represents max pooling, P avg (C i ) represents average pooling, and f 7×7 represents that the selection of the convolution kernel size in the convolution operation is 7×7, and e is 2.71828

[0086] After obtaining the fused features, the fused features are passed through skip connections to the decoder for predicting road surface defects. The decoder module includes several upsampling modules and a multi-layer feature attention fusion module (i.e., the MAF module). The features extracted from the Unet encoder and the Mamba encoder are fused to obtain the fused features, which are then passed to the decoder. First, they are processed by the upsampling module and then fed into the MAF module. The MAF module can effectively enhance the attention of the decoding end to the segmentation target defects, thereby improving the prediction accuracy and the generalization of the network. The MAF module processes features through an encoding-decoding upper and lower layer attention mechanism and skip connection fusion. The encoding-decoding upper and lower layer attention mechanism helps to emphasize multi-scale feature extraction. The shallower scale emphasizes capturing context information, such as the category and location of the target, and the deeper scale emphasizes capturing the details of the target. Skip connection fusion is beneficial for fusing information at different scales, such as between the encoder and the decoder, and between the upper and lower layers. Finally, the output features of each layer are restored to the same size as the input features, and then multi-scale information is fused through the Concat operation (i.e., the concatenation operation) to finally output the prediction result of the road surface defect, which is used to determine whether there are defects on the road surface and other situations.

[0087] In the present invention, by processing the fused RGBD multi-source input using a defect detection and segmentation network, the detection rate of road surface defects can be effectively improved. Compared with the traditional method of detecting defects using road surface images, this method makes full use of the color, edges in the RGBD multi-source data, and the depth information of the road surface point cloud, effectively avoiding the problem in the traditional method of identifying shadows, water stains, asphalt repair areas or dirty areas of different colors on the road surface as road surface defects, and effectively improving the accuracy of road surface defect detection. The defect detection and segmentation network is trained with a large amount of RGB image data and point cloud data to ensure the accuracy of the detection results.

[0088] Please refer to Figure 1 , the road surface defect detection method described in the embodiment of the present invention includes the following steps:

[0089] Step S400: Determine the corresponding road surface defect point cloud based on the prediction result, and determine the position, area, and depth of the road surface defect based on the road surface defect point cloud.

[0090] Specifically, when the prediction result is obtained, the corresponding point cloud is extracted from the road surface point cloud based on the prediction result to obtain the road surface defect point cloud. The road surface defect point cloud is the point cloud corresponding to the predicted road surface defect. Then, the position, area, and depth of the defect can be specifically calculated according to the road surface defect point cloud.

[0091] In one implementation, determining the position, area, and depth of the road surface defect based on the road surface defect point cloud includes:

[0092] Calculate the longitude mean, latitude mean, and altitude mean of all the points in the road surface defect point cloud as the position coordinates of the road surface defect;

[0093] Project the road surface defect point cloud onto a two-dimensional plane to obtain two-dimensional plane data, and use the convex hull algorithm to calculate the two-dimensional plane data to obtain the boundary area of the road surface point cloud;

[0094] Based on the calculation of the boundary area, obtain the area of the road surface defect;

[0095] Calculate the height difference between the height of the road surface defect point cloud and the height of the fitted road surface to obtain the depth of the road surface defect.

[0096] Specifically, the point cloud data is a set composed of a large number of three-dimensional coordinate points, which can describe in detail the shape and surface characteristics of the road surface defect position. By analyzing the distribution and density of these points, the area and depth of the defect can be further calculated. The method for calculating the area of the road defect is as follows: project the point cloud data of the defect area onto a two-dimensional plane, and then use a geometric algorithm to estimate the covered area. Use the convex hull algorithm to determine the boundary of the defect to obtain the boundary area of the road surface defect, and finally use the polygon area calculation formula to obtain the area of the road surface defect. Assume that the vertices of the boundary area are (x1, y1), (x2, y2),..., (x n , y n ), then the calculation formula for the polygon area S is:

[0097]

[0098] In one implementation, the method further includes:

[0099] Generate a condition assessment report for the road defect based on the position, area, and depth of the road surface defect.

[0100] Specifically, the process of generating the condition assessment report for the road defect is as Figure 5 shown. When the position, area, and depth are gradually calculated, a condition assessment report containing information such as the type, influence range, and severity of the defect can be generated. Users can carry out subsequent road maintenance work based on the condition assessment report.

[0101] In summary, the present invention obtains the road surface point cloud and RGB image corresponding to the target road surface; processes the road surface point cloud and the RGB image to obtain RGBD multi-source data; inputs the RGBD multi-source data into a trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, outputs the prediction result of road surface defects; determines the corresponding road surface defect point cloud based on the prediction result, and determines the position, area, and depth of the road surface defects based on the road surface defect point cloud. The present invention combines point cloud data and RGB images to generate RGBD multi-source data, uses a trained defect detection and segmentation network to predict defects, and determines the specific position, area, and depth of the defects according to the prediction result, which can effectively improve the accuracy of road surface defect detection.

[0102] In one embodiment, as Figure 6 shown, based on the above road surface defect detection method, the present invention also correspondingly provides a road surface defect detection device, including:

[0103] A data acquisition module 100, configured to acquire the road surface point cloud and RGB image corresponding to the target road surface;

[0104] A multi-source data generation module 200, configured to process the road surface point cloud and the RGB image to obtain RGBD multi-source data;

[0105] A defect prediction module 300, configured to input the RGBD multi-source data into a trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, output the prediction result of road surface defects;

[0106] A defect information generation module 400, configured to determine the corresponding road surface defect point cloud based on the prediction result, and determine the position, area, and depth of the road surface defects based on the road surface defect point cloud.

[0107] In one embodiment, the device further includes:

[0108] A regional point cloud acquisition unit, configured to acquire the regional point cloud collected by the lidar for the target region, where the target region includes the target road surface and the peripheral region of the target road surface;

[0109] A point cloud segmentation unit, configured to segment the regional point cloud by using a road surface point cloud segmentation algorithm to obtain the road surface point cloud corresponding to the target road surface.

[0110] In one embodiment, the device further includes:

[0111] A point cloud division unit, configured to divide the road surface point cloud into normal point cloud and outlier point cloud by using a preset cylindrical model;

[0112] A depth value calculation unit, configured to fit a road surface using the normal point cloud to obtain a fitted road surface, calculate the distance from the road surface point cloud to the fitted road surface, and obtain the depth value of the road surface point cloud;

[0113] A depth map generation unit, configured to perform interpolation processing on the depth values of the road surface point cloud to obtain the interpolated depth values of the road surface point cloud, and generate a depth map corresponding to the road surface point cloud from the interpolated depth values of the road surface point cloud;

[0114] A data fusion unit, configured to perform distortion removal processing on the RGB image to obtain a first RGB image, and register it with the depth map to obtain RGBD multi-source data.

[0115] In one embodiment, the defect detection and segmentation network includes an input layer, an encoder module, a multi-modal fusion module, a decoder module, and an output layer; the defect prediction module includes:

[0116] A data input unit, configured to input the RGBD multi-source data into the input layer of the defect detection and segmentation network, and transmit it to the encoder module through the input layer;

[0117] An encoding unit, configured to process the first RGB image and the depth map in the RGBD multi-source data using the encoder module to obtain several image features and depth features;

[0118] A fusion unit, which inputs all the image features and the depth features into the multi-modal fusion module, uses the multi-modal fusion module to perform channel attention processing and spatial attention processing on each image feature and each depth feature respectively, and performs feature enhancement and fusion through an exponential operation to obtain several fusion features;

[0119] A decoding unit, configured to input all the fusion features into the decoder module, and process all the fusion features through the decoder module to obtain a prediction result of the road surface defect;

[0120] An output unit, configured to output the prediction result through the output layer.

[0121] In one embodiment, the defect information generation module includes:

[0122] A position calculation unit, configured to calculate the longitude mean, latitude mean, and altitude mean of all the point clouds in the road surface defect point cloud as the position coordinates of the road surface defect;

[0123] A boundary region generation unit, configured to project the road surface defect point cloud onto a two-dimensional plane to obtain two-dimensional plane data, and calculate the boundary region of the road surface point cloud using a convex hull algorithm for the two-dimensional plane data;

[0124] An area calculation unit for calculating based on the boundary region to obtain the area of the road surface defect;

[0125] A depth calculation unit for calculating the height difference between the height of the road surface defect point cloud and the height of the fitted road surface to obtain the depth of the road surface defect.

[0126] In one embodiment, the device further includes:

[0127] A report generation unit for generating a condition assessment report of the road defect based on the position, area, and depth of the road surface defect.

[0128] Based on the above embodiment, the present invention also provides a terminal, the principle block diagram of which can be as Figure 7 shown. The above terminal includes a processor, a memory, a network interface, and a display screen connected through a device bus. Among them, the processor of the terminal is used to provide computing and control capabilities. The memory of the terminal includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating device and a road surface defect detection program. The internal memory provides an environment for the operation of the operating device and the road surface defect detection program in the non-volatile storage medium. The network interface of the terminal is used to communicate with an external terminal through a network connection. When the road surface defect detection program is executed by the processor, it implements the steps of any one of the above road surface defect detection methods. The display screen of the terminal can be a liquid crystal display screen or an electronic ink display screen.

[0129] Those skilled in the art can understand that Figure 7 the principle block diagram shown in

[0130] is only a block diagram of some structures related to the solution of the present invention, and does not constitute a limitation on the terminal to which the solution of the present invention is applied. The specific terminal may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0131] In one embodiment, a terminal is provided. The above terminal includes a memory, a processor, and a road surface defect detection program stored on the above memory and executable on the above processor. When the above road surface defect detection program is executed by the above processor, it implements the steps of any one of the road surface defect detection methods provided by the embodiments of the present invention.

[0132] It should be understood that the sequence numbers of the steps in the above embodiments do not indicate the order of execution, and the order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.

[0133] Those skilled in the art can clearly understand that for the convenience and conciseness of description, only the above-mentioned division of each functional unit and module is used as an example. In practical applications, the above functions can be allocated to different functional units and modules as needed, that is, the internal structure of the above device can be divided into different functional units or modules to complete all or part of the functions described above. Each functional unit and module in the embodiment can be integrated into a processing unit, or each unit can exist physically alone, or two or more units can be integrated into one unit. The above integrated unit can be implemented in the form of hardware or in the form of a software functional unit. In addition, the specific names of each functional unit and module are only for the convenience of mutual distinction and do not limit the protection scope of the present invention. The specific working processes of the units and modules in the above device can refer to the corresponding processes in the foregoing method embodiments and will not be elaborated herein.

[0134] In the above embodiments, each embodiment is described with emphasis. For the parts not detailed or recorded in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0135] Those of ordinary skill in the art can realize that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be implemented by electronic hardware, or by a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. Skilled professionals can use different methods for each specific application to implement the described functions, but such implementation should not be considered to exceed the scope of the present invention.

[0136] In the embodiments provided by the present invention, it should be understood that the disclosed device / terminal device and method can be implemented in other ways. For example, the device / terminal device embodiments described above are only illustrative. For example, the above-mentioned division of modules or units is only a logical function division. In actual implementation, there can be other division methods. For example, multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0137] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or perform equivalent replacements for some of the technical features; and these modifications or replacements do not essentially depart from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A road surface defect detection method, characterized in that, The method includes: Obtaining the road surface point cloud and RGB image corresponding to the target road surface; Processing the road surface point cloud and the RGB image to obtain RGBD multi-source data; Inputting the RGBD multi-source data into a trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, outputting the prediction result of the road surface defect; Determining the corresponding road surface defect point cloud based on the prediction result, and determining the position, area and depth of the road surface defect based on the road surface defect point cloud.

2. The pavement defect detection method according to claim 1, wherein Before obtaining the road surface point cloud corresponding to the target road surface, it further includes: Obtaining the regional point cloud collected by the lidar for the target area, where the target area includes the target road surface and the surrounding area of the target road surface; Using a road surface point cloud segmentation algorithm to segment the regional point cloud to obtain the road surface point cloud corresponding to the target road surface.

3. The road surface defect detection method according to claim 1, characterized in that Processing the road surface point cloud and the RGB image to obtain RGBD multi-source data, including: Dividing the road surface point cloud into normal point cloud and outlier point cloud by using a preset cylinder model; Using the normal point cloud to fit the road surface to obtain a fitted road surface, calculating the distance from the road surface point cloud to the fitted road surface to obtain the depth value of the road surface point cloud; After performing interpolation processing on the depth value of the road surface point cloud, obtaining the interpolated depth value of the road surface point cloud, and generating a depth map corresponding to the road surface point cloud from the interpolated depth value of the road surface point cloud; After performing distortion removal processing on the RGB image, obtaining a first RGB image, and registering it with the depth map to obtain RGBD multi-source data.

4. The pavement defect detection method according to claim 3, characterized in that, The defect detection and segmentation network includes an input layer, an encoder module, a multi-modal fusion module, a decoder module and an output layer; Inputting the RGBD multi-source data into a trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, outputting the prediction result of the road surface defect, including: Inputting the RGBD multi-source data into the input layer of the defect detection and segmentation network, and transmitting it to the encoder module through the input layer; Using the encoder module to process the first RGB image and the depth map in the RGBD multi-source data to obtain several image features and depth features; Inputting all the image features and the depth features into the multi-modal fusion module, using the multi-modal fusion module to perform channel attention processing and spatial attention processing on each image feature and each depth feature respectively, and performing feature enhancement and fusion through an exponential operation to obtain several fusion features; Inputting all the fusion features into the decoder module, and after being processed by the decoder module for all the fusion features, obtaining the prediction result of the road surface defect; Outputting the prediction result through the output layer.

5. The pavement defect detection method according to claim 4, characterized in that The encoder module includes a Unet encoder, a Mamba encoder, and several visual state space modules. The Unet encoder contains several convolutional layers, which are used to extract image features of different scales through convolutional operations. The Mamba encoder contains an initial layer and several image patch embedding layers. The initial layer is used to normalize the depth image, and the image patch embedding layers are used to divide the input depth image into multiple image patches and map each image patch to a feature vector of a preset dimension. The visual state space module is used to align the depth features output by the corresponding image patch embedding layer with the image features output by the corresponding convolutional layer in the Unet encoder in terms of size.

6. The road surface defect detection method according to claim 3, characterized in that, Determining the location, area, and depth of the road surface defect based on the road surface defect point cloud includes: Calculating the longitude mean, latitude mean, and height mean of all the point clouds in the road surface defect point cloud as the location coordinates of the road surface defect; Projecting the road surface defect point cloud onto a two-dimensional plane to obtain two-dimensional plane data, and using the convex hull algorithm to calculate the two-dimensional plane data to obtain the boundary region of the road surface point cloud; Calculating the area of the road surface defect based on the boundary region; Calculating the height difference between the height of the road surface defect point cloud and the height of the fitted road surface to obtain the depth of the road surface defect.

7. The pavement defect detection method according to claim 1, characterized in that The method further includes: Generating a condition assessment report of the road defect based on the location, area, and depth of the road surface defect.

8. A road surface defect detection device, characterized in that, Including: A data acquisition module, which is used to acquire the road surface point cloud and RGB image corresponding to the target road surface; A multi-source data generation module, which is used to process the road surface point cloud and the RGB image to obtain RGBD multi-source data; A defect prediction module, which is used to input the RGBD multi-source data into a trained defect detection and segmentation network, and after being processed by the defect detection and segmentation network, output the prediction result of the road surface defect; A defect information generation module, which is used to determine the corresponding road surface defect point cloud based on the prediction result, and determine the location, area, and depth of the road surface defect based on the road surface defect point cloud.

9. A terminal, characterized in that, The terminal includes a memory, a processor, and a road surface defect detection program stored on the memory and executable on the processor. When the road surface defect detection program is executed by the processor, the steps of the road surface defect detection method according to any one of claims 1-7 are implemented.

10. A computer-readable storage medium, characterized in that, A road surface defect detection program is stored on the computer-readable storage medium. When the road surface defect detection program is executed by the processor, the steps of the road surface defect detection method according to any one of claims 1-7 are implemented.

Images